Two five-star reviews arrived the same week, asking for opposite things

Appray's interview is eight rounds of questions before it gives you a Build, Kill or Prove verdict. I picked eight after cutting it down from something longer during testing, and I'd more or less stopped thinking about the number. Then two reviews landed four days apart, both about the exact same part of the app, wanting exactly opposite things.

Reviewer one: "Wish it were quicker, I know roughly what I think before I start." Reviewer two: "Glad it isn't one of those apps that spits out an answer in ninety seconds, I wouldn't trust that for a decision I'm about to spend three months on." Both four and five stars. Both, as far as I can tell, real users who'd actually used the thing rather than someone leaving a drive-by rating.

The instinct to average

My first thought, honestly, was six rounds. Split the difference, annoy both people slightly less. I got as far as sketching which two rounds I'd cut before I stopped, because six rounds doesn't serve either job. The person who wants speed still has to sit through six questions they consider excessive. The person who wants rigour now has a shorter interview feeding a verdict they were already worried was too quick. Averaging two complaints doesn't produce a answer either person asked for, it produces a third thing nobody asked for, and I'd have shipped it feeling like I'd made a sensible compromise.

I don't have the data to break the tie

Here's the part that's specific to how I built this app. Appray runs entirely on your device, no account, nothing sent anywhere, no analytics. That was a deliberate choice, the whole point is that what you're deciding to build doesn't leave your phone. Most days I think that trade was right.

This was not one of those days. A normal app would have a funnel: started interview, got to round four, got to round eight, requested verdict. I could have looked at where people actually drop off and had an answer in ten minutes instead of a week of turning two reviews over in my head. I built the one kind of app where that shortcut doesn't exist, on purpose, and then hit the exact decision where I wanted it back.

Reading both reviews properly instead of skimming them

What actually moved this along was going back and reading each review in full rather than the line I'd remembered from it. I'd been treating "too long" and "too fast" as two votes on the same dial, more questions versus fewer questions. Reading the whole thing, reviewer one's complaint was mostly about not being able to leave and come back without losing progress, they mentioned starting it on a commute and having to restart later. That's not really a complaint about eight rounds, it's a complaint about no save state. Reviewer two, further down, wasn't asking for more questions either. They wanted the verdict screen to show which of their own answers had actually driven it, because "trust it more if I can see my reasoning, not just a colour." Also not a request about round count.

Neither review was actually about the length of the interview. They both used "long" and "fast" as the nearest word for a feeling that had a completely different cause. I'd have averaged my way to a worse interview solving neither problem, and felt good about it, because on a skim the two reviews looked like a clean disagreement I'd cleverly split down the middle.

What I actually did

I left the eight rounds alone. Both real issues went into the same plain note where every other request goes: one person on session resume, one person on showing your own answers back to you at the verdict. Neither is a build yet, both are a single data point each, which by my own rule (three questions: who's asking, is it more than one person, would I have thought of it myself) isn't enough on its own. But they're filed as what they actually are now, not as a vague "some people want it shorter, some want it longer" that I'd have quietly ignored the next time I looked at that note, because averaged-out feedback like that reads as noise and gets skipped.

The lesson I keep relearning is that two people disagreeing loudly is not the same as two people describing the same feature. It usually feels the same from the founder's chair, you get two messages using opposite words about the same screen, and the natural move is to find the midpoint. Most of the time there isn't a midpoint to find, because they were never actually talking about the same thing. You only find that out by reading the whole review instead of the sentence you quoted to yourself while getting annoyed about it.