Appray used to run on GPT-4. I pulled it after three weeks.

I don't bring this up in the App Store listing or on the landing page, for obvious reasons. "No AI" reads better as a principle than as a thing I arrived at because the AI version kept embarrassing me. But a reader asked me last week, on a Quora thread about whether people validate startup ideas with ChatGPT, whether I'd ever tried building the verdict logic on top of an actual model instead of writing the rules myself. I had. It's a fair question and it deserves a straight answer rather than the tidy origin story.

Why I reached for it first

The whole point of Appray is to take someone's answers to eight rounds of questions and turn them into a Build, Kill or Prove verdict, plus interview questions written from what they actually said. That sounds exactly like a job for a language model, and in June it took me four days to get a working prototype: collect the eight answers, drop them into a prompt with the verdict categories explained, send it to GPT-4, parse the response. It worked on the first real try. I remember being pleased with myself for about two weeks.

The afternoon I noticed something was wrong

I was re-testing a feature idea I'd run through Appray three times already while writing demo copy, a project-tagging feature for a notes app. First run: Kill. Second run, same answers, typed in fresh rather than copy-pasted: Prove. Third run: Build, with a paragraph explaining why tagging was clearly core to the product.

Same eight answers, worded slightly differently each time because I was retyping them rather than pasting, three different verdicts. I went back and ran the exact same input, character for character, forty times in a row. Fourteen of the forty came back different from the majority verdict. Not close calls with a sentence of hedging either. Flat contradictions, delivered with the same confident tone each time.

That is the one finding from this whole experiment I'd ask you to actually remember, if you remember nothing else: a model can sound equally certain about four different answers to the same question. Confidence in the output tells you nothing about whether the input was understood, let alone understood consistently.

What it actually cost, beyond the money

The money was real but small, something like four to six cents a verdict at the rate I was calling the API, which is nothing at app-store volume. The bigger cost was the one I hadn't priced in: every verdict took seven to eleven seconds to come back, and Appray runs entirely on the device with no network call anywhere else in it, so a GPT-4 round trip was the single slowest thing happening in the whole app by a wide margin. Offline doesn't work either, which matters for an app that's explicitly meant to be used sitting on a train with no signal, thinking about a feature.

None of that was the reason I ripped it out, though. I could have cached aggressively, or eaten the latency, or shipped an online-only mode and called it a limitation. The inconsistency is what killed it, because the entire premise of Appray is that the verdict is accountable to your own answers. If you can get a different verdict by rerunning the exact same input, there is nothing to be accountable to. It's a coin flip wearing a blazer.

What replaced it

I rewrote the whole thing as a decision tree in Swift, no model involved. Each of the eight questions maps to a small set of weighted signals, there are nineteen rules total, and the rules are the actual thing the app is selling, not dressing around a black box. Run the same answers through it a thousand times and you get the same verdict a thousand times, because it's arithmetic, not a guess. I can also show you, in plain language, exactly which answer tipped a verdict from Kill to Prove, because the rule that fired is right there in the code, not buried in weights I can't read.

It took five weeks to rebuild what GPT-4 had done in four days, and the first version of the decision tree was worse than the model at handling answers that didn't fit neatly into a category. People don't always answer "yes" or "no," they write "sort of, depends on the customer," and a rule-based system has to be told explicitly what to do with that in a way a language model doesn't. I spent most of those five weeks on exactly that problem; the happy-path logic took three days.

Where I think the honest answer is more boring than either side wants

I'm not going to tell you AI is useless for this, because it clearly isn't, for the parts of the job that don't need to be repeatable. If I were still drafting interview questions by hand for every category of feature, a model would save me real time, and plenty of people building something like this should use one for exactly that. Where it falls over is the part where the product's whole promise is "the same situation gets the same answer," and that promise is precisely the one a language model can't make about itself, because nobody, including the people who trained it, can tell you in advance which of several plausible completions it'll pick on a given run.

So when people ask me whether they should run their startup idea through ChatGPT before building, my answer isn't no. It's: fine, if you're using it to generate better questions to ask real customers, or to notice an angle you hadn't considered. Don't use it as the verdict, and don't trust that getting a reassuring answer once means you'd get the same one twice. I only found that out because I happened to retype the same answers instead of pasting them, and the app told me three different things in one afternoon.