Your AI Product Is Flying Blind

Product analytics was invented to answer one question: what are users doing? The tools built to answer it were designed in an era when software was deterministic: event tracking, funnels, cohort analysis, A/B testing. Click a button, trigger an action. The product is fixed. The output is predictable. The only variable is the user.

In that world, instrumentation is straightforward. You define the events that matter, you fire them when users trigger them, you aggregate and analyze. Mixpanel, Amplitude, Google Analytics: the entire stack was built on this assumption. It works because the product itself is stable. You are measuring one moving thing against a fixed reference point.

This held through the mobile era. The interfaces changed: touch instead of click, smaller screens, location context. But the underlying contract did not. Tap a button, get a result. The result is the same every time. Analytics adapted to new surfaces. The paradigm it was built on stayed intact.

AI arrived and broke the contract at its foundation.

The first thing AI changed was the output. It is no longer deterministic. The same user, the same input, a different response. Sometimes subtly different, sometimes completely different. The product is no longer fixed. You now have two variables: model behavior and user behavior, with no stable reference point between them. Traditional analytics was built to measure one variable. It has no framework for two.

The second thing AI changed was the interface. AI is embedded in search results, document editors, code completion, recommendation systems, background automation, decision support tools that surface outputs without a chat window in sight. What unites all of these is that output is now generated, not retrieved. A button has one state. Generated output has infinite states, and every exchange is unique. Funnels do not work because there is no fixed path. Events do not capture meaning because the same event, "user sent message" or "recommendation surfaced," contains nothing about what the model produced or whether it was any good. The instrumentation primitives that analytics was built on collapse against generated interfaces.

The third thing AI changed was the potential location of value creation. Traditional software creates value inside the product: the task completes, the completion is logged, the loop closes. Many AI products work the same way. But a meaningful subset do not. When the model outputs a recommendation, a plan, a diagnosis, a draft, the value is created when the user acts on it, somewhere outside the session, after the product interaction ends. The consequence begins elsewhere. For those products, the analytics loop does not close by default, and designing instrumentation that reaches it requires a different strategy entirely.

What is needed instead is three layers, connected.

The first layer is observability. What the model did. Every input, every output, every intermediate step, every tool call, every token count, every latency measurement. Not summaries. Traces. The complete record of what happened inside the system on every single inference. The tooling exists: Langsmith, Arize, Braintrust, Helicone, PostHog. Most teams have some version running. The data is there.

The second layer is evaluation. Whether what the model did was any good. That judgment requires domain knowledge no off-the-shelf framework can supply. Not a generic helpfulness score on a scale of one to five. That number has been proven to correlate with nothing that matters. A domain-specific, binary judgment: did this output pass or fail, for this product, for this user, in this context. Built by looking at real traces, doing error analysis manually, finding real failure modes, letting failure categories emerge from the data rather than being imposed on it, and calibrating an automated judge against a real domain expert until the judge can be trusted. This is where most teams fall short. They skip evaluation entirely, or they implement it badly and trust it anyway, or they buy an off-the-shelf framework that gives them metrics that sound meaningful and are not.

The third layer is behavioral instrumentation. What users did in response to what the model produced. Not generic product analytics. Not sessions, funnels, and DAU applied to an AI product like a label on the wrong box. Instrumentation designed specifically around the moments where model output and user action intersect. Did the user act on what the model said? Did they correct it? Did they leave immediately after a specific output? Did they return? Did they escalate to a human? These signals are not captured by default by any analytics tool because they require knowing which user action followed which model output. That join requires deliberate instrumentation design, not out-of-the-box event tracking.

Any two layers without the third is incomplete. Observability and evaluation without behavioral instrumentation tells you model quality but not whether it matters to users. Behavioral instrumentation and observability without evaluation tells you what users did but not what the model actually produced or why. Evaluation and behavioral instrumentation without observability gives you quality signals and behavior signals but no trace to debug when the two diverge. The join between all three is what produces actionable signal. That join is what almost no team has built.

The reason the join is rare is not technical. Connecting a trace to a product analytics event is light engineering work. The reason it is rare is that the three layers are owned by different people with different incentives and different tools. The ML engineer owns the traces. The product manager owns the analytics. The evaluation system, if it exists at all, lives with whoever had time to build it. Nobody owns the connection. Nobody is accountable to the question the connection would answer.

And the question is important. It is: when the model fails, does it matter to the user?

A model that hallucinates a product feature that does not exist has failed technically. If the user was browsing, never intended to act, and moved on, the failure was invisible. No churn, no complaint, no signal. A model that describes a loan rate as fixed when it is variable has failed the user in a way no trace and no eval will surface on its own. The user accepted the recommendation, signed the contract, discovered the error three months later. The product session showed a positive outcome: flow completed. The actual failure happened elsewhere.

This is the hardest version of the problem. The output is gone. The user is gone. The consequence is somewhere else. The only instrumentation available is the signals just before the user leaves: did they hesitate, did they correct, did they ask a follow-up, did they abandon. Those signals are weak proxies for what actually happened after. Designing an instrumentation strategy around those proxies, understanding their limits, and being honest about what they can and cannot tell you. That is the work.

It is not a tool problem. It is a judgment problem: deciding what to measure, at what granularity, with what proxy signals, given the structural limits of your context. That judgment cannot be automated. You might be tempted to try, but you will only be fed your own biases back. It requires understanding the product, the users, the workflow, and the specific question you are trying to answer. Every product has a different answer. There is no off-the-shelf version of this.

As AI moves from feature to foundation, embedded in every surface and not just chat interfaces, analytics has to be rebuilt from the question up.

Not: which events do I track.

But: given what my product does, given my users, given where value is actually created, what is the most informative signal I can reasonably collect?