How to Measure If Your AI Feature Is Actually Working (A Guide for Product Teams)

You shipped an AI feature three weeks ago. The CEO loves it. A few users posted nice things on Twitter. But your conversion numbers haven't moved, support tickets haven't dropped, and you have no idea whether the thing is actually helping anyone — or just burning API credits.

This is the most common failure mode for AI features in production: nobody defines what "working" means before shipping. You end up measuring impressions or API call volume, which tells you people are clicking the feature, not that it's useful. Measuring AI feature performance requires different thinking than measuring traditional product features, because the output is non-deterministic and user trust is fragile.

Here's a practical framework for figuring out whether your AI feature is earning its keep.

Why traditional product metrics aren't enough

With a regular feature — say, a new filter on a search page — you measure click-through rate, and you're mostly done. The filter either works or it doesn't. AI features are different because:

1. The output quality varies per request. The same prompt can produce good output one time and garbage the next.
2. Users lose trust fast. One bad response can make a user stop using the feature entirely, even if 90% of responses are good.
3. Cost scales with usage. Unlike traditional features where server costs are relatively flat, every AI interaction has a direct token cost.

You need metrics that capture quality, trust, and economics — not just engagement.

The four metrics that actually matter

1. Task completion rate

This is the single most important metric. Did the user accomplish what they were trying to do with the AI feature?

How you measure this depends on the feature:

• AI search: Did the user click a result, or did they immediately refine their query?
• AI writing assistant: Did the user keep the generated text, or delete it and write their own?
• AI chatbot: Did the user's conversation end with a resolution, or did they escalate to a human?

Here's a simple instrumentation pattern:

// src/lib/ai-metrics.ts
interface AIInteractionEvent {
    featureId: string
    sessionId: string
    timestamp: number
    action: 'generated' | 'accepted' | 'rejected' | 'edited' | 'escalated'
    metadata?: Record<string, string | number>
}

export function trackAIInteraction(event: AIInteractionEvent): void {
    // Send to your analytics pipeline — Segment, PostHog, Mixpanel, etc.
    fetch('/api/analytics/ai-interaction', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(event),
    }).catch((err) => console.error('Analytics tracking failed:', err))
}

// Usage in a summarization feature
export function trackSummaryOutcome(
    sessionId: string,
    outcome: 'kept' | 'edited' | 'discarded'
): void {
    trackAIInteraction({
        featureId: 'doc-summary',
        sessionId,
        timestamp: Date.now(),
        action: outcome === 'kept' ? 'accepted' : outcome === 'edited' ? 'edited' : 'rejected',
    })
}

Calculate task completion rate as: (accepted + edited) / total_generated. If this is below 60%, your feature has a quality problem.

2. User override rate

This is the percentage of times a user modifies or replaces the AI output. It's distinct from rejection — an override means the AI got them partway there but wasn't good enough.

A high override rate (above 40%) isn't necessarily bad. It might mean the feature is useful as a starting point. But if override rate is high and task completion rate is low, users are struggling with your feature.

Track overrides by comparing the AI output with what the user actually submitted:

// src/lib/override-tracking.ts
import { diffWords } from 'diff'

interface OverrideMetrics {
    originalLength: number
    finalLength: number
    changePercentage: number
    wasFullRewrite: boolean
}

export function calculateOverride(aiOutput: string, userFinal: string): OverrideMetrics {
    const changes = diffWords(aiOutput, userFinal)
    const changedChars = changes
        .filter((part) => part.added || part.removed)
        .reduce((sum, part) => sum + (part.value?.length ?? 0), 0)
    const totalChars = Math.max(aiOutput.length, userFinal.length)
    const changePercentage = totalChars > 0 ? (changedChars / totalChars) * 100 : 0

    return {
        originalLength: aiOutput.length,
        finalLength: userFinal.length,
        changePercentage,
        wasFullRewrite: changePercentage > 80,
    }
}

3. Cost per successful interaction

This is where most teams get surprised. You need to track not just total API spend, but cost per successful outcome.

// src/lib/cost-tracker.ts
interface CostEntry {
    featureId: string
    inputTokens: number
    outputTokens: number
    model: string
    wasSuccessful: boolean
}

const MODEL_PRICING: Record<string, { input: number; output: number }> = {
    'claude-sonnet-4-20250514': { input: 3.0 / 1_000_000, output: 15.0 / 1_000_000 },
    'claude-haiku-4-5-20251001': { input: 0.8 / 1_000_000, output: 4.0 / 1_000_000 },
    'gpt-4o': { input: 2.5 / 1_000_000, output: 10.0 / 1_000_000 },
}

export function calculateCostPerSuccess(entries: CostEntry[]): {
    totalCost: number
    successfulInteractions: number
    costPerSuccess: number
    costPerInteraction: number
} {
    let totalCost = 0
    let successfulInteractions = 0

    for (const entry of entries) {
        const pricing = MODEL_PRICING[entry.model]
        if (!pricing) continue

        const cost = entry.inputTokens * pricing.input + entry.outputTokens * pricing.output
        totalCost += cost
        if (entry.wasSuccessful) successfulInteractions++
    }

    return {
        totalCost,
        successfulInteractions,
        costPerSuccess: successfulInteractions > 0 ? totalCost / successfulInteractions : 0,
        costPerInteraction: entries.length > 0 ? totalCost / entries.length : 0,
    }
}

If your AI chatbot costs $0.12 per conversation but only resolves 30% of queries, your effective cost per resolution is $0.40. That might still be cheaper than a human agent, but now you can actually compare.

4. Repeat usage rate

Do users come back to the feature? First-time usage is curiosity. Second and third time is signal. If your 7-day retention for the AI feature is below 20%, users tried it and decided it wasn't worth their time.

This one you can track with standard product analytics — no AI-specific instrumentation needed. Just filter your retention cohorts by feature usage.

How to A/B test AI features

A/B testing AI features has a wrinkle: the output isn't deterministic. You can't just compare "with feature" vs "without feature" because two users with the feature might get wildly different quality responses.

Here's an approach that works:

Test the prompt, not just the feature

Instead of testing AI on/off, test different prompts or configurations against each other. This gives you much more actionable data:

// src/lib/ab-test-ai.ts
interface ABTestConfig {
    testId: string
    variants: {
        id: string
        systemPrompt: string
        model: string
        temperature: number
        weight: number // Traffic allocation (0-1)
    }[]
}

export function selectVariant(config: ABTestConfig, userId: string): (typeof config.variants)[number] {
    // Deterministic assignment based on user ID
    let hash = 0
    const key = `${config.testId}:${userId}`
    for (let i = 0; i < key.length; i++) {
        hash = (hash * 31 + key.charCodeAt(i)) | 0
    }
    const bucket = Math.abs(hash) % 1000 / 1000

    let cumulative = 0
    for (const variant of config.variants) {
        cumulative += variant.weight
        if (bucket < cumulative) return variant
    }
    return config.variants[config.variants.length - 1]
}

// Example: testing a shorter vs longer system prompt
const summaryTest: ABTestConfig = {
    testId: 'summary-prompt-v2',
    variants: [
        {
            id: 'control',
            systemPrompt: 'Summarize the following document in 2-3 paragraphs.',
            model: 'claude-sonnet-4-20250514',
            temperature: 0.3,
            weight: 0.5,
        },
        {
            id: 'structured',
            systemPrompt:
                'Summarize the following document. Start with a one-sentence TLDR, then list 3-5 key points as bullet points.',
            model: 'claude-sonnet-4-20250514',
            temperature: 0.3,
            weight: 0.5,
        },
    ],
}

Sample size matters more than you think

Because AI output varies, you need larger sample sizes to reach statistical significance. A good rule of thumb: plan for 2-3x the sample size you'd use for a traditional A/B test. If you'd normally want 1,000 users per variant, plan for 2,500.

Measure quality, not just engagement

Don't just measure "did they click." Measure the downstream outcome. For a summarization feature, track whether users who got summaries from variant A were more likely to complete their task (e.g., share the document, take an action) than those who got summaries from variant B.

When to kill an AI feature

This is the hardest part, and most teams never do it. Here are the kill signals:

Kill immediately if:

• Task completion rate is below 30% after two weeks of optimization
• Users actively complain about the feature in support tickets (not just "it could be better" — actual frustration)
• Cost per successful interaction exceeds the value it delivers (e.g., your AI support bot costs more per resolution than your human agents)

Investigate before killing if:

• Task completion rate is 30-50% — there might be a prompt or UX fix
• Override rate is high but completion rate is decent — users might be getting value but the output needs refinement
• Usage is low but satisfaction is high — you might have a discoverability problem, not a quality problem

Keep iterating if:

• Task completion rate is above 50% and trending up
• Users who use the feature have measurably better outcomes than those who don't
• Cost per success is within your budget and trending down

Building a simple dashboard

You don't need a fancy tool for this. A simple SQL query against your analytics data will do:

-- Weekly AI feature health check
SELECT
    feature_id,
    DATE_TRUNC('week', created_at) AS week,
    COUNT(*) AS total_interactions,
    COUNT(*) FILTER (WHERE action = 'accepted') AS accepted,
    COUNT(*) FILTER (WHERE action = 'edited') AS edited,
    COUNT(*) FILTER (WHERE action = 'rejected') AS rejected,
    ROUND(
        100.0 * COUNT(*) FILTER (WHERE action IN ('accepted', 'edited'))
        / NULLIF(COUNT(*), 0), 1
    ) AS completion_rate_pct,
    ROUND(
        100.0 * COUNT(*) FILTER (WHERE action = 'edited')
        / NULLIF(COUNT(*) FILTER (WHERE action IN ('accepted', 'edited')), 0), 1
    ) AS override_rate_pct
FROM ai_interactions
WHERE created_at > NOW() - INTERVAL '8 weeks'
GROUP BY feature_id, DATE_TRUNC('week', created_at)
ORDER BY feature_id, week;

Run this every Monday. If task completion rate drops two weeks in a row, something changed — maybe a model update, maybe a shift in user behavior. Either way, you need to investigate.

The meta-lesson

The biggest mistake product teams make with AI features is treating them as "set and forget." Traditional features are mostly deterministic — once they work, they work. AI features drift. Models get updated, user expectations shift, and prompt performance degrades as edge cases accumulate.

Build measurement into the feature from day one. Not after launch, not "when we have time." The instrumentation code above is maybe an hour of work. Skipping it means you'll spend months debating whether the feature is working based on anecdotes and gut feelings.

Track task completion rate, override rate, cost per success, and repeat usage. If all four are healthy, you've got a real feature. If they're not, you've got expensive theater.

What's next

If you've identified that your AI feature has a quality problem, the next step is often improving your prompts and evaluation pipeline. Check out How to Test AI Features: Unit Testing LLM-Powered Code for patterns on building automated quality checks that catch regressions before your users do.