User Intent Prediction #13: Doing Nothing Won.
My AI lost a debate to a rock. The rock made $500 more.
I built a Ferrari to drive in a swamp. The tractor won.
I pitted my smart, adaptive, learning Bandit against a “dumb” baseline.
The Contenders:
- Smart Bandit: Thompson Sampling, Bayesian updates, Contextual features.
- Dumb Baseline: “Do Nothing” (Always show standard price).
The Setup:
- Duration: 2 weeks
- Traffic: 50/50 split (10,000 users each)
- Goal: Maximize revenue
The Result:
- Smart Bandit: $12,400 Revenue (186 sales)
- Dumb Baseline: $12,900 Revenue (193 sales)
My AI lost $500. It worked hard to lose that money.
The Breakdown
Where did the Bandit lose?
| Scenario | Bandit Action | Baseline Action | Outcome |
|---|---|---|---|
| High Intent User | Full Price ✅ | Full Price ✅ | Tie |
| Medium Intent User | Discount (Exploring) ❌ | Full Price ✅ | Lost $10 margin |
| Low Intent User | Full Price (Exploring) ❌ | Full Price ❌ | Tie (both lost sale) |
The Bandit made mistakes in both directions:
- It gave discounts to users who would have paid full price. (Lost margin).
- It gave full price to users who needed a discount. (Lost sale).
It was “learning” which users needed discounts. But the learning cost more than the profit.
The “Data Starvation” Problem
Why did it fail? Because 1.5% conversion is a desert.
For every 100 users, the Bandit gets 1 or 2 signals (sales). 98 signals are “No Sale.”
To learn a pattern, the Bandit needs thousands of sales. To get thousands of sales, it needs hundreds of thousands of users.
I didn’t have Google-scale traffic. My Bandit was starving. It was making random guesses because it didn’t have enough data to be smart.
The Math:
- Bandit Convergence: Needs ~1,000 sales to learn.
- My Traffic: 1,000 users/day × 1.5% conversion = 15 sales/day.
- Time to Converge: 67 days.
I couldn’t afford 67 days of “learning tax.”
The Cost of Complexity
The “Dumb Baseline” had one advantage: Stability. It didn’t explore. It didn’t make mistakes “to learn.” It just worked.
The Bandit paid a “learning tax” on every user. And the tax was higher than the profit.
The Lesson
Don’t use Big Data tools on Small Data problems.
I was using algorithms designed for Google Ads (billions of clicks) on a niche diet funnel. I over-estimated the power of the algorithm. I under-estimated the cost of the learning curve.
Sometimes “dumb” is just “efficient.”
This leads to the final fix: Calibration—if the model is dumb, at least make it honest.