User Intent Prediction #4: Perfect Clustering Becomes Zero Prediction.
This was the moment I realized ML can make you feel smart while keeping you stupid.
I built a model that could perfectly organize my users. It just couldn’t tell me which ones would pay.
In machine learning, “good metrics” usually mean “good model.” If your clustering score is 0.85 (out of 1.0), you pop the champagne.
My results:
- Clustering Quality: 0.85 (Excellent)
- Prediction Accuracy: 0.52 (Random)
I had built the world’s best system for categorizing window shoppers.
The Illusion of Success
I trained a neural network to compress user behavior into 32 numbers (embeddings). Then I visualized them.
It was beautiful.
- Cluster A: Users who quit at the “Age” question.
- Cluster B: Users who quit at the “Weight” question.
- Cluster C: Users who read the whole plan.
The model understood the funnel perfectly. It separated users by their journey.
I showed the clusters to the team. Everyone clapped. “Look! We can segment users by behavior!”
Then we checked the revenue.
The “Sneezed Red Dots” Problem
I took that beautiful map and colored the dots by “Purchased: Yes/No.”
I expected to see a “Buyer Island.” A specific region where high-intent users lived.
Reality: It looked like someone sneezed red dots randomly across the map.
Buyers weren’t a cluster. They were a rounding error with a credit card.
- Cluster A had 5% buyers.
- Cluster B had 3% buyers.
- Cluster C had 4% buyers.
That’s noise.
The model successfully separated “people who like apples” from “people who like oranges.” But I was trying to find “people who are hungry.” And hungry people like both.
Why It Failed
Clustering organizes behavior. Purchasing is emotion.
Remember: funnel conversion is 1.5–2.5%. Noise is 98%. Clustering organizes the noise. It doesn’t find the signal.
The model learned what users did.
- “This user clicked 5 times.”
- “This user clicked 50 times.”
It didn’t learn why they did it.
- “This user clicked 5 times because they were bored.”
- “This user clicked 5 times because they were rushing to pay.”
To the model, they looked the same. They landed in the same cluster. But one bought, and one didn’t.
The Lesson
Structure is not signal.
You can have a perfectly organized dataset that contains zero business value. My model was an expert at describing the past. It was clueless about the future.
I didn’t need a librarian. I needed someone who could tell me who’s about to pull out a credit card.
This connects to how one number defeated the neural network—sometimes a simple formula finds the signal that complex clustering misses.