User Intent Prediction #1: 32 Dimensions to 8. Everything Broke.
32 dimensions of pure behavioral beauty. And absolutely zero business value.
I had embeddings that described my users perfectly. They captured every click, every pause, every hesitation in a brutal 60-screen weight-loss funnel.
Clustering was beautiful. The model could tell apart users who quit at “Age” from users who quit at “Weight.” Prediction was useless. 0.52 prediction accuracy.
I didn’t want beautiful embeddings. I wanted revenue.
So I thought: “Maybe there’s too much noise. Let’s compress it.”
I tried PCA. I tried t-SNE. I tried UMAP. I wanted to distill the essence of the user.
Result: Everything got worse.
The Experiment
I took the 32-dimensional embeddings and squeezed them down to 8 dimensions using PCA.
The Logic: PCA keeps the dimensions with the highest variance. It throws away the noise.
The Reality:
| Feature | Prediction Accuracy |
|---|---|
| 32D Embeddings | 0.52 |
| PCA → 8D | 0.51 |
| motivation_score_v1 | 0.96 |
PCA worked exactly as advertised. It kept 95% of the variance. It just threw away the only signal that mattered.
Variance ≠ Intent
In a funnel with 1.5% conversion, the “main” behavior is not buying. 98.5% of users drop out. Their behavior dominates the dataset.
Weight-loss funnels are chaos. People click like they’re drunk.
- They skip the quiz.
- They read the intro.
- They click back.
PCA sees this chaos and says: “This is important! This is where the variance is!”
But the purchase signal—the tiny hesitation of a desperate user—is a microscopic fraction of the total variance. To PCA, that signal looks like noise. So it deleted it.
The Pretty Pictures Trap
I ran t-SNE to visualize the embeddings. It made stunning pictures. Distinct islands of users. Beautiful gradients.
I showed them to the team. “Look, we can see the user segments!”
Then I colored the dots by “Purchased.” It was a mess. Buyers were scattered everywhere.
t-SNE made beautiful islands. Too bad buyers didn’t live on any of them.
The Lesson
Intent is not variance.
I spent weeks acting like an ML engineer, optimizing for “information retention.” I should have been acting like a founder, looking for the 1.5% who pay.
If the signal isn’t in 32 dimensions, it won’t magically appear in 8.
Compression doesn’t fix missing intent. It just shrinks the failure.
This leads to how one number defeated the neural network—sometimes you need to stop compressing and start engineering.