User Intent Prediction #7: Backward and Forward. More Work. Worse Results.
When nothing worked, I did what every engineer does: I added more complexity.
I thought looking at the past wasn’t enough. sequence model failed, sequence model failed, so obviously… I built something twice as stupid.
I thought: “What if the model could see the whole session at once? Backward and forward?”
So I trained a Bidirectional sequence model (bidirectional sequence model). It reads the sequence from start to end, AND from end to start.
The Theory: It understands the “full context” of the user’s journey. The Reality: It doubled my cloud bill and gave me nothing.
The Experiment
| Model | Parameters | Training Time | Prediction Accuracy |
|---|---|---|---|
| sequence model | 160K | 1.0x | 0.52 |
| bidirectional sequence model | 330K | 2.3x | 0.57 |
I doubled the parameters. I doubled the training time. I wasn’t buying insight. I was burning compute.
And 0.57 is the statistical equivalent of ¯_(ツ)_/¯.
The “Time Travel” Fallacy
Bisequence models work great for text. In the sentence “The bank of the river,” knowing “river” helps you understand “bank.” You need to look forward to understand the past.
But funnels aren’t sentences. Time flows one way. Users don’t live their sessions backward. They don’t teleport from Screen 20 back to Screen 3 to “add context.”
A user at Screen 10 doesn’t know what happens at Screen 20. Looking backward from the future (Screen 20) to the past (Screen 10) adds noise, not signal.
I was trying to predict if a user would buy. The bidirectional sequence model was cheating by looking at the end of the session. And even with cheating, it barely worked.
The Lesson
Complexity is not a virtue.
I added complexity because I was desperate for a result. “Maybe if I make it smarter, it will work.”
But a smarter model on the wrong data is just a more expensive random number generator.
Time flows forward. Users flow forward. My model was the only idiot going backwards.
This leads to trying to predict two things at once—if one task fails, maybe two will succeed? (Spoiler: No).