A machine learning model is trained by showing it examples and letting it find the patterns that predict an outcome. A model that fits its training examples almost perfectly sounds like a good thing, until it's tested on new data it hasn't seen and performs noticeably worse than a less precise model would have. That's overfitting: the model learned the noise in its training data, quirks and coincidences specific to those exact examples, and mistook it for signal.

An ICP built purely by profiling your best existing customers is vulnerable to the same failure, for exactly the same structural reason.

What overfitting actually means, and why it's not the same as being wrong

An overfit model isn't wrong about its training data, it's often extremely accurate there. The problem only shows up on new data, because some of what it "learned" wasn't a genuine pattern at all, it was coincidence that happened to be present in that specific training set and nowhere else.

This is a subtler failure than simply building a bad model. A bad model is wrong everywhere. An overfit model looks impressively right on the data you built it from, which is exactly what makes the mistake hard to notice until it's tested against something new.

The one-sentence distinction

Overfitting isn't a model that's wrong. It's a model that mistook coincidence for cause, and the mistake is invisible until it meets data it wasn't built from.

How an ICP overfits to its best historical customers

Building an ICP by profiling your best 10 or 20 existing customers is standard, sensible practice. The risk appears when every shared trait among those accounts gets treated as if it were causally important, rather than checking which traits actually explain why those customers succeeded.

Overfitting concept
ICP-building equivalent
Training data
Your existing best-fit customer accounts, the ones the ICP is being built to describe.
Noise mistaken for signal
Coincidental shared traits, a specific CRM in their tech stack, a shared region, all being closed by the same rep, that had nothing to do with why those accounts actually succeeded.
Poor performance on new data
A genuinely great-fit prospect gets scored out of the ICP for failing to match a criterion that was never actually causal in the first place.

The signs an ICP has been overfit

The pattern is recognisable once you know what to look for, and it shows up in how the ICP behaves against new prospects, not just how it reads on paper.

1
It reads more like a description of specific companies than a category

A genuinely useful ICP should describe a category of company you haven't necessarily encountered yet. If the criteria list feels like it could only have been written by looking at the exact logos already in your customer base, it likely has been.

2
Good-fit prospects keep getting scored out for unexplainable reasons

If sales keeps flagging accounts that feel like an obvious fit but fail a specific ICP criterion, and nobody can articulate why that criterion should matter causally, that's a strong sign the criterion was noise from the original cohort, not genuine signal.

3
The definition hasn't been revisited since it was first built

A model trained once and never retrained gets worse as the world it's predicting drifts away from its training data. An ICP is no different, a definition frozen at its original best-customer snapshot will fit the current market progressively worse over time.

Building an ICP that generalises, not just memorises

The fix borrows directly from how overfitting is addressed in machine learning: test against data the model wasn't built from, and keep the model simpler than maximum precision would allow.

How to build and test an ICP that generalises
  • Hold out a validation set of accounts: Build the ICP from most of your best customers, then check whether it correctly identifies the remaining ones you deliberately left out. An ICP that fails this test was likely fit to noise in the original group.
  • Keep the criteria list short and causally justified: Every criterion should have an explainable reason it predicts success, not just a statistical correlation within your current customer set. Fewer, well-justified criteria generalise better than many maximally precise ones.
  • Test explicitly against near-miss prospects: Look at accounts that were disqualified narrowly, and check whether sales genuinely believes they were a poor fit or whether the ICP excluded them on a technicality.
  • Revisit and retest on a schedule, not just when pipeline feels off: Treat the ICP the way a data team retrains a model, periodically, using current data, rather than treating the first version as a permanent definition.
Key takeaways
  • Overfitting happens when a model, or an ICP, learns coincidence from its training data as if it were genuine signal
  • An ICP built purely from your best existing customers risks encoding incidental shared traits as if they were causally important
  • Warning signs include an ICP that reads like a list of specific companies, unexplainable disqualifications, and a definition nobody has revisited
  • Validate against held-out accounts and keep the criteria list short and causally justified, then retest periodically as the market shifts
Build an ICP that holds up against new prospects, not just old ones

Our GTM Audit stress-tests your ICP against genuinely new accounts, not just the customer base it was built from, so it's identifying your next best-fit customer rather than just describing the last one.

Read Building Your ICP →