Entity resolution is the hard part of esports data

Bottom line: Fetching esports match data is a solved problem. Deciding that NAVI, Natus Vincere and Na'Vi Junior are two organisations and not one — and keeping that decision stable across nearly six years and 1,495 team entities — is where the work actually is. This is how our pipeline handles it, and where it still fails.

Every "build an esports data pipeline" post spends its length on the fetching. Fetching is the easy half. The moment you hold records describing the same match from more than one vantage point, you own an entity-resolution problem, and it does not go away. It has a name because it is everywhere: the UK Ministry of Justice maintains an open-source library, Splink, for exactly this class of record linkage. A library narrows the candidates. It still cannot tell you whether an academy side is its parent.

The problem, concretely

A professional Counter-Strike organisation is not a stable identifier. Our pipeline currently holds 4,633 players, 1,495 team entities and 876 tournaments distinct and stable across 22,705 matches and nearly six years of history, and every one of those is a decision that has to survive the next ingestion run without drifting.

Over that same span, teams have:

• renamed (sponsorship changes, rebrands);
• been acquired, absorbing an existing roster wholesale;
• fielded academy sides whose names are the parent's name plus a suffix;
• appeared under regional or transliterated variants of the same name;
• disbanded and re-formed with an overlapping roster and a new brand.

Only the third and fifth of those should create a new entity. The others should not. Get it wrong in one direction and a team's history fractures across three rows; get it wrong in the other and you have merged an academy side's results into its parent's.

The same problem exists for players, in miniature: nicknames change, and the same person appears under two or three handles across the years we cover.

What we actually do

Three mechanisms, in order of how much work they do:

1. A stable internal ID, assigned once. Every team, player and tournament gets a UUID at first sight, and that ID never changes. Names are attributes of an entity, not the identity of one. This sounds obvious. It is the single decision that makes everything downstream tractable, and it is the one most quick pipelines skip because a name is right there and looks unique.

2. An alias table, not a rename. When a team rebrands, we do not overwrite the name — we record the new one and keep the old as an alias pointing at the same UUID. Queries against either name land on the same entity, and renames become additive rather than destructive. This is not an abstract nicety: it is what keeps the 6,470 CS matches in our corpus queryable at all. A match played in 2021 under a name the org has since dropped still resolves to that org today, so its pre-CS2 history sits alongside its current results instead of stranded under a dead label. Lose the alias and more than a quarter of the corpus goes dark to anyone searching by current names.

3. Idempotent, convergent writes. Ingestion never assumes an empty table and never assumes it is running for the first time. Every upsert is keyed on a natural identifier and produces the same end state whether it runs once or twenty times:

-- Illustrative, not our real schema: an upsert keyed on a stable natural key.
-- Run it once or twenty times and the row converges to the same final state.
INSERT INTO teams (id, natural_key, display_name, updated_at)
VALUES (:uuid, :natural_key, :display_name, now())
ON CONFLICT (natural_key) DO UPDATE
  SET display_name = EXCLUDED.display_name,
      updated_at   = now();

The natural key is the anchor: a new name updates the display, it never spawns a second row. That is the whole trick, and it is why a re-run after a partial failure is safe, which means we can re-run freely, which means backfills are cheap.

Where it still fails

Honesty is more useful than a clean narrative, so:

• Academy sides remain the worst case. A suffix convention is not universal, and some organisations field a secondary roster under a wholly unrelated brand. These need a human decision, and we make them by hand.
• Disband-and-reform is genuinely ambiguous. When four of five players carry to a new organisation, whether that is continuity or a new entity is a judgement, not a fact. We treat the organisation as the entity, which means roster continuity does not merge two brands. Reasonable people would choose differently.
• Reconciling depth across events. Per-round detail exists for the events that were covered closely and not for the rest. We do not synthesise it. A gap stays a gap, because an interpolated round is indistinguishable from a real one once it is in the table, and that is a lie you cannot take back.

The lesson

The interesting engineering in a data product is almost never the ingestion. It is the schema decisions that let you keep the data coherent as reality churns underneath it: stable IDs, additive aliases, idempotent writes, and a refusal to invent rows you do not have.

The result of all that is 22,705 completed matches, 4,633 players and 1,495 teams that stay stable across queries, described in more depth in what the corpus actually looks like. Where each of those figures comes from, and how we define "completed" and "stable", lives in our methodology. If you would rather not build any of this, the CS2 data API hands you the resolved version: entities already merged, aliases already followed, and IDs that will not move under you.