# Catalogue scorecard

Vulnerability-Lookup ranks the known-exploited-vulnerability (KEV) catalogues it holds
by how well they back, situate, lead and maintain the assertions they publish. Every
catalogue conforming to [GCVE-BCP-07](https://gcve.eu/bcp/gcve-bcp-07/) that this
instance imports gets a score, a grade and a breakdown, computed from its own entries
against the rules on this page.

The [assigner scorecard](scorecard.md) asks how well a CNA or GNA fills a record. A
KEV catalogue publishes something else: *assertions*. This vulnerability is
exploited, here is why, since when, where. The catalogue scorecard asks how well each
claim is backed, how early it came, and whether the catalogue is looked after. It
borrows the assigner scorecard's shape (graded checks in four pillars of 25 points, a
rolling window, a volume threshold, a versioned methodology, prepared keys) and
scores different things.

The scorecard is at `/kev-catalogs/scorecard`; each catalogue has a page at
`/kev-catalogs/scorecard/<uuid>` and a badge at `/kev-catalogs/scorecard/<uuid>/badge.svg`,
and every card of the catalogues page carries its grade. The same figures are served
by `/api/stats/scorecard/catalogues` and `/api/stats/scorecard/catalogue`.

## What is scored

A catalogue is scored on the BCP-07 entries it publishes, as this instance holds
them. The catalogue is identified by the UUID every entry carries as its origin; names
come from the GCVE registry's `kev` references and from this instance's own identity.

Four things are deliberate:

- **Evidence is weighed by where it comes from.** An entry whose evidence is a
  honeypot observation the catalogue made itself is not the same entry as one whose
  evidence is "catalogue A lists it". The second is *derivative*. An evidence record is
  derivative when it declares another origin in `evidence.gcve.origin_uuid` (the
  BCP-07 field made for exactly this), or when its source or details name another
  catalogue, whether one this instance holds or one of the aggregators the engine
  knows by name (CISA, KEVIntel, VulnCheck, Shadowserver). A catalogue naming itself
  is never derivative. Derivative evidence earns a quarter of the "evidence present"
  credit and nothing for its type, signal or confidence, which were inherited.
  First-hand evidence (incident response, telemetry, honeypot, sinkhole) and
  independent reports earn the full credit.
- **Templated evidence is discounted.** A catalogue whose every entry carries the
  same type, signal and confidence is not stating a confidence per entry, it is
  applying a label. When one triple covers 95% of a catalogue's evidence records, the
  signal and confidence checks earn at most half. On a Vulnerability-Lookup instance
  the importer-fed catalogues trip this today, because each importer attaches one
  constant evidence record per entry: the figure is shown so that it can be read for
  what it is, and the importers are the place to fix it.
- **Mirrors are inferred, not declared.** When catalogue B holds nearly every entry
  of catalogue A, lists them within days and is almost never first, B mirrors A. The
  two are then not independent witnesses: B's listing of a vulnerability does not
  corroborate A's, and neither competes with the other for the lead. The relation is
  reported with its three figures, so nobody has to take the label on faith.
- **Timeliness is scored.** Unlike a CNA's reservation-to-publication delay, which is
  an embargo as often as a backlog, a catalogue that lists an exploited vulnerability
  later than another simply lists it later.

## The rules

Each rule returns a credit between 0 and 1. The *entry* rules are scored per entry and
averaged over the catalogue's current entries in the period; the *catalogue* rules
are one figure of the catalogue. A catalogue's score is the weighted sum, out of 100.
Four pillars, 25 points each:

**Substantiate** -- is the claim backed, and by whom?

| Rule | Points | Credit |
| --- | --- | --- |
| Evidence | 8 | At least one evidence record. Derivative evidence only, citing another catalogue, earns 0.25. |
| Evidence type | 5 | A typed evidence record that is not derivative; a type of `unknown` earns nothing. |
| Signal | 7 | The strongest signal on non-derivative evidence: `confirmed_compromise` 1.0, `successful_exploitation` 0.9, `in_the_wild_attempts` 0.7, `mass_scanning` 0.5, `weaponized_exploit_available` 0.4. Halved when the catalogue's evidence is templated. |
| Confidence | 5 | A confidence from the BCP-07 table on non-derivative evidence; another number in 0..1 earns 0.6, a label earns 0.5. Halved when the catalogue's evidence is templated. |

**Situate** -- when, where and how bad?

| Rule | Points | Credit |
| --- | --- | --- |
| First seen | 7 | `timestamps.first_seen_at`; `asserted_at` alone earns 0.7. |
| Last seen | 4 | `timestamps.last_seen_at`. |
| Characteristics | 6 | The four characteristics (`remote_code_execution`, `authentication_required`, `local_access_required`, `severity`); the credit is the share stated. |
| Scope | 5 | One of `scope.observation_regions`, `victim_countries`, `sector` or `asset_exposure` earns 0.5; two or more earn 1.0. |
| References | 3 | A reference beyond the vulnerability's own record page; only that page earns 0.4. |

**Lead** -- how early does the catalogue say it? Measured on the vulnerabilities the
catalogue shares with at least one *independent* catalogue: not one it mirrors, not
one mirroring it.

| Rule | Points | Credit |
| --- | --- | --- |
| First to list | 15 | The share of the shared vulnerabilities this catalogue listed first or the same day. |
| Lag | 10 | The median days behind the first lister on the shared vulnerabilities it did not list first: 1.0 at one day or less, falling to nothing at 90 days. |

**Maintain** -- is the catalogue looked after?

| Rule | Points | Credit |
| --- | --- | --- |
| Retractions | 12 | One minus the share of listings withdrawn within 30 days of being listed. |
| Status reason | 8 | The share of current entries whose `status_reason` is not `unknown`. |
| Freshness | 5 | The latest listing: 1.0 within 30 days, falling to nothing at the end of the window. |

Grades from the score: **A** from 90, **B** from 75, **C** from 60, **D** from 45,
**E** below, the same boundaries as the assigner scorecard.

A catalogue sharing no vulnerability with an independent catalogue has no lead to
measure. The pillar is then reported as *not assessable* and the score is taken over
the 75 points that could be assessed, so a catalogue that is alone in its field is
neither punished nor rewarded for it. Every answer states the assessable weight.

### Listing dates

The lead rules need to know when a catalogue listed a vulnerability. BCP-07 has four
timestamps, and this engine reads `asserted_at` first: it is "when an authority or
source officially declared exploitation", which is what a lead measures.
`first_seen_at` is the earliest known exploitation activity, which "may be estimated
and updated retroactively", so a catalogue back-dating it would win every lead. The
fallbacks are `first_seen_at`, `recorded_at`, then `status.status_updated_at`.

The timeline and lead-time sections of the catalogues page read `first_seen_at`
first. That is the right order for "exploited since", which is their question; the
two readings answer different questions and are not meant to agree.

### Mirrors

Catalogue B mirrors catalogue A when all three hold over the current entries:

- B holds at least 90% of A's entries;
- B lists them with a median lag of at most two days after A;
- B is strictly first on at most 5% of them.

A needs the volume threshold's number of entries to be an upstream at all: a
catalogue of three entries is contained by everyone. The three figures travel with
the relation and are reported with it. A smaller catalogue listing a subset of a
larger one the same day cannot be told from that catalogue's source by these figures
alone and is reported as the upstream; in practice a cherry-picker lags, which breaks
the third condition.

The relation is used in two places. For the lead, a mirror and its upstream never
compete: the upstream would win every tie and the mirror would never be first. For
corroboration, a mirror's listing is not an independent witness.

### Retractions

A listing withdrawn within 30 days of being listed was, with hindsight, not
exploited. The share of such listings is the one false-positive figure the data
supports, and it is scored. BCP-07 asks producers to retain a withdrawn assertion as a
tombstone rather than delete it, so a catalogue that withdraws properly is visible
here; one that silently deletes is not, and that is a limit of the figure.

## Ranking, window and threshold

The ranking is built on a **rolling window** of the last 180 days of listings, on the
listing date. A catalogue needs **10 listings in the window** to be ranked; below that
its score is shown and marked unranked, because five entries are not evidence of a
practice. Both figures are options of the preparing pass and are reported with every
answer. Each catalogue also has its all-time figures and its score per listing year.

## Context figures

Shown beside the score and never in it, per catalogue:

- **Provenance of the evidence**: how many entries are backed by first-hand evidence,
  by independent reports only, by derivative evidence only, or by nothing.
- **Unique**: entries no other catalogue lists.
- **Corroborated**: entries an independent catalogue also lists.
- **Led**: entries this catalogue listed before an independent catalogue did. A
  catalogue whose unique entries get adopted elsewhere is doing real work, and this
  is where it shows.
- **Mirrored**: entries also listed by a catalogue in a mirror relation with this one.
- **Templated**: whether the evidence is one constant triple.
- **Mirrors and followers**: the catalogues this one mirrors, and the ones mirroring
  it, with the containment, lag and lead-share figures.
- **Withdrawn and retracted** listings in the period, and the median lag in days.

## The pages and the badge

The scorecard page ranks the catalogues over the window, lists the inferred mirror
relations with their three figures, and restates the rules. A catalogue's page shows
the credit earned on each rule and the points it is worth, the pillars (with "not
assessable" where a pillar could not be measured), the context figures, the
catalogues it mirrors and the ones mirroring it, and its score per listing year.

Each catalogue has an embeddable **badge** showing its grade and score over the
window. It reads "unranked" under the threshold rather than failing, so a page
embedding it does not break the day the catalogue goes quiet. The catalogue's page
gives the Markdown to paste.

## The API

| Endpoint | Answers |
| --- | --- |
| `GET /api/stats/scorecard/catalogues[?period=window\|all\|YYYY][&limit=N][&output=markdown]` | Every catalogue, ranked ones first, best first, with the preparation metadata and the mirror relations. |
| `GET /api/stats/scorecard/catalogue?catalogue=<uuid>` | One catalogue: credit and points per rule, pillars, context figures, mirrors, the per-year trend. |

Like every prepared figure, an unprepared scorecard answers `available: false` rather
than an empty leaderboard, and the methodology carries a version: figures prepared
under older rules are reported as unavailable rather than read against rules they
were not computed with.

## Methodology history

The rules are versioned. Every change to a rule, a weight, a threshold or a grade
boundary is a new version: an entry in `METHODOLOGY` in
`vulnerabilitylookup/kev_scorecard.py`, and a row here (a test keeps the two in step).
The version is stored with the prepared figures, together with the
Vulnerability-Lookup release that prepared them, so a score can always be traced to
the rules it was computed under. Figures prepared under an older version are reported
as unavailable until the next pass rewrites them under the current one.

| Version | Since | Change | Reference |
| --- | --- | --- | --- |
| 1 | 2026-10-11 | Initial rules: fourteen graded checks in four pillars of 25 points; evidence weighed by provenance, templated evidence discounted, mirrors inferred and excluded from lead and corroboration; listing dates read `asserted_at` first; 10 listings over 180 days to be ranked. | [#711](https://github.com/vulnerability-lookup/vulnerability-lookup/pull/711) |

## Preparing the figures

Nothing on the read side walks an entry. The figures are prepared by

```bash
# every catalogue this instance holds entries of, withdrawn entries included
$ poetry run kev_scorecard

# the two knobs every answer reports
$ poetry run kev_scorecard --window-days 180 --min-entries 10
```

The pass reads the KEV tables of the PostgreSQL database and writes the figures to
the storage layer under `stats:kev:scorecard:*` (documented in
`vulnerabilitylookup/kev_scorecard.py`), in one transaction, so a reader mid-run sees
the previous run's figures rather than none. It refuses to overwrite the stored
figures when it finds no entry at all: an empty table is indistinguishable from an
import that has not run yet, and the previous figures are the better answer.

The pass takes seconds on a few thousand entries, so an hourly cron entry is cheap
and keeps the grade on the catalogue cards within an hour of a new import; an
instance whose KEV importers run on a fixed schedule can place it a few minutes after
them instead. The two knobs are the defaults and need no flags:

```text
# m h dom mon dow  command
45 * * * *  cd /path/to/vulnerability-lookup && poetry run kev_scorecard
```

The console script exists once `poetry install` has run, since it is a new entry of
`pyproject.toml`; until then `poetry run python -m website.scripts.kev_scorecard`
runs the same pass. When the pass refuses to overwrite (no entry at all) it exits
non-zero, so a cron line chaining it with other commands should separate them with
`;` rather than `&&`.

## What the importers should carry

The engine can only weigh evidence a catalogue states. Two things make the first
check honest instead of heuristic, and both are the importers' to do:

- an importer republishing another catalogue's listing should set
  `evidence.gcve.origin_uuid` to that catalogue's UUID, which is what the field is
  for;
- an importer that has a per-entry observation (a honeypot's connection counts and
  observation date, for one) should keep carrying it in `evidence.details`, which is
  what tells first-hand evidence from a label.