Catalogue scorecard#

Vulnerability-Lookup ranks the known-exploited-vulnerability (KEV) catalogues it holds by how well they back, situate, lead and maintain the assertions they publish. Every catalogue conforming to GCVE-BCP-07 that this instance imports gets a score, a grade and a breakdown, computed from its own entries against the rules on this page.

The assigner scorecard asks how well a CNA or GNA fills a record. A KEV catalogue publishes something else: assertions. This vulnerability is exploited, here is why, since when, where. The catalogue scorecard asks how well each claim is backed, how early it came, and whether the catalogue is looked after. It borrows the assigner scorecard’s shape (graded checks in four pillars of 25 points, a rolling window, a volume threshold, a versioned methodology, prepared keys) and scores different things.

The scorecard is at /kev-catalogs/scorecard; each catalogue has a page at /kev-catalogs/scorecard/<uuid> and a badge at /kev-catalogs/scorecard/<uuid>/badge.svg, and every card of the catalogues page carries its grade. The same figures are served by /api/stats/scorecard/catalogues and /api/stats/scorecard/catalogue.

What is scored#

A catalogue is scored on the BCP-07 entries it publishes, as this instance holds them. The catalogue is identified by the UUID every entry carries as its origin; names come from the GCVE registry’s kev references and from this instance’s own identity.

Four things are deliberate:

The rules#

Each rule returns a credit between 0 and 1. The entry rules are scored per entry and averaged over the catalogue’s current entries in the period; the catalogue rules are one figure of the catalogue. A catalogue’s score is the weighted sum, out of 100. Four pillars, 25 points each:

Substantiate – is the claim backed, and by whom?

Rule	Points	Credit
Evidence	8	At least one evidence record. Derivative evidence only, citing another catalogue, earns 0.25.
Evidence type	5	A typed evidence record that is not derivative; a type of `unknown` earns nothing.
Signal	7	The strongest signal on non-derivative evidence: `confirmed_compromise` 1.0, `successful_exploitation` 0.9, `in_the_wild_attempts` 0.7, `mass_scanning` 0.5, `weaponized_exploit_available` 0.4. Halved when the catalogue’s evidence is templated.
Confidence	5	A confidence from the BCP-07 table on non-derivative evidence; another number in 0..1 earns 0.6, a label earns 0.5. Halved when the catalogue’s evidence is templated.

Situate – when, where and how bad?

Rule	Points	Credit
First seen	7	`timestamps.first_seen_at`; `asserted_at` alone earns 0.7.
Last seen	4	`timestamps.last_seen_at`.
Characteristics	6	The four characteristics (`remote_code_execution`, `authentication_required`, `local_access_required`, `severity`); the credit is the share stated.
Scope	5	One of `scope.observation_regions`, `victim_countries`, `sector` or `asset_exposure` earns 0.5; two or more earn 1.0.
References	3	A reference beyond the vulnerability’s own record page; only that page earns 0.4.

Lead – how early does the catalogue say it? Measured on the vulnerabilities the catalogue shares with at least one independent catalogue: not one it mirrors, not one mirroring it.

Rule	Points	Credit
First to list	15	The share of the shared vulnerabilities this catalogue listed first or the same day.
Lag	10	The median days behind the first lister on the shared vulnerabilities it did not list first: 1.0 at one day or less, falling to nothing at 90 days.

Maintain – is the catalogue looked after?

Rule	Points	Credit
Retractions	12	One minus the share of listings withdrawn within 30 days of being listed.
Status reason	8	The share of current entries whose `status_reason` is not `unknown`.
Freshness	5	The latest listing: 1.0 within 30 days, falling to nothing at the end of the window.

Grades from the score: A from 90, B from 75, C from 60, D from 45, E below, the same boundaries as the assigner scorecard.

A catalogue sharing no vulnerability with an independent catalogue has no lead to measure. The pillar is then reported as not assessable and the score is taken over the 75 points that could be assessed, so a catalogue that is alone in its field is neither punished nor rewarded for it. Every answer states the assessable weight.

Listing dates#

The lead rules need to know when a catalogue listed a vulnerability. BCP-07 has four timestamps, and this engine reads asserted_at first: it is “when an authority or source officially declared exploitation”, which is what a lead measures. first_seen_at is the earliest known exploitation activity, which “may be estimated and updated retroactively”, so a catalogue back-dating it would win every lead. The fallbacks are first_seen_at, recorded_at, then status.status_updated_at.

The timeline and lead-time sections of the catalogues page read first_seen_at first. That is the right order for “exploited since”, which is their question; the two readings answer different questions and are not meant to agree.

Mirrors#

Catalogue B mirrors catalogue A when all three hold over the current entries:

B holds at least 90% of A’s entries;
B lists them with a median lag of at most two days after A;
B is strictly first on at most 5% of them.

A needs the volume threshold’s number of entries to be an upstream at all: a catalogue of three entries is contained by everyone. The three figures travel with the relation and are reported with it. A smaller catalogue listing a subset of a larger one the same day cannot be told from that catalogue’s source by these figures alone and is reported as the upstream; in practice a cherry-picker lags, which breaks the third condition.

The relation is used in two places. For the lead, a mirror and its upstream never compete: the upstream would win every tie and the mirror would never be first. For corroboration, a mirror’s listing is not an independent witness.

Retractions#

A listing withdrawn within 30 days of being listed was, with hindsight, not exploited. The share of such listings is the one false-positive figure the data supports, and it is scored. BCP-07 asks producers to retain a withdrawn assertion as a tombstone rather than delete it, so a catalogue that withdraws properly is visible here; one that silently deletes is not, and that is a limit of the figure.

Ranking, window and threshold#

The ranking is built on a rolling window of the last 180 days of listings, on the listing date. A catalogue needs 10 listings in the window to be ranked; below that its score is shown and marked unranked, because five entries are not evidence of a practice. Both figures are options of the preparing pass and are reported with every answer. Each catalogue also has its all-time figures and its score per listing year.

Context figures#

Shown beside the score and never in it, per catalogue:

Provenance of the evidence: how many entries are backed by first-hand evidence, by independent reports only, by derivative evidence only, or by nothing.
Unique: entries no other catalogue lists.
Corroborated: entries an independent catalogue also lists.
Led: entries this catalogue listed before an independent catalogue did. A catalogue whose unique entries get adopted elsewhere is doing real work, and this is where it shows.
Mirrored: entries also listed by a catalogue in a mirror relation with this one.
Templated: whether the evidence is one constant triple.
Mirrors and followers: the catalogues this one mirrors, and the ones mirroring it, with the containment, lag and lead-share figures.
Withdrawn and retracted listings in the period, and the median lag in days.

The pages and the badge#

The scorecard page ranks the catalogues over the window, lists the inferred mirror relations with their three figures, and restates the rules. A catalogue’s page shows the credit earned on each rule and the points it is worth, the pillars (with “not assessable” where a pillar could not be measured), the context figures, the catalogues it mirrors and the ones mirroring it, and its score per listing year.

Each catalogue has an embeddable badge showing its grade and score over the window. It reads “unranked” under the threshold rather than failing, so a page embedding it does not break the day the catalogue goes quiet. The catalogue’s page gives the Markdown to paste.

The API#

Endpoint	Answers
`GET /api/stats/scorecard/catalogues[?period=window\|all\|YYYY][&limit=N][&output=markdown]`	Every catalogue, ranked ones first, best first, with the preparation metadata and the mirror relations.
`GET /api/stats/scorecard/catalogue?catalogue=<uuid>`	One catalogue: credit and points per rule, pillars, context figures, mirrors, the per-year trend.

Like every prepared figure, an unprepared scorecard answers available: false rather than an empty leaderboard, and the methodology carries a version: figures prepared under older rules are reported as unavailable rather than read against rules they were not computed with.

Methodology history#

The rules are versioned. Every change to a rule, a weight, a threshold or a grade boundary is a new version: an entry in METHODOLOGY in vulnerabilitylookup/kev_scorecard.py, and a row here (a test keeps the two in step). The version is stored with the prepared figures, together with the Vulnerability-Lookup release that prepared them, so a score can always be traced to the rules it was computed under. Figures prepared under an older version are reported as unavailable until the next pass rewrites them under the current one.

Version	Since	Change	Reference
1	2026-10-11	Initial rules: fourteen graded checks in four pillars of 25 points; evidence weighed by provenance, templated evidence discounted, mirrors inferred and excluded from lead and corroboration; listing dates read `asserted_at` first; 10 listings over 180 days to be ranked.	#711

Preparing the figures#

Nothing on the read side walks an entry. The figures are prepared by

# every catalogue this instance holds entries of, withdrawn entries included
$ poetry run kev_scorecard

# the two knobs every answer reports
$ poetry run kev_scorecard --window-days 180 --min-entries 10

The pass reads the KEV tables of the PostgreSQL database and writes the figures to the storage layer under stats:kev:scorecard:* (documented in vulnerabilitylookup/kev_scorecard.py), in one transaction, so a reader mid-run sees the previous run’s figures rather than none. It refuses to overwrite the stored figures when it finds no entry at all: an empty table is indistinguishable from an import that has not run yet, and the previous figures are the better answer.

The pass takes seconds on a few thousand entries, so an hourly cron entry is cheap and keeps the grade on the catalogue cards within an hour of a new import; an instance whose KEV importers run on a fixed schedule can place it a few minutes after them instead. The two knobs are the defaults and need no flags:

# m h dom mon dow  command
45 * * * *  cd /path/to/vulnerability-lookup && poetry run kev_scorecard

The console script exists once poetry install has run, since it is a new entry of pyproject.toml; until then poetry run python -m website.scripts.kev_scorecard runs the same pass. When the pass refuses to overwrite (no entry at all) it exits non-zero, so a cron line chaining it with other commands should separate them with ; rather than &&.

What the importers should carry#

The engine can only weigh evidence a catalogue states. Two things make the first check honest instead of heuristic, and both are the importers’ to do:

an importer republishing another catalogue’s listing should set evidence.gcve.origin_uuid to that catalogue’s UUID, which is what the field is for;
an importer that has a per-entry observation (a honeypot’s connection counts and observation date, for one) should keep carrying it in evidence.details, which is what tells first-hand evidence from a label.

Catalogue scorecard

Contents