Command Line Interface#

Run every command from the project directory.

Core#

Services#

$ poetry run start                     # start everything
$ poetry run stop                      # stop everything
$ poetry run run_backend --start       # Valkey and Kvrocks only (--stop to stop)
$ poetry run feeders_manager --start   # all enabled feeders (--stop to stop)
$ poetry run start_website             # the website only
$ poetry run restart_website

feeders_manager --start --reimport makes every feeder reimport its whole source.

Run an individual feeder#

$ poetry run nvd_importer
$ poetry run cvelist_importer
$ poetry run github_importer

The full list is under [project.scripts] in pyproject.toml.

Dumps#

$ poetry run dump --feed nvd                # one source, as JSON
$ poetry run dump --correlations --index    # identifier correlations
$ poetry run euvd_dump                      # CVE ↔ EUVD mapping, as CSV

--index adds the file to the download index. dump --all also writes the correlations.
A failing auxiliary export (comments, bundles, sightings, KEV entries, correlations) is skipped and logged; the command then exits non-zero and names it.
Files are replaced atomically: a failed run keeps the previous export.
Run poetry run dump --index after euvd_dump if the index does not list it yet.

dumps/correlations.json maps each identifier to its directly linked identifiers:

{
  "cve-2024-1234": ["ghsa-abcd-efgh-ijkl", "pysec-2024-1"],
  "ghsa-abcd-efgh-ijkl": ["cve-2024-1234"],
  "pysec-2024-1": ["cve-2024-1234"]
}

Identifiers are lowercased; GCVE-0-<year>-<n> is merged into cve-<year>-<n>.
Self-links and identifiers with no link are left out.
Links are direct, not transitive, and not a claim of equivalence.
It is not guaranteed to be symmetric: it is read from live storage. Add the reverse edges yourself if you need them.

Daily example:

0 7 * * *  cd /path/to/vulnerability-lookup && poetry run dump --correlations --index

Bootstrap from Vulnerability-Lookup dumps#

With Kvrocks running, import the NDJSON files of a trusted instance into a fresh one:

$ poetry run import_dump /path/to/dumps
$ poetry run index_vulnerabilities

The file name (without .ndjson) becomes the source name.
Existing documents are kept; --replace overwrites them. Re-running is safe and never triggers notifications.
Comments, bundles, sightings and KEV entries are ignored. Unknown documents are counted, not fatal.
Dumps without the vulnerability-lookup:id and vulnerability-lookup:score annotations still import, but their documents look modified at import time.

Dumps are a one-time bootstrap, not a sync. Do not schedule dump downloads: an import never updates existing documents, and full downloads cost the publisher bandwidth. Enable the feeders afterwards, and use the API’s since= parameter.

Update the documentation#

$ cd docs; poetry run make html

The API reference comes from docs/_static/files/swagger.json. Regenerate it first with dump_openapi when the API changed.

Web service#

Flask commands are prefixed with poetry run flask --app website.app.

Database#

$ poetry run flask --app website.app db_init
$ poetry run flask --app website.app db_backup                    # also run by `poetry run update`
$ poetry run flask --app website.app db_diagram                   # models diagram
$ poetry run flask --app website.app seed-rbac [--reset-defaults] # also run by `poetry run update`

seed-rbac adds the missing RBAC roles, permissions and policies. --reset-defaults restores the built-in ones, overwriting your changes.

User management#

$ poetry run flask --app website.app create_user --login <login> --email <email> --password <password>
$ poetry run flask --app website.app create_admin --login <login> --email <email> --password <password>
$ poetry run flask --app website.app user_list
$ poetry run flask --app website.app user_delete --login <login>
$ poetry run flask --app website.app user_get_api_key --login <login>

Data management#

$ poetry run flask --app website.app update_warninglists
$ poetry run flask --app website.app update_gcve_registry
$ poetry run flask --app website.app sync_organizations [--login <login>]
$ poetry run python tools/update_cnw_assigners.py
$ poetry run flask --app website.app import_osi_approved_licenses
$ poetry run flask --app website.app import_languages

update_warninglists: MISP warning lists.
update_gcve_registry: local copy of the GCVE registry and the CNA partner catalogue. Can run in cron.
sync_organizations: syncs the organizations with the GCVE registry. --login is the owner, admin by default.
update_cnw_assigners.py: the EU CSIRT coordinated CNAs shown on the dashboard, from enisaeu/CNW (rows whose CNA column is Yes). Can run in cron, or from the admin maintenance page. A failure keeps the previous list.

Sightings#

$ poetry run flask --app website.app clean_sightings --source <source>
$ poetry run flask --app website.app clean_sightings --source '<pattern>' --regex  # PostgreSQL only
$ poetry run flask --app website.app backfill_sightings
$ poetry run flask --app website.app dedup_sightings [--yes]
$ poetry run flask --app website.app retype_sightings [--yes]

clean_sightings: deletes the sightings of a source, or of a matching pattern.
backfill_sightings: creates sightings from existing comments (seen), bundles (seen, one per vulnerability) and KEV entries (exploited), with their original timestamps. Safe to re-run.
dedup_sightings: removes repeated sightings of the same observation (same vulnerability, URL or Telegram/… source, type, author and origin), keeping the oldest. Label sources like CISA-KEV are left alone.
retype_sightings: by default retypes the Exploit-DB sightings pushed as exploited (source starting with https://www.exploit-db.com/exploits/) to published-proof-of-concept, deleting the duplicates. --source-prefix, --from and --to select other sightings. Safe to re-run.

Without --yes, dedup_sightings and retype_sightings are a dry run. Both only touch sightings created on this instance: --all-origins includes synced ones (the next sync may bring them back), --author <login> narrows to one user. Sync never updates a sighting it already holds, so each instance runs them itself.

Move sightings between instances:

$ poetry run flask --app website.app export_sightings sightings.ndjson              # source instance
$ poetry run flask --app website.app import_sightings sightings.ndjson --dry-run    # target: preview
$ poetry run flask --app website.app import_sightings sightings.ndjson --author admin

Both stream, so they handle volumes the browser export times out on. - is stdout/stdin (export_sightings - | gzip).
The format is the one of dumps/sightings.ndjson; the import also reads the browser export.
--local-only exports only the sightings created on this instance.
The import keeps uuid, origin and timestamp, and skips sightings already present. Unknown authors are replaced by --author (admin by default). Invalid records are counted, not fatal.

Regenerate the OpenAPI specification#

$ poetry run flask --app website.app dump_openapi           # write docs/_static/files/swagger.json
$ poetry run flask --app website.app dump_openapi --check   # exit non-zero if it is out of date

No running instance is needed. The output is stable, and the instance UUID is replaced by an example (--example-origin sets another).
It refuses to write unless user_accounts, cna and cpe_enabled are all on, and names the missing ones. --allow-partial writes anyway.
The OpenAPI workflow runs --check on every pull request, with every module on.
A pre-commit hook regenerates it when you touch website/web/api/ or the version in pyproject.toml. It skips, with a reason, on an instance that cannot produce the complete reference. When it regenerates, the commit stops so you can stage the file.
A relative --output is taken from VULNERABILITYLOOKUP_HOME. The command prints the path used.

Health and integrity checks#

$ poetry run flask --app website.app index_health
$ poetry run euvd --integrity-scan

index_health: reports whether the state indexes still match the corpus. A stale index shows no error anywhere else.
euvd --integrity-scan: counts EUVD identifier breaches (several CVEs or none, pointer disagreements, published CVEs without EUVD) for /metrics. Exits non-zero on a breach. Run it daily. See Metrics.

Prepared statistics figures#

The per-source /api/stats/* figures are prepared in advance, not computed per request. A figure never prepared answers available: false. /api/stats/vulnerability/status is always live.

Run these four commands once after installing or upgrading, then in cron:

# m h dom mon dow  command
3 * * *  cd /path/to/vulnerability-lookup && poetry run index_vulnerabilities
4 * * *  cd /path/to/vulnerability-lookup && poetry run euvd --stats-scan --stats-source cvelistv5
* * * *  cd /path/to/vulnerability-lookup && poetry run scorecard --window-only
* * * *  cd /path/to/vulnerability-lookup && poetry run kev_scorecard

All four are idempotent.

Source names#

A source (--source, --stats-source, ?source=) is the name records are stored under, not the feeder’s section in config/modules.cfg. For example, [feeder:cvelist] stores cvelistv5. List them with:

$ redis-cli -p 10002 HKEYS last_updates

GNA sources are the gna-<id> entries, one per GNA whose records this instance pulls. A GNA with neither API nor dump (such as GNA 2, EUVD) never becomes a source.
The euvd source holds EUVD records, which is unrelated to GNA 2. Official EUVD numbers are issued only by ENISA’s instance, by a component that is not part of Vulnerability-Lookup, so no other instance holds euvd records of its own.

The reindex: counts, rankings and CVSS distributions#

$ poetry run index_vulnerabilities                                  # every source, GNAs included
$ poetry run index_vulnerabilities --source cvelistv5 --source gna-1

Derives the counts per year and status, the top vendors and assigners and the CVSS distributions, and runs the full assigner scorecard.
Without --source, it covers every source in last_updates. Name sources to make the nightly run shorter.
Run it at least once on a new, restored or upgraded instance before trusting any figure: a stale index looks normal but is wrong.

The scorecards#

$ poetry run scorecard                # assigner scorecard, whole history (also run by every reindex)
$ poetry run scorecard --window-only  # only the rolling window: cheap, hourly
$ poetry run kev_scorecard            # KEV catalogue scorecard, seconds

See assigner scorecard and catalogue scorecard. Without the hourly scorecard --window-only, the pages show what the last reindex prepared. The wall of shame has its own switch, WEB_MODULES["wall_of_shame"] (off by default).

Upgrading from the EUVD-only figures#

The figures used to be keyed euvd:stats:*; they are now stats:<source>:*. Run once after upgrading:

$ poetry run euvd --migrate-stats-keys

It is idempotent and does nothing on an instance that never held EUVD figures. Without it, the time-to-exploit snapshot stays available: false.

Background services#

$ poetry run flask --app website.app notify_users   # notifications
$ poetry run flask --app website.app sync           # synchronization

See Notifications and Synchronization.

Command Line Interface

Contents