Command Line Interface

This section describes the various commands available.

All commands below are run from the Vulnerability-Lookup project directory using poetry run.

Core

Start all the services

$ poetry run start

Stop all the services

$ poetry run stop

Start/stop the backend (Valkey and Kvrocks)

$ poetry run run_backend --start
$ poetry run run_backend --stop

Start/stop all enabled feeders

$ poetry run feeders_manager --start
$ poetry run feeders_manager --stop

Start only the website

$ poetry run start_website

Restart only the website

$ poetry run restart_website

Dump a source in a JSON file

$ poetry run dump --feed nvd

Bootstrap from Vulnerability-Lookup dumps

Download the NDJSON files published by a trusted Vulnerability-Lookup instance (for example, the CIRCL dump directory), then import the directory into a fresh instance while Kvrocks is running:

$ poetry run import_dump /path/to/dumps
$ poetry run index_vulnerabilities

The filename (without .ndjson) becomes the source name. Existing vulnerability documents are never overwritten by default — a skipped document only gains metadata and source-index membership it does not have yet, so the command is safe to resume and re-running it never re-triggers notifications. Pass --replace to replace existing documents. Collaborative SQL exports (comments, bundles, sightings, and KEV entries) are ignored, documents whose shape is not recognized are counted and reported without aborting the file, and a failing file does not block the remaining dumps.

Dumps produced by the current dump command annotate each line with the physical storage key and index score (vulnerability-lookup:id and vulnerability-lookup:score), which the importer uses to rebuild storage faithfully. Older dumps without these annotations still import, with the identifier derived from the document content and the file’s modification time used as the index score — freshness-based features (recent listings, since= queries) will then treat those documents as modified at import time.

After bootstrapping, enable and run the normal feeders to receive regular updates. Dumps are intended for one-time initial bootstrap, not periodic synchronization: do not schedule recurring dump downloads and imports (a daily cron, for instance). Without --replace an import never updates a document that already exists, so a polled corpus silently goes stale — and repeatedly fetching full dumps shifts significant bandwidth costs onto the publisher, who may rate-limit or block clients seen doing it. The feeders and the API’s since= parameter exist precisely so nobody has to re-download the whole dataset.

Run an individual feeder manually

$ poetry run nvd_importer
$ poetry run cvelist_importer
$ poetry run github_importer

See pyproject.toml under [project.scripts] for the full list of available feeder commands.

Update the documentation

$ cd docs; poetry run make html

The build reads the API reference from docs/_static/files/swagger.json, which is not written by hand – regenerate it first with dump_openapi whenever the API changed.

Web service

This section describes the main commands related to the web service. All Flask commands are prefixed with poetry run flask --app website.app.

Database

Init the database:

$ poetry run flask --app website.app db_init

Backup the PostgreSQL database (also runs automatically during poetry run update):

$ poetry run flask --app website.app db_backup

Generate a database models diagram:

$ poetry run flask --app website.app db_diagram

User management

Create a user:

$ poetry run flask --app website.app create_user --login <login> --email <email> --password <password>

Create an admin:

$ poetry run flask --app website.app create_admin --login <login> --email <email> --password <password>

List all users:

$ poetry run flask --app website.app user_list

Delete a user:

$ poetry run flask --app website.app user_delete --login <login>

Retrieve a user’s API key:

$ poetry run flask --app website.app user_get_api_key --login <login>

Data management

Update MISP warning lists (the administrator will be prompted to run this during updates):

$ poetry run flask --app website.app update_warninglists

Update the local copy of the GCVE registry (can be scheduled via cron):

$ poetry run flask --app website.app update_gcve_registry

Import OSI-approved licenses:

$ poetry run flask --app website.app import_osi_approved_licenses

Import source code languages:

$ poetry run flask --app website.app import_languages

Clean sightings by source or pattern:

$ poetry run flask --app website.app clean_sightings

Backfill sightings from existing data:

$ poetry run flask --app website.app backfill_sightings

Creates sightings retroactively from all existing comments (seen), bundles (seen, one per vulnerability), and KEV entries (exploited). Each sighting inherits the creation_timestamp of its source object. The command is idempotent: running it multiple times will not create duplicates, as it skips any entry whose source URL already exists in the database.

Regenerate the OpenAPI specification

The API reference published in the documentation is a snapshot of the specification, committed at docs/_static/files/swagger.json. This command rewrites it from the API object itself, so no instance has to be running and the result cannot be an older deployment’s:

$ poetry run flask --app website.app dump_openapi

Two things make this preferable to curling /api/swagger.json off a live instance. The output is stable: flask-restx serialises the methods of an endpoint out of a set, so two dumps of an unchanged API otherwise differ by well over a thousand lines of reordering, and the diff of a real change is unreadable. And the instance UUID that reaches the schema as the documented default of every vulnerability_lookup_origin field is replaced by a fixed example, so the published reference does not advertise whichever instance happened to produce it (--example-origin chooses another).

The specification only covers what the instance exposes: the namespaces behind user_accounts, cna and cpe_enabled are registered at import time, so an instance with one of them off has no routes to document for it. The shipped samples alone are three paths short (/cna_credential/, /cna_credential/{credential_id} and /cpe_editor/status/{vulnerability_id}). Rather than quietly publishing the smaller file, the command refuses to write and names the modules to turn on; pass --allow-partial if you really do want only what this instance exposes.

--check writes nothing and exits non-zero when the committed file is not what the current code would produce:

$ poetry run flask --app website.app dump_openapi --check

That is the form the OpenAPI workflow runs on every pull request, which is what keeps the snapshot honest. It needs no storage, cache or database – the specification comes from the API object, which touches none of them – only the samples with every optional module enabled, the profile .github/workflows/openapi.yml writes before checking. Regenerating from a full instance reproduces that byte for byte.

A pre-commit hook regenerates the snapshot for you when you touch website/web/api/ or bump the version in pyproject.toml (tools/regenerate_openapi_snapshot.py). It cannot be the gate – it runs on whoever is committing, with whatever modules they happen to have enabled, and pre-commit install is a choice each contributor makes – so it is built to step aside rather than get it wrong: it checks that the instance can produce the complete reference before doing anything, and skips with a one-line reason when it cannot. What it does not do is pass quietly when the generator is broken; that fails, with the error. When it does regenerate, the commit stops so you can stage the result and look at it.

Note

A relative --output is taken from VULNERABILITYLOOKUP_HOME: importing the application changes the working directory. The command prints the path it actually used.

Health and integrity checks

Report whether the derived state indexes still account for the corpus. Nothing else surfaces a stale index: every read path that trusts one is quietly wrong while the UI, the API and the logs look normal.

$ poetry run flask --app website.app index_health

Count EUVD identifier invariant breaches — records carrying several CVEs or none, pointer disagreements, and published CVEs with no EUVD — and store the result for the /metrics endpoint to report. Exits non-zero when an invariant is broken, so it can be scheduled and left to speak up:

$ poetry run euvd --integrity-scan

It walks the whole identifier space, so run it on a schedule (daily is ample) rather than expecting the metrics endpoint to compute it per scrape. See Metrics.

Prepared statistics figures

The EUVD statistics surfaces (the public statistics page and the analytics dashboard) read figures that are prepared in advance rather than recomputed per visitor: at ~380k records, counting severities or scanning the known-exploited catalogue on request would not answer in a page load.

Two commands produce them, and both are meant to run on a schedule:

# counts per year and status, top vendors, top assigners, and the CVSS
# distribution -- all derived from the stored records
$ poetry run index_vulnerabilities --source euvd

# average time to exploit -- spans PostgreSQL (KEV listing dates) and Kvrocks
# (publication dates), so it is computed as a job and stored as a snapshot
$ poetry run euvd_seed_counters --stats-scan

--stats-scan refuses to overwrite the stored figure when it finds no exploited KEV entries at all: an empty catalogue table is indistinguishable from an import that has not run yet, and the previous snapshot is the better answer.

Whatever the cadence, run the reindex at least once before trusting any counter on a new or restored instance: VL’s derived indexes are rebuilt from the stored records, and a stale index makes a “by status” figure report a fraction of reality while looking entirely normal.

Background services

Launch the email notification service:

$ poetry run flask --app website.app notify_users

Launch the synchronization service (see Synchronization service for details):

$ poetry run flask --app website.app sync