Performance Tuning#

Settings for each layer of a production deployment: Gunicorn, Kvrocks, Valkey/Redis, PostgreSQL, and optionally PgBouncer.

Production Reference Architecture#

The topology of the CIRCL/GCVE production instances, a good reference for high traffic:

        flowchart LR
    Client(["Client<br/>HTTP/2 + TLS"])
    HA["HAProxy<br/>TLS termination<br/>L4/L7 load balancing"]
    V["Varnish<br/>HTTP reverse-proxy cache"]
    NGX_D["Nginx<br/>open-data dumps"]
    NGX_S["Nginx<br/>static-asset CDN"]
    VL["Vulnerability-Lookup<br/>Gunicorn + Flask"]
    PGB["PgBouncer<br/>connection pooling"]
    PG[("PostgreSQL")]
    KV[("Kvrocks<br/>primary storage")]
    VAL[("Valkey")]
    DUMPS[("Daily Kvrocks<br/>NDJSON dumps")]
    STATIC[("Flask /static/<br/>CSS · JS · images")]

    Client -->|HTTPS| HA
    HA -->|HTTP/2| V
    HA -->|"HTTP/2<br/>/dumps/"| NGX_D
    HA -->|"HTTP/2<br/>/static"| NGX_S
    V -->|HTTP/1.1| VL
    NGX_D --> DUMPS
    NGX_S --> STATIC
    VL -->|SQL| PGB
    PGB --> PG
    VL -->|RESP| KV
    KV -.->|replication| VAL

Compared to the Nginx-only setup, it adds HAProxy, Varnish and two Nginx instances (dumps under /dumps/, static assets under /static).

HAProxy (TLS termination and load balancing)#

HAProxy is the only component exposed to clients. It:

terminates TLS and speaks HTTP/2 to clients and to Varnish;
balances across several Varnish or application instances, with health checks;
keeps long-lived connections to the backends and absorbs connection bursts;
can pass the client IP with the PROXY protocol (Gunicorn runs with --proxy-protocol).

Example configuration#

Three backends: the cache (Varnish, the application as backup), the dumps, and the static assets.

frontend web
  mode http
  bind [::]:80 accept-proxy
  bind [::]:443 ssl crt /etc/haproxy/certs/ strict-sni accept-proxy

  # clean header
  http-request del-header X-Forwarded-Proto
  http-request del-header X-Forwarded-For

  # redirect to https
  http-request redirect scheme https code 308 unless { ssl_fc }
  http-request add-header X-Forwarded-Proto https if { ssl_fc } # tell the backend we are using https

  default_backend cache
  acl vulnerability hdr(host) -i vulnerability.circl.lu
  acl vulnerability-dump path_beg /dumps/
  acl vulnerability-cdn path_beg /static/

  use_backend vulnerability-cdn if vulnerability-cdn
  use_backend vulnerability-dump if vulnerability-dump
  use_backend cache if vulnerability

backend cache
  option httpchk
  http-check send meth GET uri /api/system/configInfo hdr Host vulnerability.circl.lu
  option forwardfor
  server varnish localhost:6081 check proto h2
  server vulnerability [2001:db8::1]:10001 check backup

backend vulnerability-dump
  option httpchk
  http-check send meth GET uri / hdr Host vulnerability.circl.lu
  http-request set-path "%[path,regsub(^/dumps/,/)]"
  option forwardfor
  server vulnerability [2001:db8::2]:80 check

backend vulnerability-cdn
  option httpchk
  http-check send meth GET uri / hdr Host vulnerability.circl.lu
  option forwardfor
  server vulnerability [2001:db8::3]:80 check

Notes:

The application server is a backup of Varnish: if Varnish fails, traffic still flows, only slower.
set-path "%[path,regsub(^/dumps/,/)]" strips /dumps/, so the files sit at the root of that Nginx.
The health check uses /api/system/configInfo.

Varnish (HTTP reverse-proxy cache)#

Varnish serves repeated GET requests from memory, without reaching Gunicorn, Kvrocks or PostgreSQL. Vulnerability records change rarely and are read often, so even short TTLs absorb most of the load and traffic spikes.

Cache stable reads: /vuln/<id>, /api/vulnerability/<id>, static assets.
Bypass the cache for POST/PUT/DELETE, authenticated requests and personal pages.
Prefer Cache-Control headers set by the application over VCL rules.

Example configuration#

A minimal VCL, as on the CIRCL instance. It passes requests with x-api-key, strips every cookie except the session cookie and the SSO flow cookie (without it, SSO sign-ins fail as expired), gzips text, and caches for 15 minutes with a 2-hour grace.

backend default {
    .host = "2001:db8::1";
    .port = "10001";
    .first_byte_timeout = 30s;
    .between_bytes_timeout = 20s;
}

sub vcl_recv {
    # don't cache authenticated api requests
    if (req.http.x-api-key) {
        return (pass);
    }
    # remove cookie to force cache for special bootstrap content
    if (req.url ~ "^/bootstrap($|/.*)") {
        unset req.http.cookie;
    }
    # remove unwanted cookie to cache more requests; keep the session cookie and
    # the single sign-on flow cookie (docs/sso.md)
    if (req.http.Cookie) {
        set req.http.Cookie = ";" + req.http.Cookie;
        set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
        set req.http.Cookie = regsuball(req.http.Cookie, ";(vulnerability-lookup|__Host-vl_sso_flow|vl_sso_flow)=", "; \1=");
        set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
        set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

        if (req.http.Cookie == "") {
            unset req.http.Cookie;
        }
    }
}

sub vcl_backend_response {
    set beresp.ttl = 15m;
    set beresp.grace = 2h;
    # Allow browser to cache requests longer
    unset beresp.http.Cache-Control;
    set beresp.http.cache-control = "max-age=900";
    # compress text content
    if (beresp.http.content-type ~ "(application/json|text/html|image/svg|text/css|text/javascript)") {
        set beresp.do_gzip = true;
    }
    # increase cache time
    if (!beresp.http.last-modified) {
        set beresp.http.last-modified = now;
    }
}

Nginx (static-file backends)#

HAProxy routes /dumps/ and /static to two Nginx instances, so file traffic never reaches Varnish or Gunicorn.

/dumps/: the daily NDJSON exports written by bin/dump.py (see Command-Line Interface). Served from disk with sendfile and autoindex on. The location /dumps/ block is in Reverse Proxy (Nginx).
/static: the application’s CSS, JavaScript and images, kept apart from dynamic requests.

Storage and database backends#

PgBouncer → PostgreSQL: users, comments, bundles, sightings, watchlists. See PgBouncer and PostgreSQL.
Kvrocks → Valkey: Kvrocks (TCP 10002, Redis protocol) replicates to Valkey for a hot standby and read capacity. See Kvrocks.
The local cache (Valkey/Redis on a Unix socket) is not drawn. See Valkey/Redis.

Gunicorn (Web Server)#

Set in config/generic.json. bin/start_website.py starts Gunicorn, and bin/gunicorn_conf.py reads the four website_* settings below. A setting set to null keeps Gunicorn’s own default.

website_workers: size against the web tier’s memory budget and measured load. Each gevent worker handles many clients; increasing workers also multiplies database pools and private memory. The sample’s 49 workers is not a sizing rule.
website_worker_connections: maximum simultaneous clients per gevent worker, including idle keep-alive and SSE clients (default 100).
website_max_requests: recycle a worker after 1,000 requests by default; set to 0 to disable. website_max_requests_jitter adds up to 100 random requests to stagger restarts. Recycling contains accumulated memory growth, but cannot protect against a single oversized request.
website_graceful_timeout: allow active requests 30 seconds to drain when a worker exits, including recycling, and when the website is stopped or restarted: stop and restart_website wait for it before forcing the stop. Long exports and SSE connections can be interrupted; streaming clients must reconnect. Increase this for your workload.
Worker silence timeout: 300 seconds. With gevent, this is not a per-request deadline while the worker’s event loop continues sending heartbeats.

The shared bin/gunicorn_conf.py reads these settings for the website launcher and the separate website Docker image. Existing instance configurations fall back to sample values for missing keys. Gunicorn CLI options override them. Restart the website after changing the settings.

When sizing the web tier, reserve memory for Kvrocks, Redis and PostgreSQL if they share the container. Compare container memory and worker PSS/private memory rather than summing RSS, which counts preloaded shared pages in each worker. Load-test large searches and streams as well as typical requests, and monitor latency and errors when reducing concurrency.

Search score lookups are processed in batches of at most 1,000 commands, without a transaction. This bounds pipeline queues, not complete searches: candidate lists and sorting still grow with the number of matches, even for a small response page.

Flags passed to Gunicorn:

--worker-class gevent       # async worker for concurrent I/O
--timeout 300               # worker silence timeout
--reuse-port                # SO_REUSEPORT for load balancing across workers
--proxy-protocol            # preserve client IPs behind a reverse proxy
--preload                   # load the application once, before forking
--forwarded-allow-ips=*     # trust X-Forwarded-* from any proxy

Kvrocks (Primary Storage)#

Put your tuning in storage/kvrocks.local.conf (copy storage/kvrocks.conf first). It is gitignored and used instead of kvrocks.conf when present. Do not edit kvrocks.conf itself: git pull overwrites it.

workers 8                   # I/O threads: about the number of CPU cores
maxclients 10000            # keep below the OS file descriptor limit (ulimit -n)
tcp-backlog 511             # keep net.core.somaxconn >= this value

RocksDB:

rocksdb.block_cache_size 4096           # MB; 25-50% of RAM on a dedicated server
rocksdb.max_open_files 8096             # -1 keeps all files open
rocksdb.write_buffer_size 64            # MB; raise for bulk imports
rocksdb.max_write_buffer_number 4
rocksdb.max_background_jobs 4           # raise on many cores for faster compaction
rocksdb.target_file_size_base 128       # MB
rocksdb.max_total_wal_size 512          # MB

Disk I/O:

max-io-mb 0                 # set a limit if compaction slows down requests
max-replication-mb 0

Valkey/Redis (Cache Layer)#

Configured in cache/cache.conf, on a Unix socket only:

port 0                      # TCP disabled — Unix socket only
unixsocket cache.sock
unixsocketperm 700
tcp-keepalive 300

Only memory needs watching: redis-cli -s cache/cache.sock INFO memory. Set maxmemory and an eviction policy (e.g. allkeys-lru) to bound it.

PostgreSQL#

# Connection / memory
max_connections = 200                     # keep moderate when using PgBouncer
shared_buffers = 64GB                     # ~25% of RAM
work_mem = 64MB                           # per query memory; adjust if heavy queries
maintenance_work_mem = 4GB

# WAL / checkpoints
wal_buffers = 16MB
checkpoint_completion_target = 0.9
max_wal_size = 4GB
min_wal_size = 1GB

# Autovacuum
autovacuum = on
autovacuum_max_workers = 10
autovacuum_naptime = 10s
autovacuum_vacuum_cost_limit = 2000

# Logging
log_connections = on
log_disconnections = on
log_min_duration_statement = 2000         # log slow queries > 2s

PgBouncer (Optional)#

Recommended with many Gunicorn workers: it pools connections to PostgreSQL.

[databases]
vulnlookup = host=127.0.0.1 port=5432 dbname=vulnlookup

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt
admin_users = vuln-lookup

pool_mode = transaction                     # best for web apps
default_pool_size = 200                     # number of server connections per DB
min_pool_size = 50
reserve_pool_size = 50
reserve_pool_timeout = 5.0                  # seconds to wait for reserve
max_client_conn = 5000

# Logging / stats
logfile = /var/log/pgbouncer/pgbouncer.log
log_connections = 1
log_disconnections = 1

Point DB_CONFIG_DICT in config/website.py to PgBouncer (6432), not PostgreSQL (5432).
Here default_pool_size equals max_connections (200). Keep a few connections free for admin access.

SQLAlchemy Connection Pool#

SQLALCHEMY_ENGINE_OPTIONS in config/website.py.

Without PgBouncer (default):

SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 100,               # persistent connections in the pool
    "max_overflow": 50,             # extra connections during traffic spikes
    "pool_timeout": 30,             # seconds to wait for a connection
    "pool_recycle": 1800,           # recycle connections every 30 minutes
    "pool_pre_ping": True,          # verify connection liveness before use
}

Warning

Each worker has its own pool: up to pool_size × website_workers connections (100 × 49 = 4,900). Without PgBouncer, keep that well below max_connections.

With PgBouncer (let PgBouncer manage the pool):

SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 0,                 # no persistent connections; rely on PgBouncer
    "max_overflow": 0,
    "pool_pre_ping": True,
    "pool_timeout": 30,
    "pool_recycle": 3600,
}

Logging#

Lower log levels in production:

config/generic.json: set loglevel to WARNING or ERROR
config/website.py: set LOG_LEVEL to WARNING
storage/kvrocks.local.conf: log-level warning (default)
config/modules.cfg: level = WARNING per feeder

config/logging.json rotates files (1 MB, 5 backups).

Operating System Tuning#

File Descriptors#

# /etc/security/limits.conf  (or systemd unit override)
* soft nofile 65536
* hard nofile 65536

Network Stack#

sysctl -w net.core.somaxconn=4096
sysctl -w net.ipv4.tcp_max_syn_backlog=4096
sysctl -w net.core.netdev_max_backlog=4096
sysctl -w vm.overcommit_memory=1              # recommended for Redis/Kvrocks

Transparent Huge Pages (THP)#

Disable it for Kvrocks and Valkey (latency spikes):

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Reverse Proxy (Nginx)#

A minimal configuration:

upstream vulnlookup {
    server 127.0.0.1:10001;
    keepalive 64;
}

server {
    listen 443 ssl http2;
    server_name vulnerability.example.org;

    client_max_body_size 10m;

    location / {
        proxy_pass http://vulnlookup;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 10;
        proxy_read_timeout 300;           # match Gunicorn timeout
        proxy_send_timeout 300;
    }

    # Open-data NDJSON dumps produced daily by ``bin/dump.py``.
    # Served as static files; never proxied to Gunicorn.
    location /dumps/ {
        alias /path/to/vulnerability-lookup/dumps/;   # $VULNERABILITY_LOOKUP_HOME/dumps/

        autoindex on;                     # public directory listing
        autoindex_exact_size off;
        autoindex_localtime on;

        # Serve .ndjson with a useful content type
        types {
            application/x-ndjson  ndjson;
        }
        default_type application/octet-stream;

        sendfile on;                      # zero-copy for large dumps
        tcp_nopush on;
        gzip_static on;                   # serve pre-compressed .ndjson.gz if present

        # Dumps are regenerated daily; allow downstream caches to keep them briefly.
        expires 1h;
        add_header Cache-Control "public";

        # Only GET/HEAD make sense for a static dump tree.
        limit_except GET HEAD { deny all; }
    }
}

Gunicorn runs with --proxy-protocol. Enable the PROXY protocol on your proxy, or rely on X-Forwarded-For and remove the flag from bin/start_website.py.
The alias must point to the dumps/ directory of the Vulnerability-Lookup home, readable by Nginx.

Summary#

Layer	Key Settings	Tune When
Gunicorn	`website_workers`, `--timeout`, `--worker-class gevent`	High concurrent users, slow responses
Kvrocks (`kvrocks.local.conf`)	`workers`, `maxclients`, `rocksdb.block_cache_size`	Large dataset, slow queries, high write load
Valkey/Redis	`maxmemory`, eviction policy	Cache memory growing unbounded
PostgreSQL	`max_connections`, `shared_buffers`, `work_mem`	Slow user/account queries, high concurrency
PgBouncer	`pool_mode`, `default_pool_size`, `max_client_conn`	Many workers exhausting PostgreSQL connections
SQLAlchemy	`pool_size`, `max_overflow`, `pool_pre_ping`	Connection errors, stale connections
OS	`nofile`, `somaxconn`, THP	Connection refused errors, latency spikes

Performance Tuning

Contents

Performance Tuning#

Production Reference Architecture#

HAProxy (TLS termination and load balancing)#

Example configuration#

Varnish (HTTP reverse-proxy cache)#

Example configuration#

Nginx (static-file backends)#

Storage and database backends#

Gunicorn (Web Server)#

Kvrocks (Primary Storage)#

Valkey/Redis (Cache Layer)#

PostgreSQL#

PgBouncer (Optional)#

SQLAlchemy Connection Pool#

Logging#

Operating System Tuning#

File Descriptors#

Network Stack#

Transparent Huge Pages (THP)#

Reverse Proxy (Nginx)#

Summary#