Server Database

This page owns database selection and operational storage guidance for gonemaster-server.

Backends

BackendUse when
memoryDeveloping, testing, or running short-lived scripts.
sqliteRunning one server with small or medium data sets.
postgresRunning high-volume analysis or querying JSON arguments heavily.
mariadbReusing existing MariaDB or MySQL infrastructure.

DSNs

SQLite:

gonemaster-server \
  --db-driver sqlite \
  --db-dsn /var/lib/gonemaster/gonemaster.db

PostgreSQL:

GONEMASTER_DB_DRIVER=postgres \
GONEMASTER_DB_DSN="postgres://user:pass@host:5432/dbname?sslmode=disable" \
  gonemaster-server

MariaDB or MySQL:

GONEMASTER_DB_DRIVER=mariadb \
GONEMASTER_DB_DSN="user:pass@tcp(host:3306)/dbname" \
  gonemaster-server

parseTime=true is appended automatically for MariaDB/MySQL if it is not already present.

Stored Data

The server stores:

  • queued and running jobs
  • completed runs and log entries
  • domain and tag registry data
  • batch metadata
  • stored profiles
  • analysis cohort materialization and snapshots

Retention removes old terminal jobs and their stored results. Domain and tag records are shared metadata and are not owned by one batch.

Connection Pools

BackendMax open connectionsMax idle connectionsConnection lifetime
sqlite1defaultdefault
postgres2555 minutes
mariadb2555 minutes

SQLite is serialized to avoid concurrent writer issues. PostgreSQL and MariaDB use a small pool suitable for the server worker model.

Startup Recovery

With a persistent backend, startup performs three recovery steps:

  1. Run pending schema migrations.
  2. Mark jobs that were running during the last shutdown as failed.
  3. Re-enqueue jobs that were still queued.

This keeps the queue consistent after a restart without rewriting completed runs.

Retention

Configure automatic retention with --db-retention-days or database.retention_days:

gonemaster-server \
  --db-driver sqlite \
  --db-dsn /var/lib/gonemaster/gonemaster.db \
  --db-retention-days 90
  • 0 keeps completed results forever.
  • A positive value starts an hourly purge loop.
  • Only terminal jobs are purged.
  • Queued, running, and paused jobs are never purged automatically.

Manual batch deletion is separate from retention. It removes one batch and its derived records regardless of age. See operations.md .