Server Operations

This page covers day-to-day operation of gonemaster-server: jobs, batches, queue control, scheduled snapshots, deletion, health, and metrics.

Jobs and Batches

Use jobs for individual domain tests. Use batches for many domains or for rerunning every domain in a tag.

Interactive single-domain jobs use normal priority. Batch jobs use batch priority, so a large sweep does not block a waiting interactive user.

Create one job:

curl -s -X POST http://127.0.0.1:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com", "min_level": "NOTICE"}'

Create a batch:

curl -s -X POST http://127.0.0.1:8080/api/v1/jobs/batch \
  -H "Content-Type: application/json" \
  -d '{"from_tag": "tld", "tags": ["tld"], "description": "TLD sweep"}'

Batch submission does not support undelegated input. Use a single job when you need explicit nameservers or DS records.

Submitting client

Every job created through POST /api/v1/jobs, POST /api/v1/jobs/batch or POST /pub/api/v1/jobs records its client, and the run keeps the values:

FieldValue
client_ipThe peer address, or the X-Forwarded-For hop selected under trusted_proxy_cidrs; the address the rate limiter keys on.
client_user_agentThe User-Agent header, characters outside printable ASCII replaced by ?, cut to 256 bytes.
client_viaui when the request carries an Origin of this server, as a browser on the admin or public UI sends; api otherwise. A client can forge Origin, so ui is evidence, not proof.

Scheduled batches record no client. The fields appear on the admin job and run routes and in the admin UI job inspector; no public route returns them. client_ip on GET /api/v1/jobs and GET /api/v1/runs selects by address.

An IP address and a User-Agent are personal data. Both are deleted with the run, by the retention purge or by batch deletion; no shorter retention applies.

Queue Controls

The admin API and admin UI can:

  • pause and resume queue processing
  • reorder queued jobs within priority rules
  • remove queued jobs
  • cancel queued or running jobs

Queue changes affect pending work only. Completed runs are historical records.

Priority tiers:

TierValueAssigned to
Normal0Admin and public single-domain jobs.
Batch1Batch submissions and tag sweeps.

Workers always drain normal jobs before batch jobs. Reordering is allowed only within the tier rules.

Background Work

Besides the workers, the server runs these loops:

LoopIntervalWork
Retention purgepurge_interval_secondsDeletes terminal runs older than retention_days.
Stuck job reaper1 minuteFails abandoned running jobs; see Jobs that will not finish .
Snapshot capture30 secondsCaptures the snapshots of drained snapshot-intent batches.
Cohort scheduler1 minuteSubmits the snapshot runs of due cohort schedules.

The capture loop and the scheduler run only with the analysis controller, on the SQL backends. Both also run once at startup, so a restart captures a batch that drained while the server was down and fires a schedule that fell due. A schedule overdue at startup fires once whatever the number of occurrences it missed, unless its catch_up is false and it is over one hour late.

Scheduled batches carry origin schedule and show a scheduled badge in the Batches tab. Each firing logs one scheduled run line with cohort_id, tag, outcome, batch_id and next_run_at, at warn for outcome error. scheduler_enabled stops all firing; see configuration.md and ../analysis/cohorts.md .

Retention and Deletion

Retention purges old terminal jobs according to the configured retention window. Manual batch deletion is stronger: it deletes a batch, completed runs, entries, analysis facts, and cohort snapshots derived from that batch.

Use snapshot retire or purge when the batch itself should stay but a public snapshot should be hidden or removed.

Batch deletion entry points:

  • GET /api/v1/batches/{id}/delete-preview
  • DELETE /api/v1/batches/{id}
  • Admin UI batch inspector
  • Admin UI tag detail earlier-batches panel
  • Admin UI cohort snapshot panel

Batch deletion keeps domains and tag memberships. It removes records derived from the selected batch.

Health and Metrics

  • GET /api/v1/healthz reports liveness.
  • GET /api/v1/metrics returns JSON metrics by default.
  • GET /api/v1/metrics?format=prom returns Prometheus text exposition.

See metrics.md for metric names and dashboard guidance.

The JSON metrics endpoint accepts:

  • format=json|prom
  • window=1h|6h|24h|48h
  • include=all,health,jobs,api,quality,insights,trends
  • limit_domains=1..100
  • limit_batches=1..100

Prometheus output ignores JSON-only query parameters.

Jobs that will not finish

A job only leaves running when the worker writes its result. If that write cannot commit, the worker marks the job failed with an error beginning graduation failed:, so callers stop polling and the in-flight gauge drains.

As a backstop, a sweep every minute fails any job left at running with no worker on it for longer than stuck_job_timeout_minutes (default 20, 0 disables it). Reaped jobs are logged as reaped stuck job.

A job that runs longer than job_timeout_minutes (default 20, 0 disables it) is stopped and fails with the error timed out.

This assumes one server process per database. Two servers sharing one database will reap each other’s live jobs, so set the timeout to 0 on such a deployment.

Log lines to watch:

  • job graduation failed - a result could not be stored; the underlying database error follows.
  • marking ungraduated job failed did not persist - the fallback write failed too; the reaper or the next restart clears it.
  • startup recovery left orphan jobs ungraduated - recovery skipped some rows and started anyway.

Database contention is retried and does not appear here. A steady stream of job graduation failed points at the database; on MariaDB see database-setup.md .

Jobs that panic

A panic in the engine, or in the server code that prepares its run, fails that job only. The job ends failed with an error beginning panic:, the worker takes the next job, and the server logs engine panic at ERROR with job_id, domain, value and stack, the stack of the goroutine that panicked. The public API does not expose the error. Resubmitting the domain repeats the panic; the log line is the material for a bug report.

A panic in the worker outside the run, while it records the job’s status, stores the result or projects it into analysis cohorts, is logged as job panic at ERROR with job_id, value and stack. A job whose result was not yet stored ends failed with panic: <value>; a stored result is kept. The worker takes the next job in both cases.

Logging

The server writes operational logs (lifecycle, access log, warnings, errors) to stderr using log/slog. Two encodings are available, selected with log_format (see configuration.md ):

  • text (default): human-readable key=value lines for local use and journald.
  • json: one JSON object per line, for aggregation (Loki, ELK, cloud logging).

Ship logs by letting systemd/journald or the container runtime capture stderr; there is no in-process file rotation.

log_level sets the minimum level (debug, info, warn, error). Levels are chosen so operators can alert on error lines without parsing status codes.

Access log

One http_request line per request, on the API mounts (/api/v1, /pub/api/v1) and on the page-serving mounts (/, /public/, /analysis/, /robots.txt, /sitemap.xml). Static assets, meaning the hashed bundles, fonts, and icons a page pulls in, are served without a line, so the log holds roughly one entry per page view rather than one per file.

Page views are visitor activity and each line carries the client IP in remote and its User-Agent in user_agent, so the retention policy of whatever collects stderr also governs this data.

The line’s level follows the response status: 2xx/3xx -> info, 4xx -> warn, 5xx -> error. A 503 from the public analysis API with error code no_selectable_cohorts or no_default_cohort reports server configuration and is logged at warn. Fields:

FieldMeaning
methodHTTP method.
pathRequest path.
routeThe router’s matched pattern, prefixed with the mount, for aggregation.
statusHTTP status code.
duration_msHandler wall time in milliseconds.
bytesResponse body bytes written.
remoteClient IP (honors trusted_proxy_cidrs).
user_agentUser-Agent header, as stored in client_user_agent.
request_idCorrelation ID, also returned in the X-Request-Id header.
bodyResponse body preview, only when --debug/log_level=debug is set.

A sample JSON access line:

{"time":"2026-07-19T10:00:00Z","level":"INFO","msg":"http_request","method":"GET","path":"/api/v1/runs/r1","route":"/api/v1/runs/{id}","status":200,"duration_ms":12,"bytes":842,"remote":"203.0.113.7","user_agent":"curl/8.7.1","request_id":"9f1c2a3b4d5e6f70"}

Request IDs

Every request is tagged with a correlation ID that ties the access-log line, any handler audit/error lines, and a panic line to the same request. The ID is returned in the X-Request-Id response header. An inbound X-Request-Id is trusted only from a trusted_proxy_cidrs peer; otherwise a fresh ID is generated.

Body capture

--debug (or debug: true) implies log_level=debug and adds a body field with a truncated response-body preview to each access line. Leave it off in production to avoid logging response bodies.