Health, metrics and logs¶

Three addresses answer machines about the instance itself, rather than about anybody's job search: /healthz for whatever restarts Postulo when it stops, /metrics for Prometheus, and /logs for a log collector. They live outside the API because they are not a person's: no API token opens them, and none of them is scoped to an account.

/healthz¶

For a container's health check, a load balancer or an uptime monitor. It asks the database for a connection and nothing more, so it is cheap to call often.

{"status": "ok", "database": "ok", "version": "0.2.1"}

200 when the database answered. 503 with {"status": "error", "database": "unavailable"} when it did not. The image's own health check calls it over plain HTTP on 127.0.0.1:8000.

That check is answered whatever POSTULO_ALLOWED_HOSTS holds: a read of /healthz from inside the container itself, which no proxy passed on, is not asked for a host the instance answers to. Anybody else asking, an uptime monitor or a load balancer, needs one of those hosts, like every other address.

/metrics¶

Prometheus metrics, in the text exposition format (text/plain; version=0.0.4), read from the database at the moment of the scrape and never cached.

Switched on with POSTULO_METRICS_ENABLED=true. Off, the address is a 404, so a stranger cannot tell it exists.

A token is required. The numbers carry nothing about anybody, but they name the exact Postulo, Python and Django versions and say how many accounts there are, which is what somebody choosing a target wants. With POSTULO_METRICS_ENABLED=true and POSTULO_METRICS_TOKEN empty the endpoint answers 503 and writes a line in the log, as /logs does; with a token set a request needs Authorization: Bearer <token> or it gets 401.

Every metric is a gauge. A scrape configuration:

scrape_configs:
  - job_name: postulo
    scheme: https
    metrics_path: /metrics
    scrape_interval: 1m
    authorization:
      credentials: YOUR_METRICS_TOKEN   # leave the block out when no token is set
    static_configs:
      - targets: ["postulo.example.org"]

Noticing that a backup has not happened¶

Two metrics, both about copies of the whole instance (see Backups and your data):

• postulo_backup_last_success_timestamp_seconds (gauge): when a backup last finished and verified, by hand or by the schedule, as a Unix time. 0 when none has.

• postulo_backup_failures_total (counter): backups that failed since the instance began.

• postulo_backup_offsite_last_success_timestamp_seconds (gauge): when an archive last reached the backup destination, encrypted, as a Unix time. 0 when none has.

• postulo_backup_offsite_failures_total (counter): attempts to send an archive to the backup destination that failed. Each retry that fails counts.

Alert on time() - postulo_backup_last_success_timestamp_seconds growing past a day and a half for a daily schedule, and on increase(postulo_backup_failures_total[1d]) > 0. With a destination chosen, alert on time() - postulo_backup_offsite_last_success_timestamp_seconds the same way: a copy that has not left for two days is a copy you do not have.

Noticing that the scheduler has stopped¶

The scheduler is a loop in a container, and the way it fails is quietly: nothing errors, reminders simply stop arriving, and the first person to notice is the one who missed a deadline. Two rules cover it — the first says the loop has stopped, the second says it is running but not getting through, which is the one a broken notifier produces.

groups:
  - name: postulo
    rules:
      - alert: PostuloSchedulerStopped
        # Also fires on an instance that has no scheduler at all, which is worth knowing
        # once. Drop the rule, or run one, rather than silencing it.
        expr: time() - postulo_scheduler_last_pass_timestamp_seconds > 1800
        for: 10m
        annotations:
          summary: No scheduler pass for half an hour

      - alert: PostuloRemindersOverdue
        expr: postulo_overdue{kind="reminders"} > 0
        for: 30m
        annotations:
          summary: Reminders have fallen due and nobody has been told

      - alert: PostuloDeliveriesFailing
        expr: postulo_failures > 0
        for: 1h
        annotations:
          summary: "{{ $labels.kind }} have been failing for an hour"

If you run the scheduler from cron rather than with --loop, set the first threshold to comfortably more than the gap between runs.

The scheduler service in the Compose files has a healthcheck of its own that reads the same heartbeat from the data volume, so docker compose ps shows it as unhealthy once it has stopped going round. Before 0.4.0 it inherited the image's healthcheck, which curls the web port that this container does not serve — so it was always unhealthy and said nothing.

/logs¶

The log, for a collector somewhere else on your network — Grafana Alloy, Vector, Promtail — that can poll a URL more easily than it can have log shipping arranged off the host. Anybody who can read the container's standard output should carry on doing that.

Switched on with POSTULO_LOGS_ENDPOINT_ENABLED=true; off, it is a 404. A token is required: a log entry names connections, companies and applications, and an open log endpoint is a data leak with a URL. With the endpoint on and POSTULO_LOGS_TOKEN empty, it refuses with 503 and writes an error to the log saying why, rather than publishing anything because a variable was forgotten. It reads the file Postulo keeps in POSTULO_LOG_DIR, so with that empty there is nothing to serve.

curl -H "Authorization: Bearer YOUR_LOGS_TOKEN" \
  "https://postulo.example.org/logs?since=2026-09-10T08:00:00Z&level=WARNING"

The answer is application/x-ndjson, oldest first, the order a collector appends in: one JSON object per line, with time, level, logger and message, plus whatever fields that record carried -- request_id among them, whenever the line was written while answering a request, during a scheduler pass (pass-…) or by a background errand (errand-<number>). The same id is in the response's X-Request-ID header and at the end of gunicorn's access line, so the three logs can be laid side by side. It is answered once and not streamed. A collector polls, and a stream would hold one of the server's few workers open for as long as the collector wanted.

Reading it in pages. A collector keeps the time of the last record it received and sends it as since. The answer holds the oldest limit records later than that, so after a burst the collector gets the start of it first: an answer with limit records in it means there may be more, and asking again with the new last time fetches them; a shorter one is the end. Up to and including 0.3.0 the answer was the newest limit records, and whatever lay between them and since was never served.

• since is an instant. Any ISO-8601 offset is understood, Z included, and a time with none is read as UTC. The time of a record can be sent back exactly as it was received. A value that is not a time is refused with 400 rather than guessed at.
• Times are written to the millisecond and several records can share one. An answer never stops between two that do, so it may hold a few more than limit.
• A line in the file that is not one of Postulo's records has no time and is left out when since is given.
• Several processes write the one file, so it is nearly in the order of its times and not strictly, and the search for since allows a second for that. What a mark made of a time cannot cover is a record stamped earlier than the last one of an answer and standing after it in the file. It takes two processes writing in the same moment exactly where an answer ends.

What /metrics and /logs share¶

• Too often is 429, with Retry-After in seconds. /metrics and /logs allow POSTULO_ENDPOINT_RATE requests per calling address, 120/h by default. The limit is per address because a shared token, not an account, is what opens them. A scrape every minute is half of it. Wrong tokens have a budget of their own: ten an hour per address, and then every request from that address is 429 until the hour turns, whatever token it presents. A collector with the right token never spends it.
• Make the token with openssl rand -hex 32. One shorter than 24 characters is reported as a warning by manage.py check.
• A wrong or missing token is 401, with WWW-Authenticate: Bearer (realm="postulo-metrics" or realm="postulo-logs"), and tokens are compared in constant time.
• Nothing is cached (Cache-Control: no-store).

Every variable is in Configuration. /manifest.webmanifest is not one of these: it is what a phone reads when Postulo is installed on its home screen.