This directory contains the maintained installation templates and the operator runbook for the production analytics service on Ubuntu Server 24.04. Nothing in the repository installs, updates or activates the server-side components automatically.

The design has two separate outputs:

private static GoAccess HTML and JSON reports for each registered project subdomain and for all registered project subdomains combined;
a small public visitor-count.json file for sites which explicitly enable the counter.

The public number is the sum of daily approximate unique visits since the configured activation date. GoAccess treats requests with the same IP address, date and user agent as one visit. That is useful for a rough trend. It is not a count of people.

Data flow

eligible page request
        |
        v
dedicated Nginx analytics log (14 days)
        |
        v
GoAccess with IP anonymisation
        |
        +--> private per-site and combined HTML/JSON reports (395 days)
        |
        v
durable daily counter state --> public visitor-count.json
                                      |
                                      v
                         same-origin home-page request

private HTML reports --> Nginx Basic Auth --> stats.hamradioonline.de

Nginx writes a dedicated, reduced log. For each site, the generator gives the existing, uncompressed .1 rotation and then the current log directly to GoAccess. GoAccess uses its persistent state to skip entries already processed. The generator does not use an incremental shell pipeline or decompress older rotations. Reports, database updates and public counter data are first prepared in a staging directory. GoAccess output is validated before any published report changes. The last valid report therefore survives a failed GoAccess run.

The counter state is deliberately separate from the 395-day report database. Each day is replaced with the latest value reported by GoAccess instead of being added again. This makes repeated runs idempotent. Values older than 395 days remain in the counter state and continue to contribute to the public total.

Production platform

The confirmed production baseline is:

Ubuntu Server 24.04;
Nginx 1.24;
Node.js 18.19.1;
GoAccess 1.8.1 with GeoIP2/MMDB and OpenSSL support, but without Zlib;
a local GeoLite2-Country database;
systemd for the oneshot generator and its hourly timer;
Logrotate for the dedicated analytics log.

Missing Zlib support is intentional for this operating model. The generator processes the current uncompressed analytics log and the optional uncompressed .1 rotation. It does not process .gz files during regular operation.

Files

generate-reports.js validates configuration and state, runs GoAccess and publishes outputs atomically.
sites.example.json is the registry template.
goaccess.conf.template is rendered per report with a private database path and the configured GeoIP2 Country database.
nginx/ contains the reduced log format, request filters, public endpoint and protected report-vhost examples.
systemd/ contains a hardened oneshot service and hourly timer.
logrotate/ retains 14 daily analytics-log rotations.

These repository files are templates and source files. Their productive counterparts are installed separately:

generator: /opt/hamradioonline-analytics/generate-reports.js;
registry: /etc/hamradioonline-analytics/sites.json;
GoAccess template: /etc/hamradioonline-analytics/goaccess.conf.template;
service and timer: /etc/systemd/system/hamradioonline-analytics.service and /etc/systemd/system/hamradioonline-analytics.timer;
Logrotate configuration: /etc/logrotate.d/hamradioonline-analytics;
service state and derived outputs: /var/lib/hamradioonline-analytics.

Prerequisites

Node.js 18.19.1 or newer;
GoAccess with GeoIP2/MMDB support;
one GeoIP2 Country database, not a City database;
Nginx;
an unprivileged service account, shown as hamradio-analytics in the examples.

No npm package is required by the generator. GoAccess is the only external program it starts.

The production compatibility baseline is GoAccess 1.8.1 built with --enable-geoip=mmdb and --with-openssl, but without --with-zlib. Zlib is not required for the regular operating mode because it reads only uncompressed files. The configuration check reports the detected build features and explicitly accepts this combination.

The confirmed production baseline is Node.js 18.19.1. The generator and tests must remain compatible with it; upgrading Node.js is not part of this setup.

The service account needs read access to the dedicated analytics logs and /var/lib/GeoIP/GeoLite2-Country.mmdb. It needs write access only to its state, report and public-output directories. Access is group-based. The setup does not depend on ACLs or setfacl. Nginx receives read access to reports and the public counter through the www-data group, but no write access.

Accounts and roles

hamradio-analytics is a local system account with its own group, no regular home directory, /usr/sbin/nologin as its shell and a locked password. It runs the generator and GoAccess, reads the reduced logs and configuration, and writes only below the configured service-state directories.

Nginx runs as www-data. It writes the dedicated analytics log and reads the private reports and public counter. It must not be able to read the private GoAccess databases or public-counter-state.json, and it has no write access to generated output.

stats-reader is the current local Nginx Basic Auth username. It is not a Linux service account and not an account with an external analytics provider. Its password file is /etc/nginx/htpasswd/hamradioonline-analytics. Store the password and hash only on the server and in a future protected backup, never in Git, this runbook, screenshots or support output.

Installation and permissions

Do not trust Unix modes stored in a ZIP created on Windows. Install every script, configuration and unit with an explicit owner, group and mode. The following commands assume that the package has been unpacked into the current directory. Create the dedicated, unprivileged service account once:

if ! getent passwd hamradio-analytics >/dev/null; then
  sudo useradd --system --user-group --home-dir /nonexistent --no-create-home \
    --shell /usr/sbin/nologin hamradio-analytics
fi

Then install the generator, configuration and units:

sudo install -d -o root -g hamradio-analytics -m 0750 \
  /opt/hamradioonline-analytics /etc/hamradioonline-analytics
sudo install -o root -g hamradio-analytics -m 0750 \
  website/ops/analytics/generate-reports.js \
  /opt/hamradioonline-analytics/generate-reports.js
sudo install -o root -g hamradio-analytics -m 0640 \
  website/ops/analytics/goaccess.conf.template \
  /etc/hamradioonline-analytics/goaccess.conf.template
sudo install -o root -g hamradio-analytics -m 0640 \
  website/ops/analytics/sites.example.json \
  /etc/hamradioonline-analytics/sites.json
sudo install -o root -g root -m 0644 \
  website/ops/analytics/systemd/hamradioonline-analytics.service.example \
  /etc/systemd/system/hamradioonline-analytics.service
sudo install -o root -g root -m 0644 \
  website/ops/analytics/systemd/hamradioonline-analytics.timer.example \
  /etc/systemd/system/hamradioonline-analytics.timer
sudo install -o root -g root -m 0644 \
  website/ops/analytics/logrotate/hamradioonline-analytics.example \
  /etc/logrotate.d/hamradioonline-analytics

Create the writable tree deliberately. The state root is traversable but not readable by Nginx. Only the report and public branches use the www-data group and the set-group-ID bit:

sudo install -d -o hamradio-analytics -g hamradio-analytics -m 0711 \
  /var/lib/hamradioonline-analytics
sudo install -d -o hamradio-analytics -g hamradio-analytics -m 0750 \
  /var/lib/hamradioonline-analytics/db
sudo install -d -o hamradio-analytics -g www-data -m 2750 \
  /var/lib/hamradioonline-analytics/reports \
  /var/lib/hamradioonline-analytics/reports/combined \
  /var/lib/hamradioonline-analytics/reports/kst4contest \
  /var/lib/hamradioonline-analytics/public \
  /var/lib/hamradioonline-analytics/public/kst4contest

Generated HTML and JSON reports use mode 0640. The public visitor-count.json uses 0644. Private GoAccess databases and public-counter-state.json remain owned by hamradio-analytics and unreadable by Nginx. The service unit uses StateDirectoryMode=0711 to retain this boundary after systemd has prepared the state directory.

The productive ownership and mode boundaries are:

generator: 0750 root:hamradio-analytics;
registry and GoAccess configuration: 0640 root:hamradio-analytics;
report directories, including reports/combined and reports/kst4contest: 2750 hamradio-analytics:www-data;
report files: 0640 hamradio-analytics:www-data;
private database files: 0640 hamradio-analytics:hamradio-analytics;
public-counter-state.json: mode 0640, owner and group hamradio-analytics:hamradio-analytics;
public output directories, including public/kst4contest: mode 2750, owner and group hamradio-analytics:www-data;
public visitor-count.json: 0644 hamradio-analytics:www-data;
Basic Auth password file: 0640 root:www-data.

/opt/hamradioonline-analytics and /etc/hamradioonline-analytics are managed by root and readable by the service group. /var/lib/hamradioonline-analytics belongs to the service account. Its 0711 root allows Nginx to traverse known paths without listing or reading private state. The service has a restrictive UMask=0027; the generator therefore sets the public counter to 0644 explicitly so Nginx can serve it. Missing output directories are a configuration error. Create them deliberately with the shared group and set-group-ID modes above rather than relying on recursive creation with an unsuitable group.

Create the analytics log only when it does not already exist. Running install /dev/null unconditionally would empty an existing log:

if [ ! -e /var/log/nginx/kst4contest-analytics.log ]; then
  sudo install -o www-data -g hamradio-analytics -m 0640 /dev/null \
    /var/log/nginx/kst4contest-analytics.log
fi
sudo stat -c '%U:%G %a %n' /var/log/nginx/kst4contest-analytics.log

The resulting log owner and mode must be www-data:hamradio-analytics 640. Logrotate preserves that ownership. The generator's configuration check fails clearly if required output directories are missing or if the executing user cannot read an analytics log or the Country database.

Registry

Copy sites.example.json outside the checkout and adjust it to the private server layout. Each site entry contains:

a stable id used for state and the GoAccess database;
its exact hostname;
the current, uncompressed analyticsLog path;
the activatedOn date used by the public counter;
a publicCounter switch;
the private reportOutputDirectory;
a publicJsonPath when the public counter is enabled.

The top-level combined.reportOutputDirectory receives the combined report. Only registered sites are included. The generator rejects stats.hamradioonline.de, so the report host cannot accidentally become part of the project statistics.

To add another project subdomain later, add one registry entry and one matching dedicated access_log line to its Nginx server block. Do not enable a public counter unless that site should publish one.

Treat activatedOn as persistent data. Once counting has started, changing it would change the meaning of the total. The generator refuses to combine a new activation date with existing counter state.

The hostname is persistent identity as well. If an existing site state has a different activatedOn or hostname, do not delete the state to make the next run pass. Changing either value requires a deliberate migration or a specifically approved reset of the public count.

The generator derives the optional .1 path from analyticsLog. It is valid for .1 not to exist before the first rotation. Do not enter a rotation or a compressed .gz file in the registry.

Adding another project subdomain also requires its own Nginx analytics log, the corresponding Logrotate ownership, a prepared report directory and, when enabled, a public-output directory and counter location. The combined report uses the logs of every registered project site. The statistics vhost remains outside the registry.

Nginx logging

The relevant production configuration files are:

/etc/nginx/snippets/hamradioonline-analytics-filters.conf;
/etc/nginx/conf.d/hamradioonline-analytics-log.conf;
/etc/nginx/snippets/kst4contest-public-counter.conf;
/etc/nginx/sites-available/kst4contest.conf;
/etc/nginx/sites-available/stats.hamradioonline.de.

Install the log-format and filter maps from nginx/ in the http context. Then add a dedicated analytics access_log to every registered project server block. Keep the existing operational access log unless its replacement has been reviewed separately. If the operational log is inherited from the http context, repeat its directive in the server block before adding the analytics log; an access_log at a lower level changes inheritance.

Install Nginx snippets explicitly as root:root with mode 0644; do not copy the modes from the ZIP. Nginx map exact-string keys use the path itself, for example /visitor-count.json, without the location-modifier prefix =.

sudo install -o root -g root -m 0644 \
  website/ops/analytics/nginx/analytics-filters.conf.example \
  /etc/nginx/snippets/hamradioonline-analytics-filters.conf
sudo install -o root -g root -m 0644 \
  website/ops/analytics/nginx/analytics-log.conf.example \
  /etc/nginx/conf.d/hamradioonline-analytics-log.conf
sudo install -o root -g root -m 0644 \
  website/ops/analytics/nginx/public-counter.conf.example \
  /etc/nginx/snippets/kst4contest-public-counter.conf

Prepare the public-counter include without enabling it in the active site yet. Likewise, keep the statistics vhost disabled until the certificate bootstrap step below.

The analytics format contains only:

server name;
client IP address;
timestamp;
method;
normalized path without query string;
protocol;
status;
transferred body size;
user agent.

The fields are tab-separated. The production log is /var/log/nginx/kst4contest-analytics.log. Nginx writes it as www-data; the hamradio-analytics group can read it. The confirmed owner and mode are www-data:hamradio-analytics 0640. The analytics service receives read-only access and must never truncate or otherwise modify this log.

The format uses $uri, not $request_uri, so query strings never enter the analytics log. It also omits referrer and authenticated remote-user data. The user agent is retained because GoAccess needs it for crawler classification and its visit definition.

Only eligible GET requests can be logged. Assets, downloads, status and monitoring paths, sitemap, robots file, favicons, the update feed and the public counter endpoint are excluded. Known bots, crawlers, monitoring clients, wget and curl are rejected before logging. GoAccess applies its own crawler list as a second layer and treats unknown browser or operating system combinations as crawlers. The public counter request therefore cannot count itself, and the statistics vhost has no analytics logging of its own.

Review the monitoring-path list against the real server before activation. When a new health endpoint or asset family is added, update the filter first.

Test the complete Nginx configuration before reloading it:

sudo nginx -t

Log rotation

/etc/logrotate.d/hamradioonline-analytics rotates the dedicated analytics logs daily, retains 14 rotations and compresses older files. delaycompress is an operational requirement: it keeps the immediately preceding rotation as an uncompressed .1 file for the next generator run. The create 0640 www-data hamradio-analytics directive preserves the write/read boundary. After rotation, invoke-rc.d nginx rotate makes Nginx reopen its logs.

The generator processes, in this order:

the optional, uncompressed .1 rotation;
the current analytics log.

A missing .1 is normal, including before the first rotation. Older .gz files are retained according to Logrotate but are not imported by the regular generator. A registry path that identifies .1, another numbered rotation or a .gz file is rejected.

GoAccess reports

The template enables IP anonymisation at anonymize-level 2 before persistent aggregation, ignores crawlers, keeps 395 days, and uses a separate persistent database for every site and the combined report. It leaves only the panels needed here: visits by day, requested pages, countries, HTTP status codes and virtual hosts. Host, remote-user, referrer, keyphrase, operating-system, browser and other detailed panels are disabled.

GoAccess 1.8.1 writes the Country panel under the JSON key geolocation. Combined jobs explicitly pass --enable-panel=VIRTUAL_HOSTS and require the resulting vhosts key. Site jobs do not enable that panel. The generator treats either missing key as an invalid report rather than publishing incomplete statistics.

The GoAccess 1.8.1 JSON keys are an interface invariant. geo_location and virtual_hosts are invalid names and may appear in repository tests only as deliberately rejected negative cases. The Country panel comes from the configured MMDB; missing geolocation invalidates every report, and missing vhosts invalidates the combined report.

The Country database is provided through the registry at /var/lib/GeoIP/GeoLite2-Country.mmdb. A file whose name contains City is rejected. Do not replace it with a City database merely because one happens to be available.

The generator supplies --persist, conditionally supplies --restore, and uses an isolated --db-path through the rendered template. It passes the uncompressed .1 rotation, when present, and then the current log as direct GoAccess arguments. This chronological order also covers entries appended shortly before rotation. GoAccess tracks the processed files in its persistent state and processes only new entries on later runs. The first successful run creates each database. Later runs copy the last valid database into staging, restore it and persist the updated result only after all reports have succeeded.

Logrotate must use delaycompress, as shown in the example. This leaves .1 uncompressed for one rotation cycle. Older .gz files are not part of the regular hourly run, and importing them is a separate maintenance task outside this repository workflow. Do not add an unstable decompression pipeline to the timer service.

If the generator is unavailable for longer than the uncompressed rotation window, the regular run cannot recover entries found only in older .gz files. Preserve those files under the raw-log retention policy and plan any necessary historical import separately before resuming normal processing.

Visit and privacy boundary

GoAccess treats requests with the same IP address, date and user agent as one visit. The public number is therefore an approximate visit total, not a count of uniquely identified people. Page views remain a separate statistic.

IP addresses are processed with the configured GoAccess anonymisation level. Country resolution happens locally against GeoLite2-Country; City and host statistics are not produced. No visitor address is sent to MaxMind or another analytics service. The website sets no analytics cookie, embeds no external tracking script and sends no visit to Google Analytics, Matomo Cloud or any other analytics platform.

The home-page script requests only /visitor-count.json from the same origin. The public file contains schemaVersion, visits, since and updatedAt and no visitor-level or daily detail.

Persistence and publication

The installation has four distinct persistence layers.

Raw logs

The current analytics log and its rotations are short-lived input. The current file and .1 bridge requests across the most recent rotation. Logrotate limits raw-log retention to the published 14-day policy; backups must not silently extend that period.

GoAccess databases

Persistent detail state lives in:

/var/lib/hamradioonline-analytics/db/kst4contest;
/var/lib/hamradioonline-analytics/db/combined.

--persist writes the processing state and --restore loads it on later runs. This lets GoAccess process new log content without adding the same input from scratch on every hourly run. The keep-last 395 setting limits detailed aggregates to a rolling 395 days.

Public counter state

/var/lib/hamradioonline-analytics/public-counter-state.json stores daily visit values for each enabled site from activatedOn onward. The generator replaces a day's value when it is processed again; it does not add the value a second time. This file is the durable business source for the lifetime public total, including days which have aged out of the GoAccess detail database.

The stored hostname and activatedOn must continue to match the registry. Changing either value requires a planned migration or an approved reset, not an ad-hoc edit or deletion of the state file.

Reports and public files

The derived outputs are:

/var/lib/hamradioonline-analytics/reports/kst4contest/report.html;
/var/lib/hamradioonline-analytics/reports/kst4contest/report.json;
/var/lib/hamradioonline-analytics/reports/combined/report.html;
/var/lib/hamradioonline-analytics/reports/combined/report.json;
/var/lib/hamradioonline-analytics/public/kst4contest/visitor-count.json.

The generator prepares every GoAccess job in a run directory, validates the HTML and JSON outputs, and calculates counter updates before publication. A GoAccess or report-validation failure therefore leaves the published files unchanged. Report, state and public files are replaced atomically one file at a time. Database directories are exchanged through a temporary backup name and restored if that exchange fails. These replacements are not one filesystem transaction across every report, database and counter file; after a storage or permission failure during publication, inspect the complete set and rerun the service after correcting the cause.

Counter state and GoAccess databases are the important persistent sources. HTML/JSON reports and visitor-count.json are derived and can be rebuilt when their corresponding source state is available.

Checking and running

Do not rely on node --check alone. First inspect the installed generator's owner, mode and plausible non-zero size, then compare its SHA-256 digest with the reviewed repository file:

sudo stat -c '%U:%G %a %s %n' \
  /opt/hamradioonline-analytics/generate-reports.js
sha256sum /srv/git/kst4contest/website/ops/analytics/generate-reports.js \
  /opt/hamradioonline-analytics/generate-reports.js
/usr/bin/node --check /opt/hamradioonline-analytics/generate-reports.js

A zero-byte JavaScript file is syntactically valid and exits successfully without doing any work. File size, digest and the expected completion message are therefore part of every recovery check.

Validate paths, registry values, the template contract and GoAccess availability without producing reports:

sudo -u hamradio-analytics /usr/bin/node \
  /opt/hamradioonline-analytics/generate-reports.js \
  --registry /etc/hamradioonline-analytics/sites.json \
  --config-template /etc/hamradioonline-analytics/goaccess.conf.template \
  --check

The check prints the detected GoAccess version and whether GeoIP2/MMDB, OpenSSL and Zlib build options are present. Missing GeoIP2/MMDB support is a configuration error with exit code 2. OpenSSL remains informational. Missing Zlib is supported for this operating mode and does not make the check fail. The same message explains that only the current log and optional uncompressed .1 are processed and that older .gz files are not imported.

Exercise the complete GoAccess and output-validation path without changing published reports, databases or counter state:

sudo -u hamradio-analytics /usr/bin/node \
  /opt/hamradioonline-analytics/generate-reports.js \
  --registry /etc/hamradioonline-analytics/sites.json \
  --config-template /etc/hamradioonline-analytics/goaccess.conf.template \
  --dry-run

Run without either flag to publish. A lock prevents concurrent production runs. A dry-run uses a temporary working directory and deliberately neither needs nor creates the production lock below /run. Configuration errors use exit code 2, failure to acquire the production lock uses exit code 3, and generation or publication errors use exit code 1.

Before the first production run, seed any earlier daily values which must be preserved into public-counter-state.json. There is no honest way to recreate history which is no longer present in the raw logs. Back up this state file: it is the durable source for public totals older than the detailed retention window.

Safe verification sequence

Use this order for a new installation, a recovered service or a material generator/configuration update:

Check generator size, ownership, mode and SHA-256 against the reviewed checkout.
Run /usr/bin/node --check on the installed generator.
Run the generator with --check as hamradio-analytics.
Confirm the reported GoAccess version and GeoIP2/MMDB, OpenSSL and Zlib capability state. Missing Zlib is expected; missing MMDB support is not.
Record the hashes and timestamps of current reports, databases and counter files, then run --dry-run.
Confirm that the recorded production files did not change and that /run/hamradioonline-analytics/generator.lock was not created by the dry-run.
Start one productive run through the service unit: sudo systemctl start hamradioonline-analytics.service.
Inspect Result and ExecMainStatus and read the unit journal. A successful run ends with Analytics generation completed.
Check every generated file's path, owner, group, mode and timestamp.
Request the public JSON through HTTPS and validate its four fields.
Request both private report URLs with Basic Auth. Let the client prompt for the password; never put it directly on a command line.
Run the service a second time and confirm that the total and report values develop plausibly rather than multiplying the existing history.
Enable or re-enable the timer only after these checks pass.
Confirm the first automatic run in the journal and later verify the first real log rotation separately.

Useful service checks are:

systemctl show hamradioonline-analytics.service \
  -p Result -p ExecMainStatus
journalctl -u hamradioonline-analytics.service --since today
systemctl status hamradioonline-analytics.timer
systemctl list-timers hamradioonline-analytics.timer

systemd operation

The production oneshot service runs as hamradio-analytics with UMask=0027. Its sandbox exposes /etc/hamradioonline-analytics, /opt/hamradioonline-analytics, the Nginx logs and the Country MMDB read-only. /var/lib/hamradioonline-analytics is its only application-state write area; /run/hamradioonline-analytics holds the production lock generator.lock. The service has no network access and only the documented read/write paths.

The timer uses OnCalendar=hourly, Persistent=true, RandomizedDelaySec=4m and AccuracySec=1m and is permanently enabled in production. Multiple automatic hourly runs have completed successfully. Each run produces one site report, one combined report and one public counter, takes roughly one second on the current installation and has shown stable incremental behaviour without sudden duplicate counting.

If the installed GoAccess build unexpectedly requires network access, find the reason before weakening that restriction; local log processing and a local Country database do not require it.

Report access

The statistics vhost serves static files over HTTPS and protects the complete host with HTTP Basic Authentication. This includes /, its redirect to /combined/, and every individual report. Store the password file outside this repository. Prepare it so only root and the Nginx group can access it:

sudo install -d -o root -g www-data -m 0750 /etc/nginx/htpasswd
if [ -e /etc/nginx/htpasswd/hamradioonline-analytics ]; then
  sudo htpasswd /etc/nginx/htpasswd/hamradioonline-analytics stats-reader
else
  sudo htpasswd -c /etc/nginx/htpasswd/hamradioonline-analytics stats-reader
fi
sudo chown root:www-data /etc/nginx/htpasswd/hamradioonline-analytics
sudo chmod 0640 /etc/nginx/htpasswd/hamradioonline-analytics

Use the local account name stats-reader and enter its password interactively. Never store the resulting password hash in this repository or an installation ZIP. The example opens no GoAccess WebSocket and no additional GoAccess port. Its own access log is disabled and responses use a private, no-store cache policy.

The current private endpoints are:

https://stats.hamradioonline.de/, which redirects an authenticated request to /combined/;
https://stats.hamradioonline.de/combined/, which serves the combined report;
https://stats.hamradioonline.de/kst4contest/, which serves the site report.

All HTTPS paths, including the redirect target, remain behind Basic Auth. Reports use Cache-Control: private, no-store, X-Content-Type-Options: nosniff and X-Frame-Options: DENY; dotfiles are blocked. The vhost has no access log and no analytics log. HTTP remains open only for the ACME webroot and permanently redirects every other request to HTTPS. No IPv6 listener is configured while DNS AAAA operation remains unconfirmed.

The final HTTP/HTTPS vhost is the production configuration. The HTTP-only bootstrap template is retained solely for first provisioning or recovery when the certificate files do not yet exist. In that situation, do not activate the final HTTPS vhost first. Install the temporary bootstrap without changing the parallel APT Certbot installation or either renewal timer:

sudo install -d -o root -g root -m 0755 /var/lib/letsencrypt
sudo install -o root -g root -m 0644 \
  website/ops/analytics/nginx/stats-vhost-http-bootstrap.conf.example \
  /etc/nginx/sites-available/stats.hamradioonline.de
if [ ! -e /etc/nginx/sites-enabled/stats.hamradioonline.de ] && \
   [ ! -L /etc/nginx/sites-enabled/stats.hamradioonline.de ]; then
  sudo ln -s /etc/nginx/sites-available/stats.hamradioonline.de \
    /etc/nginx/sites-enabled/stats.hamradioonline.de
fi
sudo nginx -t
sudo systemctl reload nginx
sudo /snap/bin/certbot certonly --webroot \
  --webroot-path /var/lib/letsencrypt \
  -d stats.hamradioonline.de

Only after Certbot has created the certificate, replace the bootstrap with the final vhost. This does not remove HTTP completely. The final file keeps an IPv4 port 80 block for /.well-known/acme-challenge/ so the certificate issued with --webroot can be renewed automatically. Every other HTTP request is redirected permanently to the same URI on HTTPS. The HTTPS block uses the existing Ubuntu/Certbot TLS files /etc/letsencrypt/options-ssl-nginx.conf and /etc/letsencrypt/ssl-dhparams.pem:

sudo install -o root -g root -m 0644 \
  website/ops/analytics/nginx/stats-vhost.conf.example \
  /etc/nginx/sites-available/stats.hamradioonline.de
sudo nginx -t
sudo systemctl reload nginx

After activation, verify both authentication and renewal from the server:

curl -I https://stats.hamradioonline.de/
curl -I -u stats-reader https://stats.hamradioonline.de/
sudo /snap/bin/certbot renew --dry-run \
  --cert-name stats.hamradioonline.de

The first HTTPS request must return 401. The authenticated request must return the redirect to /combined/. Enter the Basic Auth password interactively; do not put it on the command line. The Certbot dry-run must complete while the final vhost is active.

The templates listen on IPv4 only. Add an IPv6 listener later, after the AAAA record for stats.hamradioonline.de has been confirmed and tested.

The public counter location serves only visitor-count.json through an exact Nginx alias; it does not modify the deployed Eleventy release directory. Its own access log is disabled. Responses use Cache-Control: public, max-age=3600 and X-Content-Type-Options: nosniff. The file contains the schema version, total, activation date and update time. It contains no IP address, user agent, hostname or per-day detail.

The production URL is https://kst4contest.hamradioonline.de/visitor-count.json. Before the first successful generator run, a 404 is expected; afterwards it must return 200 with application/json. The home page reveals the counter only after a valid response. A missing, invalid or unavailable counter never prevents the rest of the static site from working.

Deployment boundary

GitHub is the source repository for the website and the reviewed analytics templates. The production checkout is /srv/git/kst4contest. A root cron job runs /srv/scripts/deploy-kst4contest-website.sh every five minutes. It fetches Git, resets the checkout to origin/main, runs npm ci, builds the Eleventy website, validates VersionInfo, synchronises the result to /srv/www/kst4contest/current and finally restores the ownership expected by Nginx.

The deployment credential is stored in /etc/kst4contest-website.env. Its value and account assignment must never appear in documentation, logs or support output.

This automatic deployment updates only the static website. It does not install or overwrite:

/opt/hamradioonline-analytics;
/etc/hamradioonline-analytics;
systemd units;
Nginx or Logrotate configuration;
the Basic Auth password file;
GeoIP configuration;
Certbot configuration.

Changes to those operational files require a separate review and manual installation. The published privacy notice must remain in place whenever analytics logging is active.

Regular operation

Nginx continuously writes only eligible requests to the dedicated analytics log. The systemd timer starts the generator once per hour. Every successful run refreshes the per-site and combined reports, persists the corresponding GoAccess databases, updates daily counter values and finally publishes enabled public counters. Logrotate handles the raw log once per day and preserves the uncompressed .1 handover file required by the generator.

The normal operator signal is the service result and journal, not a permanently running process: the generator is a short-lived oneshot service. There is no GoAccess WebSocket process and no public GoAccess port.

External services and local credentials

MaxMind GeoLite2

MaxMind is used only to download and update GeoLite2-Country. The local updater configuration is /etc/GeoIP.conf with mode 0600 root:root; the local database is /var/lib/GeoIP/GeoLite2-Country.mmdb. GeoLite is enabled and the server has an Account ID and License Key, but neither value belongs in Git, this runbook, screenshots, logs or ordinary diagnostic output.

geoipupdate.service and geoipupdate.timer download updates from MaxMind. All visitor lookups then happen locally. The analytics application never sends an individual visitor address to MaxMind.

If the MMDB is missing, stale or unreadable, inspect the timer and journal, check the file mode and run the updater directly if required:

systemctl status geoipupdate.timer geoipupdate.service
journalctl -u geoipupdate.service --since today
sudo stat -c '%U:%G %a %s %y %n' \
  /etc/GeoIP.conf /var/lib/GeoIP/GeoLite2-Country.mmdb
sudo geoipupdate
sudo -u hamradio-analytics test -r \
  /var/lib/GeoIP/GeoLite2-Country.mmdb

When mmdblookup is installed, a lookup of a neutral public test address can confirm database readability without using any visitor address:

sudo -u hamradio-analytics mmdblookup \
  --file /var/lib/GeoIP/GeoLite2-Country.mmdb \
  --ip 1.1.1.1 country iso_code

An unreadable Country MMDB or GoAccess without MMDB support makes --check fail with exit code 2.

Let's Encrypt

Let's Encrypt supplies the TLS certificate for stats.hamradioonline.de. Its state is below /etc/letsencrypt/live/stats.hamradioonline.de/. The ACME webroot is /var/lib/letsencrypt, and the final Nginx vhost permanently serves /.well-known/acme-challenge/ over IPv4 port 80 so webroot renewal continues to work.

Use the Snap client explicitly as /snap/bin/certbot for this vhost. Certificate issuance and a renewal dry-run have been confirmed. The server currently also has an APT Certbot installation and both renewal timers. That duplication is a separate server-maintenance issue; do not change either installation as part of analytics maintenance.

ACME account data, private keys and certificate state live only on the server and in a future protected backup. Validate renewal with:

sudo /snap/bin/certbot renew --dry-run \
  --cert-name stats.hamradioonline.de

GitHub

GitHub supplies the source repository consumed by the website deployment. It is not an analytics processor and receives no individual analytics request or visit data. The deployment credential remains in /etc/kst4contest-website.env; only the server-side deploy process needs it.

No external analytics platform

Statistics are generated locally on the project server. There is no Google Analytics, Matomo Cloud service, external tracking script or transfer of individual visits to an analytics provider.

Troubleshooting

Start with the safe verification sequence above. Keep secrets out of commands and captured output, and do not weaken file modes merely to make a check pass.

`GoAccess JSON report has no geo_location panel`

The expected GoAccess 1.8.1 key is geolocation. An installed older generator which expects geo_location must be replaced with the reviewed repository version.

Missing Virtual Hosts panel

The combined report requires the JSON key vhosts and must start GoAccess with --enable-panel=VIRTUAL_HOSTS. virtual_hosts is not a valid replacement. Site reports do not require this panel.

The names geo_location and virtual_hosts are allowed in repository tests only as deliberately invalid negative cases.

Missing `.1` rotation

This is normal before the first rotation and whenever no previous rotation is present. The current analytics log remains required.

`.gz` rotations on a GoAccess build without Zlib

This is normal. Regular operation does not read .gz files. Do not configure a compressed or rotated file as analyticsLog; any exceptional historical import must be planned separately.

MMDB missing or unreadable

--check must fail. Verify /etc/GeoIP.conf, the GeoIP updater units, /var/lib/GeoIP/GeoLite2-Country.mmdb and the service account's read access. Run geoipupdate and a neutral local lookup as described above when needed.

Analytics log missing or unreadable

Compare the registry path with the additional access_log directive in the site vhost. Check the active Nginx configuration and verify www-data:hamradio-analytics 0640. Test Nginx before reloading it. Do not replace or repurpose the normal operational access log.

sudo nginx -t
sudo nginx -T
sudo stat -c '%U:%G %a %s %y %n' \
  /var/log/nginx/kst4contest-analytics.log
sudo -u hamradio-analytics test -r \
  /var/log/nginx/kst4contest-analytics.log

Public counter returns `404`

This is expected before the first successful generation. Afterwards inspect the service journal, public-output directory, 0644 file mode, Nginx include and exact alias path. A successful response is 200 application/json.

Statistics vhost returns `401` or `404`

401 without credentials is correct. With the valid Basic Auth login, / must redirect to /combined/. A 404 after authentication usually means that the generator has not produced report.html, the URL and report directory do not match, or Nginx cannot traverse/read the report path.

Certificate error

Check the DNS A record, certificate paths, active Nginx vhost, ACME webroot and Snap Certbot renewal. Immediately after an Nginx reload, wait briefly and retry if observations conflict, then inspect nginx -T, the Nginx journal and worker start times.

Generator lock error

Exit code 3 means that the production lock could not be acquired. Check the service, timer and running processes. Do not remove the lock blindly. Remove it only after confirming that no generator process is active and that the lock is genuinely stale.

Counter identity mismatch

If activatedOn or the hostname differs from existing counter state, stop. Do not repair this by deleting the state. Establish the cause and approve a migration or reset explicitly.

Exit code 0 but no output

Check the installed generator size, owner, mode and SHA-256 before anything else. Also look for Analytics generation completed in the journal and inspect the expected output files.

A known operator error is copying a shell prompt or continuation marker > with a command. Bash interprets a stray > as output redirection. This can truncate generate-reports.js to zero bytes and create empty files whose names look like command options. A zero-byte JavaScript file still passes node --check and exits with code 0 while doing nothing.

Identify such artifacts precisely before removing them. Reinstall the generator from the reviewed checkout with the documented owner and mode, compare its size and SHA-256, then repeat --check and --dry-run. Do not make one-off artifact names or old checksums part of the permanent procedure.

Monitoring

The systemd timer is active, and service results and errors are visible through systemd and the journal. No separate alerting channel has been confirmed. Routine checks should confirm:

the timer is active and waiting with a future trigger;
the last service result is success and ExecMainStatus=0;
the journal contains Analytics generation completed;
report and counter timestamps continue to advance;
the public total changes plausibly;
repeated runs do not multiply the existing history.

Central alerting can be handled later as part of the general server operations plan.

Backup and recovery

There is currently no comprehensive automated server backup plan. The analytics installation must be included explicitly when that server-wide backup and recovery design is implemented. Creating that backup system is not part of this repository change.

Data to include

At minimum, the later plan must cover:

/var/lib/hamradioonline-analytics/public-counter-state.json;
/var/lib/hamradioonline-analytics/db;
/etc/hamradioonline-analytics;
the installed systemd units;
Nginx analytics and vhost configuration;
/etc/logrotate.d/hamradioonline-analytics;
/etc/nginx/htpasswd/hamradioonline-analytics;
Certbot and Let's Encrypt account/certificate state;
/etc/GeoIP.conf;
the website deployment configuration and credential storage.

Handle the Basic Auth hash, MaxMind Account ID and License Key, GitHub deploy token, ACME account data and private TLS keys as secrets. Never copy them into Git, public documentation, logs or ordinary support bundles.

The generator is recoverable from GitHub, and GeoLite2-Country can be fetched again with geoipupdate. HTML/JSON reports can be rebuilt when the GoAccess databases or sufficient raw logs remain. The public JSON can be rebuilt from the counter state.

The accumulated public total is not fully recoverable without public-counter-state.json. Older detailed aggregates are not recoverable without the GoAccess databases, and historical raw requests disappear after the 14-day rotation window.

Do not let backups extend the published raw-log retention by accident. Either exclude analytics raw logs from durable backups or enforce the same confirmed retention limit in backup storage. Counter state and anonymised/aggregated GoAccess state can be governed separately.

Recovery order

Install the operating-system packages and external dependencies.
Recreate the service account and group relationships.
Create the directories with the documented owners and modes.
Install the generator and non-secret configuration.
Restore secrets and certificate state from protected backup storage.
Restore GeoLite2-Country or download it again.
Restore the GoAccess databases and public counter state.
Validate Nginx, systemd and Logrotate configuration.
Run the generator with --check.
Run --dry-run and verify that production state remains unchanged.
Start one productive run through the systemd service.
Verify reports, public counter and Basic Auth.
Enable the timer only after every preceding check succeeds.

Outstanding operational checks

The first real rotation of the dedicated analytics log still needs explicit observation. This is not a current service blocker. After rotation, confirm:

a new current log exists;
.1 exists and remains uncompressed;
both files retain the expected owners and modes;
hamradio-analytics can read both files;
the next generator run succeeds;
values do not show duplicate counting;
the older rotation is compressed on the following cycle as intended.

The general server backup/recovery implementation and any central alerting remain separate future operations tasks.

The repository contains no password, password hash, MaxMind credential, deployment token, server IP address, TLS private key or private backup destination. Keep it that way.

README.md

Server-side website statistics