From 821535a24f6d00d943884b0c12f13aa673b41cbb Mon Sep 17 00:00:00 2001 From: Marc Froehlich Date: Sat, 12 Sep 2026 22:31:03 +0200 Subject: [PATCH] Documentation on user counter and analytics to the kst4contest page --- docs/PROJECT_CONTEXT.md | 48 ++- website/ARCHITECTURE.md | 11 +- website/ops/analytics/README.md | 684 +++++++++++++++++++++++++++++--- 3 files changed, 679 insertions(+), 64 deletions(-) diff --git a/docs/PROJECT_CONTEXT.md b/docs/PROJECT_CONTEXT.md index 3eb06475..608014c8 100644 --- a/docs/PROJECT_CONTEXT.md +++ b/docs/PROJECT_CONTEXT.md @@ -1,6 +1,6 @@ # KST4Contest Project Context -Last reviewed: 2026-09-11 +Last reviewed: 2026-09-12 This file is the durable technical project context for KST4Contest. It is not a user manual and not a replacement for the changelog. Current code, tests and authoritative external specifications remain the source of truth when this document is stale or ambiguous. @@ -178,19 +178,39 @@ Current website/deployment scripts and update-feed behaviour must be inspected b ### Server-side website statistics -- The home page can display a public visit total from the same-origin `GET /visitor-count.json` endpoint. The versioned public contract contains only `schemaVersion`, `visits`, `since` and `updatedAt`. -- `visits` is the sum of daily approximate unique visits since `since`, based on GoAccess visitor semantics. It is not a count of unique people and remains separate from page views. -- The home page validates the complete payload, formats the date and number for `en-GB`, and inserts the result through `textContent`. Missing, timed-out, failed or invalid responses leave the initially hidden element invisible and do not affect the rest of the page. -- The display makes no third-party request, sends no credentials and uses no cookies or local storage. The counter request is excluded from the statistic itself. -- The counter endpoint disables its own access log and serves the public JSON with a one-hour public cache policy and `X-Content-Type-Options: nosniff`. -- GoAccess is the server-side source. A registry keeps stable site IDs, hostnames, current analytics-log paths, activation dates, public-counter switches and output targets separate for each project subdomain. The Country database is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. A combined report uses only the registered project sites; `stats.hamradioonline.de` is excluded. -- The regular generator passes each current analytics log and its optional uncompressed `.1` rotation directly to GoAccess and relies on the persistent GoAccess database for incremental processing. Logrotate therefore uses `delaycompress`. Older `.gz` rotations are not imported during regular runs, and missing Zlib support is an accepted, explicitly reported capability state for the GoAccess 1.8.1 production baseline. -- GoAccess 1.8.1 exposes the Country panel as `geolocation` in JSON. Combined report jobs explicitly enable `VIRTUAL_HOSTS` and require the resulting `vhosts` panel; individual site jobs do not enable it. Missing required panels invalidate the complete staged run. -- Node.js 18.19.1 is the production runtime baseline. `--check` requires readable input files, prepared writable output directories and GoAccess built with GeoIP2/MMDB support. OpenSSL and absent Zlib support remain informational. Dry-runs use temporary state and never acquire the production lock. -- The generator runs as `hamradio-analytics`. The state root is mode `0711`; only explicitly prepared report and public-output directories are shared read-only with Nginx through the `www-data` group. GoAccess databases and public counter state remain private. No ACL support is assumed. -- The protected statistics vhost is enabled in two stages: an IPv4-only HTTP bootstrap obtains the certificate through `/snap/bin/certbot`, then the final configuration retains an IPv4 HTTP block for the webroot ACME challenge and permanently redirects all other HTTP requests to HTTPS. The HTTPS block uses the existing Certbot TLS options and redirects authenticated requests from `/` to `/combined/`. IPv6 remains disabled until the DNS AAAA record has been confirmed. -- Dedicated analytics raw logs are retained for 14 days. IP addresses are anonymised before the detailed GoAccess aggregates are persisted for a rolling 395 days. Separate non-personal daily counter values remain available from activation onward so the public total does not shrink with the detailed retention window. -- Repository templates are installed with explicit Unix owners and modes because ZIP metadata created on Windows is not trusted. Production activation, credentials, password hashes, certificate keys, GeoIP acquisition credentials and backups remain outside the repository. +The detailed operator runbook is +[`website/ops/analytics/README.md`](../website/ops/analytics/README.md). This +section records only the durable architecture and operational boundaries. + +#### Architecture and data flow + +- Production runs on Ubuntu Server 24.04 with Nginx 1.24, Node.js 18.19.1 and GoAccess 1.8.1. GoAccess has GeoIP2/MMDB and OpenSSL support but no Zlib support; missing Zlib is intentional for this operating model. +- Nginx writes eligible `GET` page requests to a reduced, tab-separated analytics log in addition to the normal operational log. It records server name, client IP, time, method, normalized path without query string, protocol, status, bytes and user agent; referrer and remote user are omitted. +- Nginx removes assets, downloads, status/monitoring paths, update feed, counter requests and known crawler clients before logging. GoAccess supplies the second crawler-classification layer, anonymises IP addresses at the configured level and performs Country lookup locally through GeoLite2-Country. +- The hourly systemd timer starts the Node.js generator. Each registered site is processed from its optional uncompressed `.1` rotation followed by the current log; older `.gz` files are not part of regular processing. Logrotate retains raw analytics logs for 14 days and requires `delaycompress` for this handover. +- GoAccess maintains one private report/database per registered site and one combined report/database. The generator then updates its separate daily counter state and publishes `visitor-count.json` for enabled sites. Private reports are served with Basic Auth from `stats.hamradioonline.de`; the public home page requests only its same-origin counter file. + +#### Durable invariants and persistence + +- GoAccess defines a visit by IP address, date and user agent. The public value is therefore an approximate visit total, not a count of unique people, and remains separate from page views. +- The public contract contains only `schemaVersion`, `visits`, `since` and `updatedAt`. The browser uses no analytics cookie, local storage or third-party request; failed or invalid responses leave the counter hidden. +- The site registry at `/etc/hamradioonline-analytics/sites.json` owns stable site IDs, hostnames, current uncompressed log paths, activation dates and output targets. `stats.hamradioonline.de` is never registered as an analytics site. Existing counter state cannot be continued with a changed hostname or `activatedOn` without an explicit migration or reset decision. +- GoAccess databases below `/var/lib/hamradioonline-analytics/db` retain rolling 395-day detail through `--persist` and `--restore`. `/var/lib/hamradioonline-analytics/public-counter-state.json` retains daily values from activation onward and is the durable business source for the lifetime public total. Reports and public JSON files are derived outputs. +- GoAccess 1.8.1 exposes Country data as `geolocation`; combined jobs explicitly enable `VIRTUAL_HOSTS` and require `vhosts`. Individual site jobs do not enable that panel. `geo_location` and `virtual_hosts` are invalid interface names. +- Generator inputs and staged GoAccess outputs are validated before publication. Files are replaced atomically, dry-runs use temporary state without the production lock, and productive runs use `/run/hamradioonline-analytics/generator.lock`. Exit codes are 1 for runtime/publication errors, 2 for configuration errors and 3 when the production lock cannot be acquired. + +#### Components, roles and external services + +- `hamradio-analytics` is the locked, non-login service account which reads configuration, Country MMDB and analytics logs and writes service state. Nginx runs as `www-data`, writes the analytics log and can read only reports and the public counter, not private databases or counter state. `stats-reader` is a local Nginx Basic Auth login whose password file remains outside Git. +- Root-managed program/configuration lives below `/opt/hamradioonline-analytics` and `/etc/hamradioonline-analytics`; service state lives below `/var/lib/hamradioonline-analytics`. Report/public directories are shared read-only with Nginx through group ownership; no ACL dependency exists. +- MaxMind is used only by `geoipupdate` to download GeoLite2-Country. Visitor lookups are local; Account ID and License Key remain in the protected server configuration. Let's Encrypt supplies TLS for the statistics vhost through the permanent ACME webroot and `/snap/bin/certbot`. GitHub supplies website source and deployment, not analytics processing. +- The website deploy checks out `origin/main`, builds the static site and publishes it to the Nginx document root. It does not install or overwrite analytics programs/configuration, systemd, Nginx, Logrotate, GeoIP, Basic Auth or Certbot state; those remain separate manual server operations. +- Passwords, hashes, deployment credentials, MaxMind credentials, ACME account data and private TLS keys must remain outside the repository and ordinary diagnostic output. + +#### Open operational work + +- The server has no comprehensive automated backup plan yet. A future server-wide plan must include the non-regenerable counter state, GoAccess databases and protected operational configuration without extending the published 14-day raw-log retention. +- The first real rotation of the dedicated analytics log still requires an explicit operational check of `.1`, ownership/readability, subsequent generator success and absence of duplicate counting. This is not a current service blocker. ## Important Decisions and Workarounds diff --git a/website/ARCHITECTURE.md b/website/ARCHITECTURE.md index dd01315c..712a125b 100644 --- a/website/ARCHITECTURE.md +++ b/website/ARCHITECTURE.md @@ -48,4 +48,13 @@ Nginx serves this directory. ## Build Strategy -GitHub Actions will later build the 11ty website and deploy the generated output to the server. \ No newline at end of file +The production server keeps a checkout at `/srv/git/kst4contest`. A root cron +job runs `/srv/scripts/deploy-kst4contest-website.sh` every five minutes. The +script resets the checkout to `origin/main`, installs the locked npm +dependencies, builds and validates the Eleventy site, and synchronises the +result to `/srv/www/kst4contest/current` with the ownership required by Nginx. + +This deploy path updates only the static website. Server-side analytics, +Nginx, systemd, Logrotate, GeoIP, Basic Auth and Certbot configuration are +installed and maintained separately as described in +[`ops/analytics/README.md`](ops/analytics/README.md). diff --git a/website/ops/analytics/README.md b/website/ops/analytics/README.md index fea06f52..7555d8fa 100644 --- a/website/ops/analytics/README.md +++ b/website/ops/analytics/README.md @@ -1,8 +1,9 @@ # Server-side website statistics -This directory contains installation examples for the confirmed Ubuntu 24.04 -server baseline and privacy-conscious traffic statistics. Nothing here -installs or activates the production service automatically. +This directory contains the maintained installation templates and the operator +runbook for the production analytics service on Ubuntu Server 24.04. Nothing in +the repository installs, updates or activates the server-side components +automatically. The design has two separate outputs: @@ -31,6 +32,11 @@ GoAccess with IP anonymisation | v durable daily counter state --> public visitor-count.json + | + v + same-origin home-page request + +private HTML reports --> Nginx Basic Auth --> stats.hamradioonline.de ``` Nginx writes a dedicated, reduced log. For each site, the generator gives the @@ -48,6 +54,22 @@ being added again. This makes repeated runs idempotent. Values older than 395 days remain in the counter state and continue to contribute to the public total. +## Production platform + +The confirmed production baseline is: + +- Ubuntu Server 24.04; +- Nginx 1.24; +- Node.js 18.19.1; +- GoAccess 1.8.1 with GeoIP2/MMDB and OpenSSL support, but without Zlib; +- a local GeoLite2-Country database; +- systemd for the oneshot generator and its hourly timer; +- Logrotate for the dedicated analytics log. + +Missing Zlib support is intentional for this operating model. The generator +processes the current uncompressed analytics log and the optional uncompressed +`.1` rotation. It does not process `.gz` files during regular operation. + ## Files - `generate-reports.js` validates configuration and state, runs GoAccess and @@ -60,6 +82,18 @@ total. - `systemd/` contains a hardened oneshot service and hourly timer. - `logrotate/` retains 14 daily analytics-log rotations. +These repository files are templates and source files. Their productive +counterparts are installed separately: + +- generator: `/opt/hamradioonline-analytics/generate-reports.js`; +- registry: `/etc/hamradioonline-analytics/sites.json`; +- GoAccess template: + `/etc/hamradioonline-analytics/goaccess.conf.template`; +- service and timer: `/etc/systemd/system/hamradioonline-analytics.service` + and `/etc/systemd/system/hamradioonline-analytics.timer`; +- Logrotate configuration: `/etc/logrotate.d/hamradioonline-analytics`; +- service state and derived outputs: `/var/lib/hamradioonline-analytics`. + ## Prerequisites - Node.js 18.19.1 or newer; @@ -87,6 +121,24 @@ report and public-output directories. Access is group-based. The setup does not depend on ACLs or `setfacl`. Nginx receives read access to reports and the public counter through the `www-data` group, but no write access. +## Accounts and roles + +`hamradio-analytics` is a local system account with its own group, no regular +home directory, `/usr/sbin/nologin` as its shell and a locked password. It runs +the generator and GoAccess, reads the reduced logs and configuration, and +writes only below the configured service-state directories. + +Nginx runs as `www-data`. It writes the dedicated analytics log and reads the +private reports and public counter. It must not be able to read the private +GoAccess databases or `public-counter-state.json`, and it has no write access +to generated output. + +`stats-reader` is the current local Nginx Basic Auth username. It is not a +Linux service account and not an account with an external analytics provider. +Its password file is `/etc/nginx/htpasswd/hamradioonline-analytics`. Store the +password and hash only on the server and in a future protected backup, never in +Git, this runbook, screenshots or support output. + ## Installation and permissions Do not trust Unix modes stored in a ZIP created on Windows. Install every @@ -149,6 +201,31 @@ Generated HTML and JSON reports use mode `0640`. The public by Nginx. The service unit uses `StateDirectoryMode=0711` to retain this boundary after systemd has prepared the state directory. +The productive ownership and mode boundaries are: + +- generator: `0750 root:hamradio-analytics`; +- registry and GoAccess configuration: `0640 root:hamradio-analytics`; +- report directories, including `reports/combined` and + `reports/kst4contest`: `2750 hamradio-analytics:www-data`; +- report files: `0640 hamradio-analytics:www-data`; +- private database files: `0640 hamradio-analytics:hamradio-analytics`; +- `public-counter-state.json`: mode `0640`, owner and group + `hamradio-analytics:hamradio-analytics`; +- public output directories, including `public/kst4contest`: mode `2750`, + owner and group `hamradio-analytics:www-data`; +- public `visitor-count.json`: `0644 hamradio-analytics:www-data`; +- Basic Auth password file: `0640 root:www-data`. + +`/opt/hamradioonline-analytics` and `/etc/hamradioonline-analytics` are managed +by root and readable by the service group. `/var/lib/hamradioonline-analytics` +belongs to the service account. Its `0711` root allows Nginx to traverse known +paths without listing or reading private state. The service has a restrictive +`UMask=0027`; the generator therefore sets the public counter to `0644` +explicitly so Nginx can serve it. Missing output directories are a +configuration error. Create them deliberately with the shared group and +set-group-ID modes above rather than relying on recursive creation with an +unsuitable group. + Create the analytics log only when it does not already exist. Running `install /dev/null` unconditionally would empty an existing log: @@ -192,12 +269,31 @@ Treat `activatedOn` as persistent data. Once counting has started, changing it would change the meaning of the total. The generator refuses to combine a new activation date with existing counter state. +The hostname is persistent identity as well. If an existing site state has a +different `activatedOn` or hostname, do not delete the state to make the next +run pass. Changing either value requires a deliberate migration or a +specifically approved reset of the public count. + The generator derives the optional `.1` path from `analyticsLog`. It is valid for `.1` not to exist before the first rotation. Do not enter a rotation or a compressed `.gz` file in the registry. +Adding another project subdomain also requires its own Nginx analytics log, +the corresponding Logrotate ownership, a prepared report directory and, when +enabled, a public-output directory and counter location. The combined report +uses the logs of every registered project site. The statistics vhost remains +outside the registry. + ## Nginx logging +The relevant production configuration files are: + +- `/etc/nginx/snippets/hamradioonline-analytics-filters.conf`; +- `/etc/nginx/conf.d/hamradioonline-analytics-log.conf`; +- `/etc/nginx/snippets/kst4contest-public-counter.conf`; +- `/etc/nginx/sites-available/kst4contest.conf`; +- `/etc/nginx/sites-available/stats.hamradioonline.de`. + Install the log-format and filter maps from `nginx/` in the `http` context. Then add a dedicated analytics `access_log` to every registered project server block. Keep the existing operational access log unless its replacement has @@ -237,12 +333,24 @@ The analytics format contains only: - transferred body size; - user agent. -It does not contain a referrer, query string or authenticated user name. The -filter accepts only eligible page `GET` requests. It excludes the update feed, -public counter, sitemap, robots file, favicons, CSS, JavaScript, images, fonts, -source maps, manual assets and the listed monitoring paths. Known crawler user -agents are rejected before logging. GoAccess applies its own crawler list as a -second layer and treats unknown browsers or operating systems as crawlers. +The fields are tab-separated. The production log is +`/var/log/nginx/kst4contest-analytics.log`. Nginx writes it as `www-data`; the +`hamradio-analytics` group can read it. The confirmed owner and mode are +`www-data:hamradio-analytics 0640`. The analytics service receives read-only +access and must never truncate or otherwise modify this log. + +The format uses `$uri`, not `$request_uri`, so query strings never enter the +analytics log. It also omits referrer and authenticated remote-user data. The +user agent is retained because GoAccess needs it for crawler classification +and its visit definition. + +Only eligible `GET` requests can be logged. Assets, downloads, status and +monitoring paths, sitemap, robots file, favicons, the update feed and the +public counter endpoint are excluded. Known bots, crawlers, monitoring +clients, `wget` and `curl` are rejected before logging. GoAccess applies its +own crawler list as a second layer and treats unknown browser or operating +system combinations as crawlers. The public counter request therefore cannot +count itself, and the statistics vhost has no analytics logging of its own. Review the monitoring-path list against the real server before activation. When a new health endpoint or asset family is added, update the filter first. @@ -253,12 +361,32 @@ Test the complete Nginx configuration before reloading it: sudo nginx -t ``` +### Log rotation + +`/etc/logrotate.d/hamradioonline-analytics` rotates the dedicated analytics +logs daily, retains 14 rotations and compresses older files. `delaycompress` +is an operational requirement: it keeps the immediately preceding rotation +as an uncompressed `.1` file for the next generator run. The `create 0640 +www-data hamradio-analytics` directive preserves the write/read boundary. +After rotation, `invoke-rc.d nginx rotate` makes Nginx reopen its logs. + +The generator processes, in this order: + +1. the optional, uncompressed `.1` rotation; +2. the current analytics log. + +A missing `.1` is normal, including before the first rotation. Older `.gz` +files are retained according to Logrotate but are not imported by the regular +generator. A registry path that identifies `.1`, another numbered rotation or +a `.gz` file is rejected. + ## GoAccess reports -The template enables IP anonymisation before persistent aggregation, ignores -crawlers, keeps 395 days, and uses a separate persistent database for every -site and the combined report. It leaves only the panels needed here: visits by -day, requested pages, countries, HTTP status codes and virtual hosts. Host, +The template enables IP anonymisation at `anonymize-level 2` before persistent +aggregation, ignores crawlers, keeps 395 days, and uses a separate persistent +database for every site and the combined report. It leaves only the panels +needed here: visits by day, requested pages, countries, HTTP status codes and +virtual hosts. Host, remote-user, referrer, keyphrase, operating-system, browser and other detailed panels are disabled. @@ -268,6 +396,12 @@ resulting `vhosts` key. Site jobs do not enable that panel. The generator treats either missing key as an invalid report rather than publishing incomplete statistics. +The GoAccess 1.8.1 JSON keys are an interface invariant. `geo_location` and +`virtual_hosts` are invalid names and may appear in repository tests only as +deliberately rejected negative cases. The Country panel comes from the +configured MMDB; missing `geolocation` invalidates every report, and missing +`vhosts` invalidates the combined report. + The Country database is provided through the registry at `/var/lib/GeoIP/GeoLite2-Country.mmdb`. A file whose name contains `City` is rejected. Do not replace it with a City database merely because one happens to @@ -294,8 +428,100 @@ window, the regular run cannot recover entries found only in older `.gz` files. Preserve those files under the raw-log retention policy and plan any necessary historical import separately before resuming normal processing. +### Visit and privacy boundary + +GoAccess treats requests with the same IP address, date and user agent as one +visit. The public number is therefore an approximate visit total, not a count +of uniquely identified people. Page views remain a separate statistic. + +IP addresses are processed with the configured GoAccess anonymisation level. +Country resolution happens locally against GeoLite2-Country; City and host +statistics are not produced. No visitor address is sent to MaxMind or another +analytics service. The website sets no analytics cookie, embeds no external +tracking script and sends no visit to Google Analytics, Matomo Cloud or any +other analytics platform. + +The home-page script requests only `/visitor-count.json` from the same origin. +The public file contains `schemaVersion`, `visits`, `since` and `updatedAt` and +no visitor-level or daily detail. + +## Persistence and publication + +The installation has four distinct persistence layers. + +### Raw logs + +The current analytics log and its rotations are short-lived input. The current +file and `.1` bridge requests across the most recent rotation. Logrotate limits +raw-log retention to the published 14-day policy; backups must not silently +extend that period. + +### GoAccess databases + +Persistent detail state lives in: + +- `/var/lib/hamradioonline-analytics/db/kst4contest`; +- `/var/lib/hamradioonline-analytics/db/combined`. + +`--persist` writes the processing state and `--restore` loads it on later runs. +This lets GoAccess process new log content without adding the same input from +scratch on every hourly run. The `keep-last 395` setting limits detailed +aggregates to a rolling 395 days. + +### Public counter state + +`/var/lib/hamradioonline-analytics/public-counter-state.json` stores daily +visit values for each enabled site from `activatedOn` onward. The generator +replaces a day's value when it is processed again; it does not add the value a +second time. This file is the durable business source for the lifetime public +total, including days which have aged out of the GoAccess detail database. + +The stored hostname and `activatedOn` must continue to match the registry. +Changing either value requires a planned migration or an approved reset, not +an ad-hoc edit or deletion of the state file. + +### Reports and public files + +The derived outputs are: + +- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.html`; +- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.json`; +- `/var/lib/hamradioonline-analytics/reports/combined/report.html`; +- `/var/lib/hamradioonline-analytics/reports/combined/report.json`; +- `/var/lib/hamradioonline-analytics/public/kst4contest/visitor-count.json`. + +The generator prepares every GoAccess job in a run directory, validates the +HTML and JSON outputs, and calculates counter updates before publication. A +GoAccess or report-validation failure therefore leaves the published files +unchanged. Report, state and public files are replaced atomically one file at a +time. Database directories are exchanged through a temporary backup name and +restored if that exchange fails. These replacements are not one filesystem +transaction across every report, database and counter file; after a storage or +permission failure during publication, inspect the complete set and rerun the +service after correcting the cause. + +Counter state and GoAccess databases are the important persistent sources. +HTML/JSON reports and `visitor-count.json` are derived and can be rebuilt when +their corresponding source state is available. + ## Checking and running +Do not rely on `node --check` alone. First inspect the installed generator's +owner, mode and plausible non-zero size, then compare its SHA-256 digest with +the reviewed repository file: + +```sh +sudo stat -c '%U:%G %a %s %n' \ + /opt/hamradioonline-analytics/generate-reports.js +sha256sum /srv/git/kst4contest/website/ops/analytics/generate-reports.js \ + /opt/hamradioonline-analytics/generate-reports.js +/usr/bin/node --check /opt/hamradioonline-analytics/generate-reports.js +``` + +A zero-byte JavaScript file is syntactically valid and exits successfully +without doing any work. File size, digest and the expected completion message +are therefore part of every recovery check. + Validate paths, registry values, the template contract and GoAccess availability without producing reports: @@ -328,8 +554,8 @@ sudo -u hamradio-analytics /usr/bin/node \ Run without either flag to publish. A lock prevents concurrent production runs. A dry-run uses a temporary working directory and deliberately neither needs nor creates the production lock below `/run`. Configuration errors use -exit code 2, an active production lock uses exit code 3, and generation or -publication errors use exit code 1. +exit code 2, failure to acquire the production lock uses exit code 3, and +generation or publication errors use exit code 1. Before the first production run, seed any earlier daily values which must be preserved into `public-counter-state.json`. There is no honest way to recreate @@ -337,15 +563,69 @@ history which is no longer present in the raw logs. Back up this state file: it is the durable source for public totals older than the detailed retention window. -## Scheduling and report access +### Safe verification sequence + +Use this order for a new installation, a recovered service or a material +generator/configuration update: + +1. Check generator size, ownership, mode and SHA-256 against the reviewed + checkout. +2. Run `/usr/bin/node --check` on the installed generator. +3. Run the generator with `--check` as `hamradio-analytics`. +4. Confirm the reported GoAccess version and GeoIP2/MMDB, OpenSSL and Zlib + capability state. Missing Zlib is expected; missing MMDB support is not. +5. Record the hashes and timestamps of current reports, databases and counter + files, then run `--dry-run`. +6. Confirm that the recorded production files did not change and that + `/run/hamradioonline-analytics/generator.lock` was not created by the + dry-run. +7. Start one productive run through the service unit: + `sudo systemctl start hamradioonline-analytics.service`. +8. Inspect `Result` and `ExecMainStatus` and read the unit journal. A successful + run ends with `Analytics generation completed`. +9. Check every generated file's path, owner, group, mode and timestamp. +10. Request the public JSON through HTTPS and validate its four fields. +11. Request both private report URLs with Basic Auth. Let the client prompt for + the password; never put it directly on a command line. +12. Run the service a second time and confirm that the total and report values + develop plausibly rather than multiplying the existing history. +13. Enable or re-enable the timer only after these checks pass. +14. Confirm the first automatic run in the journal and later verify the first + real log rotation separately. + +Useful service checks are: + +```sh +systemctl show hamradioonline-analytics.service \ + -p Result -p ExecMainStatus +journalctl -u hamradioonline-analytics.service --since today +systemctl status hamradioonline-analytics.timer +systemctl list-timers hamradioonline-analytics.timer +``` + +## systemd operation + +The production oneshot service runs as `hamradio-analytics` with +`UMask=0027`. Its sandbox exposes `/etc/hamradioonline-analytics`, +`/opt/hamradioonline-analytics`, the Nginx logs and the Country MMDB read-only. +`/var/lib/hamradioonline-analytics` is its only application-state write area; +`/run/hamradioonline-analytics` holds the production lock +`generator.lock`. The service has no network access and only the documented +read/write paths. + +The timer uses `OnCalendar=hourly`, `Persistent=true`, +`RandomizedDelaySec=4m` and `AccuracySec=1m` and is permanently enabled in +production. Multiple automatic hourly runs have completed successfully. Each +run produces one site report, one combined report and one public counter, +takes roughly one second on the current installation and has shown stable +incremental behaviour without sudden duplicate counting. -Install the systemd files as local units after adapting paths and permissions. -The timer runs hourly, catches up after downtime and adds a small random delay. -The service has no network access and only the documented read/write paths. If the installed GoAccess build unexpectedly requires network access, find the reason before weakening that restriction; local log processing and a local Country database do not require it. +## Report access + The statistics vhost serves static files over HTTPS and protects the complete host with HTTP Basic Authentication. This includes `/`, its redirect to `/combined/`, and every individual report. Store the password file outside @@ -362,14 +642,33 @@ sudo chown root:www-data /etc/nginx/htpasswd/hamradioonline-analytics sudo chmod 0640 /etc/nginx/htpasswd/hamradioonline-analytics ``` -Choose the account name locally and enter the password interactively. Never -store the resulting password hash in this repository or the installation ZIP. +Use the local account name `stats-reader` and enter its password interactively. +Never store the resulting password hash in this repository or an installation +ZIP. The example opens no GoAccess WebSocket and no additional GoAccess port. Its own access log is disabled and responses use a private, no-store cache policy. -Do not activate the final HTTPS vhost before its certificate files exist. -First install the temporary HTTP bootstrap without changing the parallel apt -Certbot installation or either renewal timer: +The current private endpoints are: + +- `https://stats.hamradioonline.de/`, which redirects an authenticated request + to `/combined/`; +- `https://stats.hamradioonline.de/combined/`, which serves the combined + report; +- `https://stats.hamradioonline.de/kst4contest/`, which serves the site report. + +All HTTPS paths, including the redirect target, remain behind Basic Auth. +Reports use `Cache-Control: private, no-store`, +`X-Content-Type-Options: nosniff` and `X-Frame-Options: DENY`; dotfiles are +blocked. The vhost has no access log and no analytics log. HTTP remains open +only for the ACME webroot and permanently redirects every other request to +HTTPS. No IPv6 listener is configured while DNS AAAA operation remains +unconfirmed. + +The final HTTP/HTTPS vhost is the production configuration. The HTTP-only +bootstrap template is retained solely for first provisioning or recovery when +the certificate files do not yet exist. In that situation, do not activate the +final HTTPS vhost first. Install the temporary bootstrap without changing the +parallel APT Certbot installation or either renewal timer: ```sh sudo install -d -o root -g root -m 0755 /var/lib/letsencrypt @@ -430,34 +729,321 @@ own access log is disabled. Responses use activation date and update time. It contains no IP address, user agent, hostname or per-day detail. -Use this rollout order: +The production URL is +`https://kst4contest.hamradioonline.de/visitor-count.json`. Before the first +successful generator run, a `404` is expected; afterwards it must return +`200` with `application/json`. The home page reveals the counter only after a +valid response. A missing, invalid or unavailable counter never prevents the +rest of the static site from working. -1. Install the server files with explicit modes. Prepare directories, log - ownership, filters and still-inactive Nginx and systemd configuration. -2. Push the website changes, including the Privacy Policy, and let the existing - deployment cron job publish them. Until the JSON endpoint exists, the - visitor count remains hidden automatically. -3. Verify the published Privacy Policy. Only then activate analytics logging, - the report generator and timer, the public counter location and the - protected statistics vhost. -4. Run `nginx -t` before every Nginx reload and perform the real `--check` and - `--dry-run` on the server before the first production generation. +## Deployment boundary -A short period in which the Privacy Policy is already visible but logging is -not yet active is acceptable. Starting analytics logging before publishing the -updated policy is not. +GitHub is the source repository for the website and the reviewed analytics +templates. The production checkout is `/srv/git/kst4contest`. A root cron job +runs `/srv/scripts/deploy-kst4contest-website.sh` every five minutes. It fetches +Git, resets the checkout to `origin/main`, runs `npm ci`, builds the Eleventy +website, validates VersionInfo, synchronises the result to +`/srv/www/kst4contest/current` and finally restores the ownership expected by +Nginx. -## Retention and recovery +The deployment credential is stored in `/etc/kst4contest-website.env`. Its +value and account assignment must never appear in documentation, logs or +support output. -- Dedicated analytics raw logs: 14 days through the Logrotate example. -- Anonymised detailed GoAccess aggregates: rolling 395 days. -- Public daily counter values: retained from activation onward. +This automatic deployment updates only the static website. It does not install +or overwrite: -Back up the counter state and, if fast report recovery matters, the GoAccess -database directories. Reports themselves are derived output. To recover, stop -the timer, restore the state and database directories with their ownership, -run `--check`, then run `--dry-run` before publishing again. +- `/opt/hamradioonline-analytics`; +- `/etc/hamradioonline-analytics`; +- systemd units; +- Nginx or Logrotate configuration; +- the Basic Auth password file; +- GeoIP configuration; +- Certbot configuration. -The repository contains no password, password hash, MaxMind download key, -server IP address, TLS private key or private backup destination. Keep it that -way. +Changes to those operational files require a separate review and manual +installation. The published privacy notice must remain in place whenever +analytics logging is active. + +## Regular operation + +Nginx continuously writes only eligible requests to the dedicated analytics +log. The systemd timer starts the generator once per hour. Every successful +run refreshes the per-site and combined reports, persists the corresponding +GoAccess databases, updates daily counter values and finally publishes enabled +public counters. Logrotate handles the raw log once per day and preserves the +uncompressed `.1` handover file required by the generator. + +The normal operator signal is the service result and journal, not a permanently +running process: the generator is a short-lived oneshot service. There is no +GoAccess WebSocket process and no public GoAccess port. + +## External services and local credentials + +### MaxMind GeoLite2 + +MaxMind is used only to download and update GeoLite2-Country. The local updater +configuration is `/etc/GeoIP.conf` with mode `0600 root:root`; the local +database is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. GeoLite is enabled and the +server has an Account ID and License Key, but neither value belongs in Git, +this runbook, screenshots, logs or ordinary diagnostic output. + +`geoipupdate.service` and `geoipupdate.timer` download updates from MaxMind. +All visitor lookups then happen locally. The analytics application never sends +an individual visitor address to MaxMind. + +If the MMDB is missing, stale or unreadable, inspect the timer and journal, +check the file mode and run the updater directly if required: + +```sh +systemctl status geoipupdate.timer geoipupdate.service +journalctl -u geoipupdate.service --since today +sudo stat -c '%U:%G %a %s %y %n' \ + /etc/GeoIP.conf /var/lib/GeoIP/GeoLite2-Country.mmdb +sudo geoipupdate +sudo -u hamradio-analytics test -r \ + /var/lib/GeoIP/GeoLite2-Country.mmdb +``` + +When `mmdblookup` is installed, a lookup of a neutral public test address can +confirm database readability without using any visitor address: + +```sh +sudo -u hamradio-analytics mmdblookup \ + --file /var/lib/GeoIP/GeoLite2-Country.mmdb \ + --ip 1.1.1.1 country iso_code +``` + +An unreadable Country MMDB or GoAccess without MMDB support makes `--check` +fail with exit code 2. + +### Let's Encrypt + +Let's Encrypt supplies the TLS certificate for `stats.hamradioonline.de`. Its +state is below `/etc/letsencrypt/live/stats.hamradioonline.de/`. The ACME +webroot is `/var/lib/letsencrypt`, and the final Nginx vhost permanently serves +`/.well-known/acme-challenge/` over IPv4 port 80 so webroot renewal continues +to work. + +Use the Snap client explicitly as `/snap/bin/certbot` for this vhost. +Certificate issuance and a renewal dry-run have been confirmed. The server +currently also has an APT Certbot installation and both renewal timers. That +duplication is a separate server-maintenance issue; do not change either +installation as part of analytics maintenance. + +ACME account data, private keys and certificate state live only on the server +and in a future protected backup. Validate renewal with: + +```sh +sudo /snap/bin/certbot renew --dry-run \ + --cert-name stats.hamradioonline.de +``` + +### GitHub + +GitHub supplies the source repository consumed by the website deployment. It +is not an analytics processor and receives no individual analytics request or +visit data. The deployment credential remains in +`/etc/kst4contest-website.env`; only the server-side deploy process needs it. + +### No external analytics platform + +Statistics are generated locally on the project server. There is no Google +Analytics, Matomo Cloud service, external tracking script or transfer of +individual visits to an analytics provider. + +## Troubleshooting + +Start with the safe verification sequence above. Keep secrets out of commands +and captured output, and do not weaken file modes merely to make a check pass. + +### `GoAccess JSON report has no geo_location panel` + +The expected GoAccess 1.8.1 key is `geolocation`. An installed older generator +which expects `geo_location` must be replaced with the reviewed repository +version. + +### Missing Virtual Hosts panel + +The combined report requires the JSON key `vhosts` and must start GoAccess with +`--enable-panel=VIRTUAL_HOSTS`. `virtual_hosts` is not a valid replacement. +Site reports do not require this panel. + +The names `geo_location` and `virtual_hosts` are allowed in repository tests +only as deliberately invalid negative cases. + +### Missing `.1` rotation + +This is normal before the first rotation and whenever no previous rotation is +present. The current analytics log remains required. + +### `.gz` rotations on a GoAccess build without Zlib + +This is normal. Regular operation does not read `.gz` files. Do not configure +a compressed or rotated file as `analyticsLog`; any exceptional historical +import must be planned separately. + +### MMDB missing or unreadable + +`--check` must fail. Verify `/etc/GeoIP.conf`, the GeoIP updater units, +`/var/lib/GeoIP/GeoLite2-Country.mmdb` and the service account's read access. +Run `geoipupdate` and a neutral local lookup as described above when needed. + +### Analytics log missing or unreadable + +Compare the registry path with the additional `access_log` directive in the +site vhost. Check the active Nginx configuration and verify +`www-data:hamradio-analytics 0640`. Test Nginx before reloading it. Do not +replace or repurpose the normal operational access log. + +```sh +sudo nginx -t +sudo nginx -T +sudo stat -c '%U:%G %a %s %y %n' \ + /var/log/nginx/kst4contest-analytics.log +sudo -u hamradio-analytics test -r \ + /var/log/nginx/kst4contest-analytics.log +``` + +### Public counter returns `404` + +This is expected before the first successful generation. Afterwards inspect +the service journal, public-output directory, `0644` file mode, Nginx include +and exact alias path. A successful response is `200 application/json`. + +### Statistics vhost returns `401` or `404` + +`401` without credentials is correct. With the valid Basic Auth login, `/` +must redirect to `/combined/`. A `404` after authentication usually means that +the generator has not produced `report.html`, the URL and report directory do +not match, or Nginx cannot traverse/read the report path. + +### Certificate error + +Check the DNS A record, certificate paths, active Nginx vhost, ACME webroot and +Snap Certbot renewal. Immediately after an Nginx reload, wait briefly and +retry if observations conflict, then inspect `nginx -T`, the Nginx journal and +worker start times. + +### Generator lock error + +Exit code 3 means that the production lock could not be acquired. Check the +service, timer and running processes. Do not remove the lock blindly. Remove it +only after confirming that no generator process is active and that the lock is +genuinely stale. + +### Counter identity mismatch + +If `activatedOn` or the hostname differs from existing counter state, stop. +Do not repair this by deleting the state. Establish the cause and approve a +migration or reset explicitly. + +### Exit code 0 but no output + +Check the installed generator size, owner, mode and SHA-256 before anything +else. Also look for `Analytics generation completed` in the journal and inspect +the expected output files. + +A known operator error is copying a shell prompt or continuation marker `>` +with a command. Bash interprets a stray `>` as output redirection. This can +truncate `generate-reports.js` to zero bytes and create empty files whose names +look like command options. A zero-byte JavaScript file still passes +`node --check` and exits with code 0 while doing nothing. + +Identify such artifacts precisely before removing them. Reinstall the +generator from the reviewed checkout with the documented owner and mode, +compare its size and SHA-256, then repeat `--check` and `--dry-run`. Do not make +one-off artifact names or old checksums part of the permanent procedure. + +## Monitoring + +The systemd timer is active, and service results and errors are visible through +systemd and the journal. No separate alerting channel has been confirmed. +Routine checks should confirm: + +- the timer is active and waiting with a future trigger; +- the last service result is `success` and `ExecMainStatus=0`; +- the journal contains `Analytics generation completed`; +- report and counter timestamps continue to advance; +- the public total changes plausibly; +- repeated runs do not multiply the existing history. + +Central alerting can be handled later as part of the general server operations +plan. + +## Backup and recovery + +There is currently no comprehensive automated server backup plan. The +analytics installation must be included explicitly when that server-wide +backup and recovery design is implemented. Creating that backup system is not +part of this repository change. + +### Data to include + +At minimum, the later plan must cover: + +- `/var/lib/hamradioonline-analytics/public-counter-state.json`; +- `/var/lib/hamradioonline-analytics/db`; +- `/etc/hamradioonline-analytics`; +- the installed systemd units; +- Nginx analytics and vhost configuration; +- `/etc/logrotate.d/hamradioonline-analytics`; +- `/etc/nginx/htpasswd/hamradioonline-analytics`; +- Certbot and Let's Encrypt account/certificate state; +- `/etc/GeoIP.conf`; +- the website deployment configuration and credential storage. + +Handle the Basic Auth hash, MaxMind Account ID and License Key, GitHub deploy +token, ACME account data and private TLS keys as secrets. Never copy them into +Git, public documentation, logs or ordinary support bundles. + +The generator is recoverable from GitHub, and GeoLite2-Country can be fetched +again with `geoipupdate`. HTML/JSON reports can be rebuilt when the GoAccess +databases or sufficient raw logs remain. The public JSON can be rebuilt from +the counter state. + +The accumulated public total is not fully recoverable without +`public-counter-state.json`. Older detailed aggregates are not recoverable +without the GoAccess databases, and historical raw requests disappear after +the 14-day rotation window. + +Do not let backups extend the published raw-log retention by accident. Either +exclude analytics raw logs from durable backups or enforce the same confirmed +retention limit in backup storage. Counter state and anonymised/aggregated +GoAccess state can be governed separately. + +### Recovery order + +1. Install the operating-system packages and external dependencies. +2. Recreate the service account and group relationships. +3. Create the directories with the documented owners and modes. +4. Install the generator and non-secret configuration. +5. Restore secrets and certificate state from protected backup storage. +6. Restore GeoLite2-Country or download it again. +7. Restore the GoAccess databases and public counter state. +8. Validate Nginx, systemd and Logrotate configuration. +9. Run the generator with `--check`. +10. Run `--dry-run` and verify that production state remains unchanged. +11. Start one productive run through the systemd service. +12. Verify reports, public counter and Basic Auth. +13. Enable the timer only after every preceding check succeeds. + +## Outstanding operational checks + +The first real rotation of the dedicated analytics log still needs explicit +observation. This is not a current service blocker. After rotation, confirm: + +- a new current log exists; +- `.1` exists and remains uncompressed; +- both files retain the expected owners and modes; +- `hamradio-analytics` can read both files; +- the next generator run succeeds; +- values do not show duplicate counting; +- the older rotation is compressed on the following cycle as intended. + +The general server backup/recovery implementation and any central alerting +remain separate future operations tasks. + +The repository contains no password, password hash, MaxMind credential, +deployment token, server IP address, TLS private key or private backup +destination. Keep it that way.