Rsclub2_2/kst4contest
1
0
Fork 0
mirror of https://github.com/praktimarc/kst4contest.git synced 2026-09-13 04:35:35 +02:00
Code Issues Packages Projects Releases Wiki Activity

Documentation on user counter and analytics to the kst4contest page

Browse Source
This commit is contained in:
Marc Froehlich
2026-09-12 22:31:03 +02:00
parent 914a06cb7f
commit 821535a24f
3 changed files with 679 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/PROJECT_CONTEXT.md
+34 -14
View File
@@ -1,6 +1,6 @@
# KST4Contest Project Context # KST4Contest Project Context
Last reviewed: 2026-09-11 Last reviewed: 2026-09-12
This file is the durable technical project context for KST4Contest. It is not a user manual and not a replacement for the changelog. Current code, tests and authoritative external specifications remain the source of truth when this document is stale or ambiguous. This file is the durable technical project context for KST4Contest. It is not a user manual and not a replacement for the changelog. Current code, tests and authoritative external specifications remain the source of truth when this document is stale or ambiguous.
@@ -178,19 +178,39 @@ Current website/deployment scripts and update-feed behaviour must be inspected b
### Server-side website statistics ### Server-side website statistics
- The home page can display a public visit total from the same-origin `GET /visitor-count.json` endpoint. The versioned public contract contains only `schemaVersion`, `visits`, `since` and `updatedAt`. The detailed operator runbook is
- `visits` is the sum of daily approximate unique visits since `since`, based on GoAccess visitor semantics. It is not a count of unique people and remains separate from page views. [`website/ops/analytics/README.md`](../website/ops/analytics/README.md). This
- The home page validates the complete payload, formats the date and number for `en-GB`, and inserts the result through `textContent`. Missing, timed-out, failed or invalid responses leave the initially hidden element invisible and do not affect the rest of the page. section records only the durable architecture and operational boundaries.
- The display makes no third-party request, sends no credentials and uses no cookies or local storage. The counter request is excluded from the statistic itself.
- The counter endpoint disables its own access log and serves the public JSON with a one-hour public cache policy and `X-Content-Type-Options: nosniff`. #### Architecture and data flow
- GoAccess is the server-side source. A registry keeps stable site IDs, hostnames, current analytics-log paths, activation dates, public-counter switches and output targets separate for each project subdomain. The Country database is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. A combined report uses only the registered project sites; `stats.hamradioonline.de` is excluded.
- The regular generator passes each current analytics log and its optional uncompressed `.1` rotation directly to GoAccess and relies on the persistent GoAccess database for incremental processing. Logrotate therefore uses `delaycompress`. Older `.gz` rotations are not imported during regular runs, and missing Zlib support is an accepted, explicitly reported capability state for the GoAccess 1.8.1 production baseline. - Production runs on Ubuntu Server 24.04 with Nginx 1.24, Node.js 18.19.1 and GoAccess 1.8.1. GoAccess has GeoIP2/MMDB and OpenSSL support but no Zlib support; missing Zlib is intentional for this operating model.
- GoAccess 1.8.1 exposes the Country panel as `geolocation` in JSON. Combined report jobs explicitly enable `VIRTUAL_HOSTS` and require the resulting `vhosts` panel; individual site jobs do not enable it. Missing required panels invalidate the complete staged run. - Nginx writes eligible `GET` page requests to a reduced, tab-separated analytics log in addition to the normal operational log. It records server name, client IP, time, method, normalized path without query string, protocol, status, bytes and user agent; referrer and remote user are omitted.
- Node.js 18.19.1 is the production runtime baseline. `--check` requires readable input files, prepared writable output directories and GoAccess built with GeoIP2/MMDB support. OpenSSL and absent Zlib support remain informational. Dry-runs use temporary state and never acquire the production lock. - Nginx removes assets, downloads, status/monitoring paths, update feed, counter requests and known crawler clients before logging. GoAccess supplies the second crawler-classification layer, anonymises IP addresses at the configured level and performs Country lookup locally through GeoLite2-Country.
- The generator runs as `hamradio-analytics`. The state root is mode `0711`; only explicitly prepared report and public-output directories are shared read-only with Nginx through the `www-data` group. GoAccess databases and public counter state remain private. No ACL support is assumed. - The hourly systemd timer starts the Node.js generator. Each registered site is processed from its optional uncompressed `.1` rotation followed by the current log; older `.gz` files are not part of regular processing. Logrotate retains raw analytics logs for 14 days and requires `delaycompress` for this handover.
- The protected statistics vhost is enabled in two stages: an IPv4-only HTTP bootstrap obtains the certificate through `/snap/bin/certbot`, then the final configuration retains an IPv4 HTTP block for the webroot ACME challenge and permanently redirects all other HTTP requests to HTTPS. The HTTPS block uses the existing Certbot TLS options and redirects authenticated requests from `/` to `/combined/`. IPv6 remains disabled until the DNS AAAA record has been confirmed. - GoAccess maintains one private report/database per registered site and one combined report/database. The generator then updates its separate daily counter state and publishes `visitor-count.json` for enabled sites. Private reports are served with Basic Auth from `stats.hamradioonline.de`; the public home page requests only its same-origin counter file.
- Dedicated analytics raw logs are retained for 14 days. IP addresses are anonymised before the detailed GoAccess aggregates are persisted for a rolling 395 days. Separate non-personal daily counter values remain available from activation onward so the public total does not shrink with the detailed retention window.
- Repository templates are installed with explicit Unix owners and modes because ZIP metadata created on Windows is not trusted. Production activation, credentials, password hashes, certificate keys, GeoIP acquisition credentials and backups remain outside the repository. #### Durable invariants and persistence
- GoAccess defines a visit by IP address, date and user agent. The public value is therefore an approximate visit total, not a count of unique people, and remains separate from page views.
- The public contract contains only `schemaVersion`, `visits`, `since` and `updatedAt`. The browser uses no analytics cookie, local storage or third-party request; failed or invalid responses leave the counter hidden.
- The site registry at `/etc/hamradioonline-analytics/sites.json` owns stable site IDs, hostnames, current uncompressed log paths, activation dates and output targets. `stats.hamradioonline.de` is never registered as an analytics site. Existing counter state cannot be continued with a changed hostname or `activatedOn` without an explicit migration or reset decision.
- GoAccess databases below `/var/lib/hamradioonline-analytics/db` retain rolling 395-day detail through `--persist` and `--restore`. `/var/lib/hamradioonline-analytics/public-counter-state.json` retains daily values from activation onward and is the durable business source for the lifetime public total. Reports and public JSON files are derived outputs.
- GoAccess 1.8.1 exposes Country data as `geolocation`; combined jobs explicitly enable `VIRTUAL_HOSTS` and require `vhosts`. Individual site jobs do not enable that panel. `geo_location` and `virtual_hosts` are invalid interface names.
- Generator inputs and staged GoAccess outputs are validated before publication. Files are replaced atomically, dry-runs use temporary state without the production lock, and productive runs use `/run/hamradioonline-analytics/generator.lock`. Exit codes are 1 for runtime/publication errors, 2 for configuration errors and 3 when the production lock cannot be acquired.
#### Components, roles and external services
- `hamradio-analytics` is the locked, non-login service account which reads configuration, Country MMDB and analytics logs and writes service state. Nginx runs as `www-data`, writes the analytics log and can read only reports and the public counter, not private databases or counter state. `stats-reader` is a local Nginx Basic Auth login whose password file remains outside Git.
- Root-managed program/configuration lives below `/opt/hamradioonline-analytics` and `/etc/hamradioonline-analytics`; service state lives below `/var/lib/hamradioonline-analytics`. Report/public directories are shared read-only with Nginx through group ownership; no ACL dependency exists.
- MaxMind is used only by `geoipupdate` to download GeoLite2-Country. Visitor lookups are local; Account ID and License Key remain in the protected server configuration. Let's Encrypt supplies TLS for the statistics vhost through the permanent ACME webroot and `/snap/bin/certbot`. GitHub supplies website source and deployment, not analytics processing.
- The website deploy checks out `origin/main`, builds the static site and publishes it to the Nginx document root. It does not install or overwrite analytics programs/configuration, systemd, Nginx, Logrotate, GeoIP, Basic Auth or Certbot state; those remain separate manual server operations.
- Passwords, hashes, deployment credentials, MaxMind credentials, ACME account data and private TLS keys must remain outside the repository and ordinary diagnostic output.
#### Open operational work
- The server has no comprehensive automated backup plan yet. A future server-wide plan must include the non-regenerable counter state, GoAccess databases and protected operational configuration without extending the published 14-day raw-log retention.
- The first real rotation of the dedicated analytics log still requires an explicit operational check of `.1`, ownership/readability, subsequent generator success and absence of duplicate counting. This is not a current service blocker.
## Important Decisions and Workarounds ## Important Decisions and Workarounds
website/ARCHITECTURE.md
+10 -1
View File
@@ -48,4 +48,13 @@ Nginx serves this directory.
## Build Strategy ## Build Strategy
GitHub Actions will later build the 11ty website and deploy the generated output to the server. The production server keeps a checkout at `/srv/git/kst4contest`. A root cron
job runs `/srv/scripts/deploy-kst4contest-website.sh` every five minutes. The
script resets the checkout to `origin/main`, installs the locked npm
dependencies, builds and validates the Eleventy site, and synchronises the
result to `/srv/www/kst4contest/current` with the ownership required by Nginx.
This deploy path updates only the static website. Server-side analytics,
Nginx, systemd, Logrotate, GeoIP, Basic Auth and Certbot configuration are
installed and maintained separately as described in
[`ops/analytics/README.md`](ops/analytics/README.md).
website/ops/analytics/README.md
+635 -49
View File
@@ -1,8 +1,9 @@
# Server-side website statistics # Server-side website statistics
This directory contains installation examples for the confirmed Ubuntu 24.04 This directory contains the maintained installation templates and the operator
server baseline and privacy-conscious traffic statistics. Nothing here runbook for the production analytics service on Ubuntu Server 24.04. Nothing in
installs or activates the production service automatically. the repository installs, updates or activates the server-side components
automatically.
The design has two separate outputs: The design has two separate outputs:
@@ -31,6 +32,11 @@ GoAccess with IP anonymisation
| |
v v
durable daily counter state --> public visitor-count.json durable daily counter state --> public visitor-count.json
|
v
same-origin home-page request
private HTML reports --> Nginx Basic Auth --> stats.hamradioonline.de
``` ```
Nginx writes a dedicated, reduced log. For each site, the generator gives the Nginx writes a dedicated, reduced log. For each site, the generator gives the
@@ -48,6 +54,22 @@ being added again. This makes repeated runs idempotent. Values older than 395
days remain in the counter state and continue to contribute to the public days remain in the counter state and continue to contribute to the public
total. total.
## Production platform
The confirmed production baseline is:
- Ubuntu Server 24.04;
- Nginx 1.24;
- Node.js 18.19.1;
- GoAccess 1.8.1 with GeoIP2/MMDB and OpenSSL support, but without Zlib;
- a local GeoLite2-Country database;
- systemd for the oneshot generator and its hourly timer;
- Logrotate for the dedicated analytics log.
Missing Zlib support is intentional for this operating model. The generator
processes the current uncompressed analytics log and the optional uncompressed
`.1` rotation. It does not process `.gz` files during regular operation.
## Files ## Files
- `generate-reports.js` validates configuration and state, runs GoAccess and - `generate-reports.js` validates configuration and state, runs GoAccess and
@@ -60,6 +82,18 @@ total.
- `systemd/` contains a hardened oneshot service and hourly timer. - `systemd/` contains a hardened oneshot service and hourly timer.
- `logrotate/` retains 14 daily analytics-log rotations. - `logrotate/` retains 14 daily analytics-log rotations.
These repository files are templates and source files. Their productive
counterparts are installed separately:
- generator: `/opt/hamradioonline-analytics/generate-reports.js`;
- registry: `/etc/hamradioonline-analytics/sites.json`;
- GoAccess template:
`/etc/hamradioonline-analytics/goaccess.conf.template`;
- service and timer: `/etc/systemd/system/hamradioonline-analytics.service`
and `/etc/systemd/system/hamradioonline-analytics.timer`;
- Logrotate configuration: `/etc/logrotate.d/hamradioonline-analytics`;
- service state and derived outputs: `/var/lib/hamradioonline-analytics`.
## Prerequisites ## Prerequisites
- Node.js 18.19.1 or newer; - Node.js 18.19.1 or newer;
@@ -87,6 +121,24 @@ report and public-output directories. Access is group-based. The setup does
not depend on ACLs or `setfacl`. Nginx receives read access to reports and the not depend on ACLs or `setfacl`. Nginx receives read access to reports and the
public counter through the `www-data` group, but no write access. public counter through the `www-data` group, but no write access.
## Accounts and roles
`hamradio-analytics` is a local system account with its own group, no regular
home directory, `/usr/sbin/nologin` as its shell and a locked password. It runs
the generator and GoAccess, reads the reduced logs and configuration, and
writes only below the configured service-state directories.
Nginx runs as `www-data`. It writes the dedicated analytics log and reads the
private reports and public counter. It must not be able to read the private
GoAccess databases or `public-counter-state.json`, and it has no write access
to generated output.
`stats-reader` is the current local Nginx Basic Auth username. It is not a
Linux service account and not an account with an external analytics provider.
Its password file is `/etc/nginx/htpasswd/hamradioonline-analytics`. Store the
password and hash only on the server and in a future protected backup, never in
Git, this runbook, screenshots or support output.
## Installation and permissions ## Installation and permissions
Do not trust Unix modes stored in a ZIP created on Windows. Install every Do not trust Unix modes stored in a ZIP created on Windows. Install every
@@ -149,6 +201,31 @@ Generated HTML and JSON reports use mode `0640`. The public
by Nginx. The service unit uses `StateDirectoryMode=0711` to retain this by Nginx. The service unit uses `StateDirectoryMode=0711` to retain this
boundary after systemd has prepared the state directory. boundary after systemd has prepared the state directory.
The productive ownership and mode boundaries are:
- generator: `0750 root:hamradio-analytics`;
- registry and GoAccess configuration: `0640 root:hamradio-analytics`;
- report directories, including `reports/combined` and
`reports/kst4contest`: `2750 hamradio-analytics:www-data`;
- report files: `0640 hamradio-analytics:www-data`;
- private database files: `0640 hamradio-analytics:hamradio-analytics`;
- `public-counter-state.json`: mode `0640`, owner and group
`hamradio-analytics:hamradio-analytics`;
- public output directories, including `public/kst4contest`: mode `2750`,
owner and group `hamradio-analytics:www-data`;
- public `visitor-count.json`: `0644 hamradio-analytics:www-data`;
- Basic Auth password file: `0640 root:www-data`.
`/opt/hamradioonline-analytics` and `/etc/hamradioonline-analytics` are managed
by root and readable by the service group. `/var/lib/hamradioonline-analytics`
belongs to the service account. Its `0711` root allows Nginx to traverse known
paths without listing or reading private state. The service has a restrictive
`UMask=0027`; the generator therefore sets the public counter to `0644`
explicitly so Nginx can serve it. Missing output directories are a
configuration error. Create them deliberately with the shared group and
set-group-ID modes above rather than relying on recursive creation with an
unsuitable group.
Create the analytics log only when it does not already exist. Running Create the analytics log only when it does not already exist. Running
`install /dev/null` unconditionally would empty an existing log: `install /dev/null` unconditionally would empty an existing log:
@@ -192,12 +269,31 @@ Treat `activatedOn` as persistent data. Once counting has started, changing it
would change the meaning of the total. The generator refuses to combine a new would change the meaning of the total. The generator refuses to combine a new
activation date with existing counter state. activation date with existing counter state.
The hostname is persistent identity as well. If an existing site state has a
different `activatedOn` or hostname, do not delete the state to make the next
run pass. Changing either value requires a deliberate migration or a
specifically approved reset of the public count.
The generator derives the optional `.1` path from `analyticsLog`. It is valid The generator derives the optional `.1` path from `analyticsLog`. It is valid
for `.1` not to exist before the first rotation. Do not enter a rotation or a for `.1` not to exist before the first rotation. Do not enter a rotation or a
compressed `.gz` file in the registry. compressed `.gz` file in the registry.
Adding another project subdomain also requires its own Nginx analytics log,
the corresponding Logrotate ownership, a prepared report directory and, when
enabled, a public-output directory and counter location. The combined report
uses the logs of every registered project site. The statistics vhost remains
outside the registry.
## Nginx logging ## Nginx logging
The relevant production configuration files are:
- `/etc/nginx/snippets/hamradioonline-analytics-filters.conf`;
- `/etc/nginx/conf.d/hamradioonline-analytics-log.conf`;
- `/etc/nginx/snippets/kst4contest-public-counter.conf`;
- `/etc/nginx/sites-available/kst4contest.conf`;
- `/etc/nginx/sites-available/stats.hamradioonline.de`.
Install the log-format and filter maps from `nginx/` in the `http` context. Install the log-format and filter maps from `nginx/` in the `http` context.
Then add a dedicated analytics `access_log` to every registered project server Then add a dedicated analytics `access_log` to every registered project server
block. Keep the existing operational access log unless its replacement has block. Keep the existing operational access log unless its replacement has
@@ -237,12 +333,24 @@ The analytics format contains only:
- transferred body size; - transferred body size;
- user agent. - user agent.
It does not contain a referrer, query string or authenticated user name. The The fields are tab-separated. The production log is
filter accepts only eligible page `GET` requests. It excludes the update feed, `/var/log/nginx/kst4contest-analytics.log`. Nginx writes it as `www-data`; the
public counter, sitemap, robots file, favicons, CSS, JavaScript, images, fonts, `hamradio-analytics` group can read it. The confirmed owner and mode are
source maps, manual assets and the listed monitoring paths. Known crawler user `www-data:hamradio-analytics 0640`. The analytics service receives read-only
agents are rejected before logging. GoAccess applies its own crawler list as a access and must never truncate or otherwise modify this log.
second layer and treats unknown browsers or operating systems as crawlers.
The format uses `$uri`, not `$request_uri`, so query strings never enter the
analytics log. It also omits referrer and authenticated remote-user data. The
user agent is retained because GoAccess needs it for crawler classification
and its visit definition.
Only eligible `GET` requests can be logged. Assets, downloads, status and
monitoring paths, sitemap, robots file, favicons, the update feed and the
public counter endpoint are excluded. Known bots, crawlers, monitoring
clients, `wget` and `curl` are rejected before logging. GoAccess applies its
own crawler list as a second layer and treats unknown browser or operating
system combinations as crawlers. The public counter request therefore cannot
count itself, and the statistics vhost has no analytics logging of its own.
Review the monitoring-path list against the real server before activation. Review the monitoring-path list against the real server before activation.
When a new health endpoint or asset family is added, update the filter first. When a new health endpoint or asset family is added, update the filter first.
@@ -253,12 +361,32 @@ Test the complete Nginx configuration before reloading it:
sudo nginx -t sudo nginx -t
``` ```
### Log rotation
`/etc/logrotate.d/hamradioonline-analytics` rotates the dedicated analytics
logs daily, retains 14 rotations and compresses older files. `delaycompress`
is an operational requirement: it keeps the immediately preceding rotation
as an uncompressed `.1` file for the next generator run. The `create 0640
www-data hamradio-analytics` directive preserves the write/read boundary.
After rotation, `invoke-rc.d nginx rotate` makes Nginx reopen its logs.
The generator processes, in this order:
1. the optional, uncompressed `.1` rotation;
2. the current analytics log.
A missing `.1` is normal, including before the first rotation. Older `.gz`
files are retained according to Logrotate but are not imported by the regular
generator. A registry path that identifies `.1`, another numbered rotation or
a `.gz` file is rejected.
## GoAccess reports ## GoAccess reports
The template enables IP anonymisation before persistent aggregation, ignores The template enables IP anonymisation at `anonymize-level 2` before persistent
crawlers, keeps 395 days, and uses a separate persistent database for every aggregation, ignores crawlers, keeps 395 days, and uses a separate persistent
site and the combined report. It leaves only the panels needed here: visits by database for every site and the combined report. It leaves only the panels
day, requested pages, countries, HTTP status codes and virtual hosts. Host, needed here: visits by day, requested pages, countries, HTTP status codes and
virtual hosts. Host,
remote-user, referrer, keyphrase, operating-system, browser and other detailed remote-user, referrer, keyphrase, operating-system, browser and other detailed
panels are disabled. panels are disabled.
@@ -268,6 +396,12 @@ resulting `vhosts` key. Site jobs do not enable that panel. The generator treats
either missing key as an invalid report rather than publishing incomplete either missing key as an invalid report rather than publishing incomplete
statistics. statistics.
The GoAccess 1.8.1 JSON keys are an interface invariant. `geo_location` and
`virtual_hosts` are invalid names and may appear in repository tests only as
deliberately rejected negative cases. The Country panel comes from the
configured MMDB; missing `geolocation` invalidates every report, and missing
`vhosts` invalidates the combined report.
The Country database is provided through the registry at The Country database is provided through the registry at
`/var/lib/GeoIP/GeoLite2-Country.mmdb`. A file whose name contains `City` is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. A file whose name contains `City` is
rejected. Do not replace it with a City database merely because one happens to rejected. Do not replace it with a City database merely because one happens to
@@ -294,8 +428,100 @@ window, the regular run cannot recover entries found only in older `.gz`
files. Preserve those files under the raw-log retention policy and plan any files. Preserve those files under the raw-log retention policy and plan any
necessary historical import separately before resuming normal processing. necessary historical import separately before resuming normal processing.
### Visit and privacy boundary
GoAccess treats requests with the same IP address, date and user agent as one
visit. The public number is therefore an approximate visit total, not a count
of uniquely identified people. Page views remain a separate statistic.
IP addresses are processed with the configured GoAccess anonymisation level.
Country resolution happens locally against GeoLite2-Country; City and host
statistics are not produced. No visitor address is sent to MaxMind or another
analytics service. The website sets no analytics cookie, embeds no external
tracking script and sends no visit to Google Analytics, Matomo Cloud or any
other analytics platform.
The home-page script requests only `/visitor-count.json` from the same origin.
The public file contains `schemaVersion`, `visits`, `since` and `updatedAt` and
no visitor-level or daily detail.
## Persistence and publication
The installation has four distinct persistence layers.
### Raw logs
The current analytics log and its rotations are short-lived input. The current
file and `.1` bridge requests across the most recent rotation. Logrotate limits
raw-log retention to the published 14-day policy; backups must not silently
extend that period.
### GoAccess databases
Persistent detail state lives in:
- `/var/lib/hamradioonline-analytics/db/kst4contest`;
- `/var/lib/hamradioonline-analytics/db/combined`.
`--persist` writes the processing state and `--restore` loads it on later runs.
This lets GoAccess process new log content without adding the same input from
scratch on every hourly run. The `keep-last 395` setting limits detailed
aggregates to a rolling 395 days.
### Public counter state
`/var/lib/hamradioonline-analytics/public-counter-state.json` stores daily
visit values for each enabled site from `activatedOn` onward. The generator
replaces a day's value when it is processed again; it does not add the value a
second time. This file is the durable business source for the lifetime public
total, including days which have aged out of the GoAccess detail database.
The stored hostname and `activatedOn` must continue to match the registry.
Changing either value requires a planned migration or an approved reset, not
an ad-hoc edit or deletion of the state file.
### Reports and public files
The derived outputs are:
- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.html`;
- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.json`;
- `/var/lib/hamradioonline-analytics/reports/combined/report.html`;
- `/var/lib/hamradioonline-analytics/reports/combined/report.json`;
- `/var/lib/hamradioonline-analytics/public/kst4contest/visitor-count.json`.
The generator prepares every GoAccess job in a run directory, validates the
HTML and JSON outputs, and calculates counter updates before publication. A
GoAccess or report-validation failure therefore leaves the published files
unchanged. Report, state and public files are replaced atomically one file at a
time. Database directories are exchanged through a temporary backup name and
restored if that exchange fails. These replacements are not one filesystem
transaction across every report, database and counter file; after a storage or
permission failure during publication, inspect the complete set and rerun the
service after correcting the cause.
Counter state and GoAccess databases are the important persistent sources.
HTML/JSON reports and `visitor-count.json` are derived and can be rebuilt when
their corresponding source state is available.
## Checking and running ## Checking and running
Do not rely on `node --check` alone. First inspect the installed generator's
owner, mode and plausible non-zero size, then compare its SHA-256 digest with
the reviewed repository file:
```sh
sudo stat -c '%U:%G %a %s %n' \
/opt/hamradioonline-analytics/generate-reports.js
sha256sum /srv/git/kst4contest/website/ops/analytics/generate-reports.js \
/opt/hamradioonline-analytics/generate-reports.js
/usr/bin/node --check /opt/hamradioonline-analytics/generate-reports.js
```
A zero-byte JavaScript file is syntactically valid and exits successfully
without doing any work. File size, digest and the expected completion message
are therefore part of every recovery check.
Validate paths, registry values, the template contract and GoAccess Validate paths, registry values, the template contract and GoAccess
availability without producing reports: availability without producing reports:
@@ -328,8 +554,8 @@ sudo -u hamradio-analytics /usr/bin/node \
Run without either flag to publish. A lock prevents concurrent production Run without either flag to publish. A lock prevents concurrent production
runs. A dry-run uses a temporary working directory and deliberately neither runs. A dry-run uses a temporary working directory and deliberately neither
needs nor creates the production lock below `/run`. Configuration errors use needs nor creates the production lock below `/run`. Configuration errors use
exit code 2, an active production lock uses exit code 3, and generation or exit code 2, failure to acquire the production lock uses exit code 3, and
publication errors use exit code 1. generation or publication errors use exit code 1.
Before the first production run, seed any earlier daily values which must be Before the first production run, seed any earlier daily values which must be
preserved into `public-counter-state.json`. There is no honest way to recreate preserved into `public-counter-state.json`. There is no honest way to recreate
@@ -337,15 +563,69 @@ history which is no longer present in the raw logs. Back up this state file: it
is the durable source for public totals older than the detailed retention is the durable source for public totals older than the detailed retention
window. window.
## Scheduling and report access ### Safe verification sequence
Use this order for a new installation, a recovered service or a material
generator/configuration update:
1. Check generator size, ownership, mode and SHA-256 against the reviewed
checkout.
2. Run `/usr/bin/node --check` on the installed generator.
3. Run the generator with `--check` as `hamradio-analytics`.
4. Confirm the reported GoAccess version and GeoIP2/MMDB, OpenSSL and Zlib
capability state. Missing Zlib is expected; missing MMDB support is not.
5. Record the hashes and timestamps of current reports, databases and counter
files, then run `--dry-run`.
6. Confirm that the recorded production files did not change and that
`/run/hamradioonline-analytics/generator.lock` was not created by the
dry-run.
7. Start one productive run through the service unit:
`sudo systemctl start hamradioonline-analytics.service`.
8. Inspect `Result` and `ExecMainStatus` and read the unit journal. A successful
run ends with `Analytics generation completed`.
9. Check every generated file's path, owner, group, mode and timestamp.
10. Request the public JSON through HTTPS and validate its four fields.
11. Request both private report URLs with Basic Auth. Let the client prompt for
the password; never put it directly on a command line.
12. Run the service a second time and confirm that the total and report values
develop plausibly rather than multiplying the existing history.
13. Enable or re-enable the timer only after these checks pass.
14. Confirm the first automatic run in the journal and later verify the first
real log rotation separately.
Useful service checks are:
```sh
systemctl show hamradioonline-analytics.service \
-p Result -p ExecMainStatus
journalctl -u hamradioonline-analytics.service --since today
systemctl status hamradioonline-analytics.timer
systemctl list-timers hamradioonline-analytics.timer
```
## systemd operation
The production oneshot service runs as `hamradio-analytics` with
`UMask=0027`. Its sandbox exposes `/etc/hamradioonline-analytics`,
`/opt/hamradioonline-analytics`, the Nginx logs and the Country MMDB read-only.
`/var/lib/hamradioonline-analytics` is its only application-state write area;
`/run/hamradioonline-analytics` holds the production lock
`generator.lock`. The service has no network access and only the documented
read/write paths.
The timer uses `OnCalendar=hourly`, `Persistent=true`,
`RandomizedDelaySec=4m` and `AccuracySec=1m` and is permanently enabled in
production. Multiple automatic hourly runs have completed successfully. Each
run produces one site report, one combined report and one public counter,
takes roughly one second on the current installation and has shown stable
incremental behaviour without sudden duplicate counting.
Install the systemd files as local units after adapting paths and permissions.
The timer runs hourly, catches up after downtime and adds a small random delay.
The service has no network access and only the documented read/write paths.
If the installed GoAccess build unexpectedly requires network access, find the If the installed GoAccess build unexpectedly requires network access, find the
reason before weakening that restriction; local log processing and a local reason before weakening that restriction; local log processing and a local
Country database do not require it. Country database do not require it.
## Report access
The statistics vhost serves static files over HTTPS and protects the complete The statistics vhost serves static files over HTTPS and protects the complete
host with HTTP Basic Authentication. This includes `/`, its redirect to host with HTTP Basic Authentication. This includes `/`, its redirect to
`/combined/`, and every individual report. Store the password file outside `/combined/`, and every individual report. Store the password file outside
@@ -362,14 +642,33 @@ sudo chown root:www-data /etc/nginx/htpasswd/hamradioonline-analytics
sudo chmod 0640 /etc/nginx/htpasswd/hamradioonline-analytics sudo chmod 0640 /etc/nginx/htpasswd/hamradioonline-analytics
``` ```
Choose the account name locally and enter the password interactively. Never Use the local account name `stats-reader` and enter its password interactively.
store the resulting password hash in this repository or the installation ZIP. Never store the resulting password hash in this repository or an installation
ZIP.
The example opens no GoAccess WebSocket and no additional GoAccess port. Its The example opens no GoAccess WebSocket and no additional GoAccess port. Its
own access log is disabled and responses use a private, no-store cache policy. own access log is disabled and responses use a private, no-store cache policy.
Do not activate the final HTTPS vhost before its certificate files exist. The current private endpoints are:
First install the temporary HTTP bootstrap without changing the parallel apt
Certbot installation or either renewal timer: - `https://stats.hamradioonline.de/`, which redirects an authenticated request
to `/combined/`;
- `https://stats.hamradioonline.de/combined/`, which serves the combined
report;
- `https://stats.hamradioonline.de/kst4contest/`, which serves the site report.
All HTTPS paths, including the redirect target, remain behind Basic Auth.
Reports use `Cache-Control: private, no-store`,
`X-Content-Type-Options: nosniff` and `X-Frame-Options: DENY`; dotfiles are
blocked. The vhost has no access log and no analytics log. HTTP remains open
only for the ACME webroot and permanently redirects every other request to
HTTPS. No IPv6 listener is configured while DNS AAAA operation remains
unconfirmed.
The final HTTP/HTTPS vhost is the production configuration. The HTTP-only
bootstrap template is retained solely for first provisioning or recovery when
the certificate files do not yet exist. In that situation, do not activate the
final HTTPS vhost first. Install the temporary bootstrap without changing the
parallel APT Certbot installation or either renewal timer:
```sh ```sh
sudo install -d -o root -g root -m 0755 /var/lib/letsencrypt sudo install -d -o root -g root -m 0755 /var/lib/letsencrypt
@@ -430,34 +729,321 @@ own access log is disabled. Responses use
activation date and update time. It contains no IP address, user agent, activation date and update time. It contains no IP address, user agent,
hostname or per-day detail. hostname or per-day detail.
Use this rollout order: The production URL is
`https://kst4contest.hamradioonline.de/visitor-count.json`. Before the first
successful generator run, a `404` is expected; afterwards it must return
`200` with `application/json`. The home page reveals the counter only after a
valid response. A missing, invalid or unavailable counter never prevents the
rest of the static site from working.
1. Install the server files with explicit modes. Prepare directories, log ## Deployment boundary
ownership, filters and still-inactive Nginx and systemd configuration.
2. Push the website changes, including the Privacy Policy, and let the existing
deployment cron job publish them. Until the JSON endpoint exists, the
visitor count remains hidden automatically.
3. Verify the published Privacy Policy. Only then activate analytics logging,
the report generator and timer, the public counter location and the
protected statistics vhost.
4. Run `nginx -t` before every Nginx reload and perform the real `--check` and
`--dry-run` on the server before the first production generation.
A short period in which the Privacy Policy is already visible but logging is GitHub is the source repository for the website and the reviewed analytics
not yet active is acceptable. Starting analytics logging before publishing the templates. The production checkout is `/srv/git/kst4contest`. A root cron job
updated policy is not. runs `/srv/scripts/deploy-kst4contest-website.sh` every five minutes. It fetches
Git, resets the checkout to `origin/main`, runs `npm ci`, builds the Eleventy
website, validates VersionInfo, synchronises the result to
`/srv/www/kst4contest/current` and finally restores the ownership expected by
Nginx.
## Retention and recovery The deployment credential is stored in `/etc/kst4contest-website.env`. Its
value and account assignment must never appear in documentation, logs or
support output.
- Dedicated analytics raw logs: 14 days through the Logrotate example. This automatic deployment updates only the static website. It does not install
- Anonymised detailed GoAccess aggregates: rolling 395 days. or overwrite:
- Public daily counter values: retained from activation onward.
Back up the counter state and, if fast report recovery matters, the GoAccess - `/opt/hamradioonline-analytics`;
database directories. Reports themselves are derived output. To recover, stop - `/etc/hamradioonline-analytics`;
the timer, restore the state and database directories with their ownership, - systemd units;
run `--check`, then run `--dry-run` before publishing again. - Nginx or Logrotate configuration;
- the Basic Auth password file;
- GeoIP configuration;
- Certbot configuration.
The repository contains no password, password hash, MaxMind download key, Changes to those operational files require a separate review and manual
server IP address, TLS private key or private backup destination. Keep it that installation. The published privacy notice must remain in place whenever
way. analytics logging is active.
## Regular operation
Nginx continuously writes only eligible requests to the dedicated analytics
log. The systemd timer starts the generator once per hour. Every successful
run refreshes the per-site and combined reports, persists the corresponding
GoAccess databases, updates daily counter values and finally publishes enabled
public counters. Logrotate handles the raw log once per day and preserves the
uncompressed `.1` handover file required by the generator.
The normal operator signal is the service result and journal, not a permanently
running process: the generator is a short-lived oneshot service. There is no
GoAccess WebSocket process and no public GoAccess port.
## External services and local credentials
### MaxMind GeoLite2
MaxMind is used only to download and update GeoLite2-Country. The local updater
configuration is `/etc/GeoIP.conf` with mode `0600 root:root`; the local
database is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. GeoLite is enabled and the
server has an Account ID and License Key, but neither value belongs in Git,
this runbook, screenshots, logs or ordinary diagnostic output.
`geoipupdate.service` and `geoipupdate.timer` download updates from MaxMind.
All visitor lookups then happen locally. The analytics application never sends
an individual visitor address to MaxMind.
If the MMDB is missing, stale or unreadable, inspect the timer and journal,
check the file mode and run the updater directly if required:
```sh
systemctl status geoipupdate.timer geoipupdate.service
journalctl -u geoipupdate.service --since today
sudo stat -c '%U:%G %a %s %y %n' \
/etc/GeoIP.conf /var/lib/GeoIP/GeoLite2-Country.mmdb
sudo geoipupdate
sudo -u hamradio-analytics test -r \
/var/lib/GeoIP/GeoLite2-Country.mmdb
```
When `mmdblookup` is installed, a lookup of a neutral public test address can
confirm database readability without using any visitor address:
```sh
sudo -u hamradio-analytics mmdblookup \
--file /var/lib/GeoIP/GeoLite2-Country.mmdb \
--ip 1.1.1.1 country iso_code
```
An unreadable Country MMDB or GoAccess without MMDB support makes `--check`
fail with exit code 2.
### Let's Encrypt
Let's Encrypt supplies the TLS certificate for `stats.hamradioonline.de`. Its
state is below `/etc/letsencrypt/live/stats.hamradioonline.de/`. The ACME
webroot is `/var/lib/letsencrypt`, and the final Nginx vhost permanently serves
`/.well-known/acme-challenge/` over IPv4 port 80 so webroot renewal continues
to work.
Use the Snap client explicitly as `/snap/bin/certbot` for this vhost.
Certificate issuance and a renewal dry-run have been confirmed. The server
currently also has an APT Certbot installation and both renewal timers. That
duplication is a separate server-maintenance issue; do not change either
installation as part of analytics maintenance.
ACME account data, private keys and certificate state live only on the server
and in a future protected backup. Validate renewal with:
```sh
sudo /snap/bin/certbot renew --dry-run \
--cert-name stats.hamradioonline.de
```
### GitHub
GitHub supplies the source repository consumed by the website deployment. It
is not an analytics processor and receives no individual analytics request or
visit data. The deployment credential remains in
`/etc/kst4contest-website.env`; only the server-side deploy process needs it.
### No external analytics platform
Statistics are generated locally on the project server. There is no Google
Analytics, Matomo Cloud service, external tracking script or transfer of
individual visits to an analytics provider.
## Troubleshooting
Start with the safe verification sequence above. Keep secrets out of commands
and captured output, and do not weaken file modes merely to make a check pass.
### `GoAccess JSON report has no geo_location panel`
The expected GoAccess 1.8.1 key is `geolocation`. An installed older generator
which expects `geo_location` must be replaced with the reviewed repository
version.
### Missing Virtual Hosts panel
The combined report requires the JSON key `vhosts` and must start GoAccess with
`--enable-panel=VIRTUAL_HOSTS`. `virtual_hosts` is not a valid replacement.
Site reports do not require this panel.
The names `geo_location` and `virtual_hosts` are allowed in repository tests
only as deliberately invalid negative cases.
### Missing `.1` rotation
This is normal before the first rotation and whenever no previous rotation is
present. The current analytics log remains required.
### `.gz` rotations on a GoAccess build without Zlib
This is normal. Regular operation does not read `.gz` files. Do not configure
a compressed or rotated file as `analyticsLog`; any exceptional historical
import must be planned separately.
### MMDB missing or unreadable
`--check` must fail. Verify `/etc/GeoIP.conf`, the GeoIP updater units,
`/var/lib/GeoIP/GeoLite2-Country.mmdb` and the service account's read access.
Run `geoipupdate` and a neutral local lookup as described above when needed.
### Analytics log missing or unreadable
Compare the registry path with the additional `access_log` directive in the
site vhost. Check the active Nginx configuration and verify
`www-data:hamradio-analytics 0640`. Test Nginx before reloading it. Do not
replace or repurpose the normal operational access log.
```sh
sudo nginx -t
sudo nginx -T
sudo stat -c '%U:%G %a %s %y %n' \
/var/log/nginx/kst4contest-analytics.log
sudo -u hamradio-analytics test -r \
/var/log/nginx/kst4contest-analytics.log
```
### Public counter returns `404`
This is expected before the first successful generation. Afterwards inspect
the service journal, public-output directory, `0644` file mode, Nginx include
and exact alias path. A successful response is `200 application/json`.
### Statistics vhost returns `401` or `404`
`401` without credentials is correct. With the valid Basic Auth login, `/`
must redirect to `/combined/`. A `404` after authentication usually means that
the generator has not produced `report.html`, the URL and report directory do
not match, or Nginx cannot traverse/read the report path.
### Certificate error
Check the DNS A record, certificate paths, active Nginx vhost, ACME webroot and
Snap Certbot renewal. Immediately after an Nginx reload, wait briefly and
retry if observations conflict, then inspect `nginx -T`, the Nginx journal and
worker start times.
### Generator lock error
Exit code 3 means that the production lock could not be acquired. Check the
service, timer and running processes. Do not remove the lock blindly. Remove it
only after confirming that no generator process is active and that the lock is
genuinely stale.
### Counter identity mismatch
If `activatedOn` or the hostname differs from existing counter state, stop.
Do not repair this by deleting the state. Establish the cause and approve a
migration or reset explicitly.
### Exit code 0 but no output
Check the installed generator size, owner, mode and SHA-256 before anything
else. Also look for `Analytics generation completed` in the journal and inspect
the expected output files.
A known operator error is copying a shell prompt or continuation marker `>`
with a command. Bash interprets a stray `>` as output redirection. This can
truncate `generate-reports.js` to zero bytes and create empty files whose names
look like command options. A zero-byte JavaScript file still passes
`node --check` and exits with code 0 while doing nothing.
Identify such artifacts precisely before removing them. Reinstall the
generator from the reviewed checkout with the documented owner and mode,
compare its size and SHA-256, then repeat `--check` and `--dry-run`. Do not make
one-off artifact names or old checksums part of the permanent procedure.
## Monitoring
The systemd timer is active, and service results and errors are visible through
systemd and the journal. No separate alerting channel has been confirmed.
Routine checks should confirm:
- the timer is active and waiting with a future trigger;
- the last service result is `success` and `ExecMainStatus=0`;
- the journal contains `Analytics generation completed`;
- report and counter timestamps continue to advance;
- the public total changes plausibly;
- repeated runs do not multiply the existing history.
Central alerting can be handled later as part of the general server operations
plan.
## Backup and recovery
There is currently no comprehensive automated server backup plan. The
analytics installation must be included explicitly when that server-wide
backup and recovery design is implemented. Creating that backup system is not
part of this repository change.
### Data to include
At minimum, the later plan must cover:
- `/var/lib/hamradioonline-analytics/public-counter-state.json`;
- `/var/lib/hamradioonline-analytics/db`;
- `/etc/hamradioonline-analytics`;
- the installed systemd units;
- Nginx analytics and vhost configuration;
- `/etc/logrotate.d/hamradioonline-analytics`;
- `/etc/nginx/htpasswd/hamradioonline-analytics`;
- Certbot and Let's Encrypt account/certificate state;
- `/etc/GeoIP.conf`;
- the website deployment configuration and credential storage.
Handle the Basic Auth hash, MaxMind Account ID and License Key, GitHub deploy
token, ACME account data and private TLS keys as secrets. Never copy them into
Git, public documentation, logs or ordinary support bundles.
The generator is recoverable from GitHub, and GeoLite2-Country can be fetched
again with `geoipupdate`. HTML/JSON reports can be rebuilt when the GoAccess
databases or sufficient raw logs remain. The public JSON can be rebuilt from
the counter state.
The accumulated public total is not fully recoverable without
`public-counter-state.json`. Older detailed aggregates are not recoverable
without the GoAccess databases, and historical raw requests disappear after
the 14-day rotation window.
Do not let backups extend the published raw-log retention by accident. Either
exclude analytics raw logs from durable backups or enforce the same confirmed
retention limit in backup storage. Counter state and anonymised/aggregated
GoAccess state can be governed separately.
### Recovery order
1. Install the operating-system packages and external dependencies.
2. Recreate the service account and group relationships.
3. Create the directories with the documented owners and modes.
4. Install the generator and non-secret configuration.
5. Restore secrets and certificate state from protected backup storage.
6. Restore GeoLite2-Country or download it again.
7. Restore the GoAccess databases and public counter state.
8. Validate Nginx, systemd and Logrotate configuration.
9. Run the generator with `--check`.
10. Run `--dry-run` and verify that production state remains unchanged.
11. Start one productive run through the systemd service.
12. Verify reports, public counter and Basic Auth.
13. Enable the timer only after every preceding check succeeds.
## Outstanding operational checks
The first real rotation of the dedicated analytics log still needs explicit
observation. This is not a current service blocker. After rotation, confirm:
- a new current log exists;
- `.1` exists and remains uncompressed;
- both files retain the expected owners and modes;
- `hamradio-analytics` can read both files;
- the next generator run succeeds;
- values do not show duplicate counting;
- the older rotation is compressed on the following cycle as intended.
The general server backup/recovery implementation and any central alerting
remain separate future operations tasks.
The repository contains no password, password hash, MaxMind credential,
deployment token, server IP address, TLS private key or private backup
destination. Keep it that way.