mirror of
https://github.com/praktimarc/kst4contest.git
synced 2026-09-13 20:55:31 +02:00
Documentation on user counter and analytics to the kst4contest page
This commit is contained in:
Marc Froehlich
+635
-49
@@ -1,8 +1,9 @@
|
||||
# Server-side website statistics
|
||||
|
||||
This directory contains installation examples for the confirmed Ubuntu 24.04
|
||||
server baseline and privacy-conscious traffic statistics. Nothing here
|
||||
installs or activates the production service automatically.
|
||||
This directory contains the maintained installation templates and the operator
|
||||
runbook for the production analytics service on Ubuntu Server 24.04. Nothing in
|
||||
the repository installs, updates or activates the server-side components
|
||||
automatically.
|
||||
|
||||
The design has two separate outputs:
|
||||
|
||||
@@ -31,6 +32,11 @@ GoAccess with IP anonymisation
|
||||
|
|
||||
v
|
||||
durable daily counter state --> public visitor-count.json
|
||||
|
|
||||
v
|
||||
same-origin home-page request
|
||||
|
||||
private HTML reports --> Nginx Basic Auth --> stats.hamradioonline.de
|
||||
```
|
||||
|
||||
Nginx writes a dedicated, reduced log. For each site, the generator gives the
|
||||
@@ -48,6 +54,22 @@ being added again. This makes repeated runs idempotent. Values older than 395
|
||||
days remain in the counter state and continue to contribute to the public
|
||||
total.
|
||||
|
||||
## Production platform
|
||||
|
||||
The confirmed production baseline is:
|
||||
|
||||
- Ubuntu Server 24.04;
|
||||
- Nginx 1.24;
|
||||
- Node.js 18.19.1;
|
||||
- GoAccess 1.8.1 with GeoIP2/MMDB and OpenSSL support, but without Zlib;
|
||||
- a local GeoLite2-Country database;
|
||||
- systemd for the oneshot generator and its hourly timer;
|
||||
- Logrotate for the dedicated analytics log.
|
||||
|
||||
Missing Zlib support is intentional for this operating model. The generator
|
||||
processes the current uncompressed analytics log and the optional uncompressed
|
||||
`.1` rotation. It does not process `.gz` files during regular operation.
|
||||
|
||||
## Files
|
||||
|
||||
- `generate-reports.js` validates configuration and state, runs GoAccess and
|
||||
@@ -60,6 +82,18 @@ total.
|
||||
- `systemd/` contains a hardened oneshot service and hourly timer.
|
||||
- `logrotate/` retains 14 daily analytics-log rotations.
|
||||
|
||||
These repository files are templates and source files. Their productive
|
||||
counterparts are installed separately:
|
||||
|
||||
- generator: `/opt/hamradioonline-analytics/generate-reports.js`;
|
||||
- registry: `/etc/hamradioonline-analytics/sites.json`;
|
||||
- GoAccess template:
|
||||
`/etc/hamradioonline-analytics/goaccess.conf.template`;
|
||||
- service and timer: `/etc/systemd/system/hamradioonline-analytics.service`
|
||||
and `/etc/systemd/system/hamradioonline-analytics.timer`;
|
||||
- Logrotate configuration: `/etc/logrotate.d/hamradioonline-analytics`;
|
||||
- service state and derived outputs: `/var/lib/hamradioonline-analytics`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js 18.19.1 or newer;
|
||||
@@ -87,6 +121,24 @@ report and public-output directories. Access is group-based. The setup does
|
||||
not depend on ACLs or `setfacl`. Nginx receives read access to reports and the
|
||||
public counter through the `www-data` group, but no write access.
|
||||
|
||||
## Accounts and roles
|
||||
|
||||
`hamradio-analytics` is a local system account with its own group, no regular
|
||||
home directory, `/usr/sbin/nologin` as its shell and a locked password. It runs
|
||||
the generator and GoAccess, reads the reduced logs and configuration, and
|
||||
writes only below the configured service-state directories.
|
||||
|
||||
Nginx runs as `www-data`. It writes the dedicated analytics log and reads the
|
||||
private reports and public counter. It must not be able to read the private
|
||||
GoAccess databases or `public-counter-state.json`, and it has no write access
|
||||
to generated output.
|
||||
|
||||
`stats-reader` is the current local Nginx Basic Auth username. It is not a
|
||||
Linux service account and not an account with an external analytics provider.
|
||||
Its password file is `/etc/nginx/htpasswd/hamradioonline-analytics`. Store the
|
||||
password and hash only on the server and in a future protected backup, never in
|
||||
Git, this runbook, screenshots or support output.
|
||||
|
||||
## Installation and permissions
|
||||
|
||||
Do not trust Unix modes stored in a ZIP created on Windows. Install every
|
||||
@@ -149,6 +201,31 @@ Generated HTML and JSON reports use mode `0640`. The public
|
||||
by Nginx. The service unit uses `StateDirectoryMode=0711` to retain this
|
||||
boundary after systemd has prepared the state directory.
|
||||
|
||||
The productive ownership and mode boundaries are:
|
||||
|
||||
- generator: `0750 root:hamradio-analytics`;
|
||||
- registry and GoAccess configuration: `0640 root:hamradio-analytics`;
|
||||
- report directories, including `reports/combined` and
|
||||
`reports/kst4contest`: `2750 hamradio-analytics:www-data`;
|
||||
- report files: `0640 hamradio-analytics:www-data`;
|
||||
- private database files: `0640 hamradio-analytics:hamradio-analytics`;
|
||||
- `public-counter-state.json`: mode `0640`, owner and group
|
||||
`hamradio-analytics:hamradio-analytics`;
|
||||
- public output directories, including `public/kst4contest`: mode `2750`,
|
||||
owner and group `hamradio-analytics:www-data`;
|
||||
- public `visitor-count.json`: `0644 hamradio-analytics:www-data`;
|
||||
- Basic Auth password file: `0640 root:www-data`.
|
||||
|
||||
`/opt/hamradioonline-analytics` and `/etc/hamradioonline-analytics` are managed
|
||||
by root and readable by the service group. `/var/lib/hamradioonline-analytics`
|
||||
belongs to the service account. Its `0711` root allows Nginx to traverse known
|
||||
paths without listing or reading private state. The service has a restrictive
|
||||
`UMask=0027`; the generator therefore sets the public counter to `0644`
|
||||
explicitly so Nginx can serve it. Missing output directories are a
|
||||
configuration error. Create them deliberately with the shared group and
|
||||
set-group-ID modes above rather than relying on recursive creation with an
|
||||
unsuitable group.
|
||||
|
||||
Create the analytics log only when it does not already exist. Running
|
||||
`install /dev/null` unconditionally would empty an existing log:
|
||||
|
||||
@@ -192,12 +269,31 @@ Treat `activatedOn` as persistent data. Once counting has started, changing it
|
||||
would change the meaning of the total. The generator refuses to combine a new
|
||||
activation date with existing counter state.
|
||||
|
||||
The hostname is persistent identity as well. If an existing site state has a
|
||||
different `activatedOn` or hostname, do not delete the state to make the next
|
||||
run pass. Changing either value requires a deliberate migration or a
|
||||
specifically approved reset of the public count.
|
||||
|
||||
The generator derives the optional `.1` path from `analyticsLog`. It is valid
|
||||
for `.1` not to exist before the first rotation. Do not enter a rotation or a
|
||||
compressed `.gz` file in the registry.
|
||||
|
||||
Adding another project subdomain also requires its own Nginx analytics log,
|
||||
the corresponding Logrotate ownership, a prepared report directory and, when
|
||||
enabled, a public-output directory and counter location. The combined report
|
||||
uses the logs of every registered project site. The statistics vhost remains
|
||||
outside the registry.
|
||||
|
||||
## Nginx logging
|
||||
|
||||
The relevant production configuration files are:
|
||||
|
||||
- `/etc/nginx/snippets/hamradioonline-analytics-filters.conf`;
|
||||
- `/etc/nginx/conf.d/hamradioonline-analytics-log.conf`;
|
||||
- `/etc/nginx/snippets/kst4contest-public-counter.conf`;
|
||||
- `/etc/nginx/sites-available/kst4contest.conf`;
|
||||
- `/etc/nginx/sites-available/stats.hamradioonline.de`.
|
||||
|
||||
Install the log-format and filter maps from `nginx/` in the `http` context.
|
||||
Then add a dedicated analytics `access_log` to every registered project server
|
||||
block. Keep the existing operational access log unless its replacement has
|
||||
@@ -237,12 +333,24 @@ The analytics format contains only:
|
||||
- transferred body size;
|
||||
- user agent.
|
||||
|
||||
It does not contain a referrer, query string or authenticated user name. The
|
||||
filter accepts only eligible page `GET` requests. It excludes the update feed,
|
||||
public counter, sitemap, robots file, favicons, CSS, JavaScript, images, fonts,
|
||||
source maps, manual assets and the listed monitoring paths. Known crawler user
|
||||
agents are rejected before logging. GoAccess applies its own crawler list as a
|
||||
second layer and treats unknown browsers or operating systems as crawlers.
|
||||
The fields are tab-separated. The production log is
|
||||
`/var/log/nginx/kst4contest-analytics.log`. Nginx writes it as `www-data`; the
|
||||
`hamradio-analytics` group can read it. The confirmed owner and mode are
|
||||
`www-data:hamradio-analytics 0640`. The analytics service receives read-only
|
||||
access and must never truncate or otherwise modify this log.
|
||||
|
||||
The format uses `$uri`, not `$request_uri`, so query strings never enter the
|
||||
analytics log. It also omits referrer and authenticated remote-user data. The
|
||||
user agent is retained because GoAccess needs it for crawler classification
|
||||
and its visit definition.
|
||||
|
||||
Only eligible `GET` requests can be logged. Assets, downloads, status and
|
||||
monitoring paths, sitemap, robots file, favicons, the update feed and the
|
||||
public counter endpoint are excluded. Known bots, crawlers, monitoring
|
||||
clients, `wget` and `curl` are rejected before logging. GoAccess applies its
|
||||
own crawler list as a second layer and treats unknown browser or operating
|
||||
system combinations as crawlers. The public counter request therefore cannot
|
||||
count itself, and the statistics vhost has no analytics logging of its own.
|
||||
|
||||
Review the monitoring-path list against the real server before activation.
|
||||
When a new health endpoint or asset family is added, update the filter first.
|
||||
@@ -253,12 +361,32 @@ Test the complete Nginx configuration before reloading it:
|
||||
sudo nginx -t
|
||||
```
|
||||
|
||||
### Log rotation
|
||||
|
||||
`/etc/logrotate.d/hamradioonline-analytics` rotates the dedicated analytics
|
||||
logs daily, retains 14 rotations and compresses older files. `delaycompress`
|
||||
is an operational requirement: it keeps the immediately preceding rotation
|
||||
as an uncompressed `.1` file for the next generator run. The `create 0640
|
||||
www-data hamradio-analytics` directive preserves the write/read boundary.
|
||||
After rotation, `invoke-rc.d nginx rotate` makes Nginx reopen its logs.
|
||||
|
||||
The generator processes, in this order:
|
||||
|
||||
1. the optional, uncompressed `.1` rotation;
|
||||
2. the current analytics log.
|
||||
|
||||
A missing `.1` is normal, including before the first rotation. Older `.gz`
|
||||
files are retained according to Logrotate but are not imported by the regular
|
||||
generator. A registry path that identifies `.1`, another numbered rotation or
|
||||
a `.gz` file is rejected.
|
||||
|
||||
## GoAccess reports
|
||||
|
||||
The template enables IP anonymisation before persistent aggregation, ignores
|
||||
crawlers, keeps 395 days, and uses a separate persistent database for every
|
||||
site and the combined report. It leaves only the panels needed here: visits by
|
||||
day, requested pages, countries, HTTP status codes and virtual hosts. Host,
|
||||
The template enables IP anonymisation at `anonymize-level 2` before persistent
|
||||
aggregation, ignores crawlers, keeps 395 days, and uses a separate persistent
|
||||
database for every site and the combined report. It leaves only the panels
|
||||
needed here: visits by day, requested pages, countries, HTTP status codes and
|
||||
virtual hosts. Host,
|
||||
remote-user, referrer, keyphrase, operating-system, browser and other detailed
|
||||
panels are disabled.
|
||||
|
||||
@@ -268,6 +396,12 @@ resulting `vhosts` key. Site jobs do not enable that panel. The generator treats
|
||||
either missing key as an invalid report rather than publishing incomplete
|
||||
statistics.
|
||||
|
||||
The GoAccess 1.8.1 JSON keys are an interface invariant. `geo_location` and
|
||||
`virtual_hosts` are invalid names and may appear in repository tests only as
|
||||
deliberately rejected negative cases. The Country panel comes from the
|
||||
configured MMDB; missing `geolocation` invalidates every report, and missing
|
||||
`vhosts` invalidates the combined report.
|
||||
|
||||
The Country database is provided through the registry at
|
||||
`/var/lib/GeoIP/GeoLite2-Country.mmdb`. A file whose name contains `City` is
|
||||
rejected. Do not replace it with a City database merely because one happens to
|
||||
@@ -294,8 +428,100 @@ window, the regular run cannot recover entries found only in older `.gz`
|
||||
files. Preserve those files under the raw-log retention policy and plan any
|
||||
necessary historical import separately before resuming normal processing.
|
||||
|
||||
### Visit and privacy boundary
|
||||
|
||||
GoAccess treats requests with the same IP address, date and user agent as one
|
||||
visit. The public number is therefore an approximate visit total, not a count
|
||||
of uniquely identified people. Page views remain a separate statistic.
|
||||
|
||||
IP addresses are processed with the configured GoAccess anonymisation level.
|
||||
Country resolution happens locally against GeoLite2-Country; City and host
|
||||
statistics are not produced. No visitor address is sent to MaxMind or another
|
||||
analytics service. The website sets no analytics cookie, embeds no external
|
||||
tracking script and sends no visit to Google Analytics, Matomo Cloud or any
|
||||
other analytics platform.
|
||||
|
||||
The home-page script requests only `/visitor-count.json` from the same origin.
|
||||
The public file contains `schemaVersion`, `visits`, `since` and `updatedAt` and
|
||||
no visitor-level or daily detail.
|
||||
|
||||
## Persistence and publication
|
||||
|
||||
The installation has four distinct persistence layers.
|
||||
|
||||
### Raw logs
|
||||
|
||||
The current analytics log and its rotations are short-lived input. The current
|
||||
file and `.1` bridge requests across the most recent rotation. Logrotate limits
|
||||
raw-log retention to the published 14-day policy; backups must not silently
|
||||
extend that period.
|
||||
|
||||
### GoAccess databases
|
||||
|
||||
Persistent detail state lives in:
|
||||
|
||||
- `/var/lib/hamradioonline-analytics/db/kst4contest`;
|
||||
- `/var/lib/hamradioonline-analytics/db/combined`.
|
||||
|
||||
`--persist` writes the processing state and `--restore` loads it on later runs.
|
||||
This lets GoAccess process new log content without adding the same input from
|
||||
scratch on every hourly run. The `keep-last 395` setting limits detailed
|
||||
aggregates to a rolling 395 days.
|
||||
|
||||
### Public counter state
|
||||
|
||||
`/var/lib/hamradioonline-analytics/public-counter-state.json` stores daily
|
||||
visit values for each enabled site from `activatedOn` onward. The generator
|
||||
replaces a day's value when it is processed again; it does not add the value a
|
||||
second time. This file is the durable business source for the lifetime public
|
||||
total, including days which have aged out of the GoAccess detail database.
|
||||
|
||||
The stored hostname and `activatedOn` must continue to match the registry.
|
||||
Changing either value requires a planned migration or an approved reset, not
|
||||
an ad-hoc edit or deletion of the state file.
|
||||
|
||||
### Reports and public files
|
||||
|
||||
The derived outputs are:
|
||||
|
||||
- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.html`;
|
||||
- `/var/lib/hamradioonline-analytics/reports/kst4contest/report.json`;
|
||||
- `/var/lib/hamradioonline-analytics/reports/combined/report.html`;
|
||||
- `/var/lib/hamradioonline-analytics/reports/combined/report.json`;
|
||||
- `/var/lib/hamradioonline-analytics/public/kst4contest/visitor-count.json`.
|
||||
|
||||
The generator prepares every GoAccess job in a run directory, validates the
|
||||
HTML and JSON outputs, and calculates counter updates before publication. A
|
||||
GoAccess or report-validation failure therefore leaves the published files
|
||||
unchanged. Report, state and public files are replaced atomically one file at a
|
||||
time. Database directories are exchanged through a temporary backup name and
|
||||
restored if that exchange fails. These replacements are not one filesystem
|
||||
transaction across every report, database and counter file; after a storage or
|
||||
permission failure during publication, inspect the complete set and rerun the
|
||||
service after correcting the cause.
|
||||
|
||||
Counter state and GoAccess databases are the important persistent sources.
|
||||
HTML/JSON reports and `visitor-count.json` are derived and can be rebuilt when
|
||||
their corresponding source state is available.
|
||||
|
||||
## Checking and running
|
||||
|
||||
Do not rely on `node --check` alone. First inspect the installed generator's
|
||||
owner, mode and plausible non-zero size, then compare its SHA-256 digest with
|
||||
the reviewed repository file:
|
||||
|
||||
```sh
|
||||
sudo stat -c '%U:%G %a %s %n' \
|
||||
/opt/hamradioonline-analytics/generate-reports.js
|
||||
sha256sum /srv/git/kst4contest/website/ops/analytics/generate-reports.js \
|
||||
/opt/hamradioonline-analytics/generate-reports.js
|
||||
/usr/bin/node --check /opt/hamradioonline-analytics/generate-reports.js
|
||||
```
|
||||
|
||||
A zero-byte JavaScript file is syntactically valid and exits successfully
|
||||
without doing any work. File size, digest and the expected completion message
|
||||
are therefore part of every recovery check.
|
||||
|
||||
Validate paths, registry values, the template contract and GoAccess
|
||||
availability without producing reports:
|
||||
|
||||
@@ -328,8 +554,8 @@ sudo -u hamradio-analytics /usr/bin/node \
|
||||
Run without either flag to publish. A lock prevents concurrent production
|
||||
runs. A dry-run uses a temporary working directory and deliberately neither
|
||||
needs nor creates the production lock below `/run`. Configuration errors use
|
||||
exit code 2, an active production lock uses exit code 3, and generation or
|
||||
publication errors use exit code 1.
|
||||
exit code 2, failure to acquire the production lock uses exit code 3, and
|
||||
generation or publication errors use exit code 1.
|
||||
|
||||
Before the first production run, seed any earlier daily values which must be
|
||||
preserved into `public-counter-state.json`. There is no honest way to recreate
|
||||
@@ -337,15 +563,69 @@ history which is no longer present in the raw logs. Back up this state file: it
|
||||
is the durable source for public totals older than the detailed retention
|
||||
window.
|
||||
|
||||
## Scheduling and report access
|
||||
### Safe verification sequence
|
||||
|
||||
Use this order for a new installation, a recovered service or a material
|
||||
generator/configuration update:
|
||||
|
||||
1. Check generator size, ownership, mode and SHA-256 against the reviewed
|
||||
checkout.
|
||||
2. Run `/usr/bin/node --check` on the installed generator.
|
||||
3. Run the generator with `--check` as `hamradio-analytics`.
|
||||
4. Confirm the reported GoAccess version and GeoIP2/MMDB, OpenSSL and Zlib
|
||||
capability state. Missing Zlib is expected; missing MMDB support is not.
|
||||
5. Record the hashes and timestamps of current reports, databases and counter
|
||||
files, then run `--dry-run`.
|
||||
6. Confirm that the recorded production files did not change and that
|
||||
`/run/hamradioonline-analytics/generator.lock` was not created by the
|
||||
dry-run.
|
||||
7. Start one productive run through the service unit:
|
||||
`sudo systemctl start hamradioonline-analytics.service`.
|
||||
8. Inspect `Result` and `ExecMainStatus` and read the unit journal. A successful
|
||||
run ends with `Analytics generation completed`.
|
||||
9. Check every generated file's path, owner, group, mode and timestamp.
|
||||
10. Request the public JSON through HTTPS and validate its four fields.
|
||||
11. Request both private report URLs with Basic Auth. Let the client prompt for
|
||||
the password; never put it directly on a command line.
|
||||
12. Run the service a second time and confirm that the total and report values
|
||||
develop plausibly rather than multiplying the existing history.
|
||||
13. Enable or re-enable the timer only after these checks pass.
|
||||
14. Confirm the first automatic run in the journal and later verify the first
|
||||
real log rotation separately.
|
||||
|
||||
Useful service checks are:
|
||||
|
||||
```sh
|
||||
systemctl show hamradioonline-analytics.service \
|
||||
-p Result -p ExecMainStatus
|
||||
journalctl -u hamradioonline-analytics.service --since today
|
||||
systemctl status hamradioonline-analytics.timer
|
||||
systemctl list-timers hamradioonline-analytics.timer
|
||||
```
|
||||
|
||||
## systemd operation
|
||||
|
||||
The production oneshot service runs as `hamradio-analytics` with
|
||||
`UMask=0027`. Its sandbox exposes `/etc/hamradioonline-analytics`,
|
||||
`/opt/hamradioonline-analytics`, the Nginx logs and the Country MMDB read-only.
|
||||
`/var/lib/hamradioonline-analytics` is its only application-state write area;
|
||||
`/run/hamradioonline-analytics` holds the production lock
|
||||
`generator.lock`. The service has no network access and only the documented
|
||||
read/write paths.
|
||||
|
||||
The timer uses `OnCalendar=hourly`, `Persistent=true`,
|
||||
`RandomizedDelaySec=4m` and `AccuracySec=1m` and is permanently enabled in
|
||||
production. Multiple automatic hourly runs have completed successfully. Each
|
||||
run produces one site report, one combined report and one public counter,
|
||||
takes roughly one second on the current installation and has shown stable
|
||||
incremental behaviour without sudden duplicate counting.
|
||||
|
||||
Install the systemd files as local units after adapting paths and permissions.
|
||||
The timer runs hourly, catches up after downtime and adds a small random delay.
|
||||
The service has no network access and only the documented read/write paths.
|
||||
If the installed GoAccess build unexpectedly requires network access, find the
|
||||
reason before weakening that restriction; local log processing and a local
|
||||
Country database do not require it.
|
||||
|
||||
## Report access
|
||||
|
||||
The statistics vhost serves static files over HTTPS and protects the complete
|
||||
host with HTTP Basic Authentication. This includes `/`, its redirect to
|
||||
`/combined/`, and every individual report. Store the password file outside
|
||||
@@ -362,14 +642,33 @@ sudo chown root:www-data /etc/nginx/htpasswd/hamradioonline-analytics
|
||||
sudo chmod 0640 /etc/nginx/htpasswd/hamradioonline-analytics
|
||||
```
|
||||
|
||||
Choose the account name locally and enter the password interactively. Never
|
||||
store the resulting password hash in this repository or the installation ZIP.
|
||||
Use the local account name `stats-reader` and enter its password interactively.
|
||||
Never store the resulting password hash in this repository or an installation
|
||||
ZIP.
|
||||
The example opens no GoAccess WebSocket and no additional GoAccess port. Its
|
||||
own access log is disabled and responses use a private, no-store cache policy.
|
||||
|
||||
Do not activate the final HTTPS vhost before its certificate files exist.
|
||||
First install the temporary HTTP bootstrap without changing the parallel apt
|
||||
Certbot installation or either renewal timer:
|
||||
The current private endpoints are:
|
||||
|
||||
- `https://stats.hamradioonline.de/`, which redirects an authenticated request
|
||||
to `/combined/`;
|
||||
- `https://stats.hamradioonline.de/combined/`, which serves the combined
|
||||
report;
|
||||
- `https://stats.hamradioonline.de/kst4contest/`, which serves the site report.
|
||||
|
||||
All HTTPS paths, including the redirect target, remain behind Basic Auth.
|
||||
Reports use `Cache-Control: private, no-store`,
|
||||
`X-Content-Type-Options: nosniff` and `X-Frame-Options: DENY`; dotfiles are
|
||||
blocked. The vhost has no access log and no analytics log. HTTP remains open
|
||||
only for the ACME webroot and permanently redirects every other request to
|
||||
HTTPS. No IPv6 listener is configured while DNS AAAA operation remains
|
||||
unconfirmed.
|
||||
|
||||
The final HTTP/HTTPS vhost is the production configuration. The HTTP-only
|
||||
bootstrap template is retained solely for first provisioning or recovery when
|
||||
the certificate files do not yet exist. In that situation, do not activate the
|
||||
final HTTPS vhost first. Install the temporary bootstrap without changing the
|
||||
parallel APT Certbot installation or either renewal timer:
|
||||
|
||||
```sh
|
||||
sudo install -d -o root -g root -m 0755 /var/lib/letsencrypt
|
||||
@@ -430,34 +729,321 @@ own access log is disabled. Responses use
|
||||
activation date and update time. It contains no IP address, user agent,
|
||||
hostname or per-day detail.
|
||||
|
||||
Use this rollout order:
|
||||
The production URL is
|
||||
`https://kst4contest.hamradioonline.de/visitor-count.json`. Before the first
|
||||
successful generator run, a `404` is expected; afterwards it must return
|
||||
`200` with `application/json`. The home page reveals the counter only after a
|
||||
valid response. A missing, invalid or unavailable counter never prevents the
|
||||
rest of the static site from working.
|
||||
|
||||
1. Install the server files with explicit modes. Prepare directories, log
|
||||
ownership, filters and still-inactive Nginx and systemd configuration.
|
||||
2. Push the website changes, including the Privacy Policy, and let the existing
|
||||
deployment cron job publish them. Until the JSON endpoint exists, the
|
||||
visitor count remains hidden automatically.
|
||||
3. Verify the published Privacy Policy. Only then activate analytics logging,
|
||||
the report generator and timer, the public counter location and the
|
||||
protected statistics vhost.
|
||||
4. Run `nginx -t` before every Nginx reload and perform the real `--check` and
|
||||
`--dry-run` on the server before the first production generation.
|
||||
## Deployment boundary
|
||||
|
||||
A short period in which the Privacy Policy is already visible but logging is
|
||||
not yet active is acceptable. Starting analytics logging before publishing the
|
||||
updated policy is not.
|
||||
GitHub is the source repository for the website and the reviewed analytics
|
||||
templates. The production checkout is `/srv/git/kst4contest`. A root cron job
|
||||
runs `/srv/scripts/deploy-kst4contest-website.sh` every five minutes. It fetches
|
||||
Git, resets the checkout to `origin/main`, runs `npm ci`, builds the Eleventy
|
||||
website, validates VersionInfo, synchronises the result to
|
||||
`/srv/www/kst4contest/current` and finally restores the ownership expected by
|
||||
Nginx.
|
||||
|
||||
## Retention and recovery
|
||||
The deployment credential is stored in `/etc/kst4contest-website.env`. Its
|
||||
value and account assignment must never appear in documentation, logs or
|
||||
support output.
|
||||
|
||||
- Dedicated analytics raw logs: 14 days through the Logrotate example.
|
||||
- Anonymised detailed GoAccess aggregates: rolling 395 days.
|
||||
- Public daily counter values: retained from activation onward.
|
||||
This automatic deployment updates only the static website. It does not install
|
||||
or overwrite:
|
||||
|
||||
Back up the counter state and, if fast report recovery matters, the GoAccess
|
||||
database directories. Reports themselves are derived output. To recover, stop
|
||||
the timer, restore the state and database directories with their ownership,
|
||||
run `--check`, then run `--dry-run` before publishing again.
|
||||
- `/opt/hamradioonline-analytics`;
|
||||
- `/etc/hamradioonline-analytics`;
|
||||
- systemd units;
|
||||
- Nginx or Logrotate configuration;
|
||||
- the Basic Auth password file;
|
||||
- GeoIP configuration;
|
||||
- Certbot configuration.
|
||||
|
||||
The repository contains no password, password hash, MaxMind download key,
|
||||
server IP address, TLS private key or private backup destination. Keep it that
|
||||
way.
|
||||
Changes to those operational files require a separate review and manual
|
||||
installation. The published privacy notice must remain in place whenever
|
||||
analytics logging is active.
|
||||
|
||||
## Regular operation
|
||||
|
||||
Nginx continuously writes only eligible requests to the dedicated analytics
|
||||
log. The systemd timer starts the generator once per hour. Every successful
|
||||
run refreshes the per-site and combined reports, persists the corresponding
|
||||
GoAccess databases, updates daily counter values and finally publishes enabled
|
||||
public counters. Logrotate handles the raw log once per day and preserves the
|
||||
uncompressed `.1` handover file required by the generator.
|
||||
|
||||
The normal operator signal is the service result and journal, not a permanently
|
||||
running process: the generator is a short-lived oneshot service. There is no
|
||||
GoAccess WebSocket process and no public GoAccess port.
|
||||
|
||||
## External services and local credentials
|
||||
|
||||
### MaxMind GeoLite2
|
||||
|
||||
MaxMind is used only to download and update GeoLite2-Country. The local updater
|
||||
configuration is `/etc/GeoIP.conf` with mode `0600 root:root`; the local
|
||||
database is `/var/lib/GeoIP/GeoLite2-Country.mmdb`. GeoLite is enabled and the
|
||||
server has an Account ID and License Key, but neither value belongs in Git,
|
||||
this runbook, screenshots, logs or ordinary diagnostic output.
|
||||
|
||||
`geoipupdate.service` and `geoipupdate.timer` download updates from MaxMind.
|
||||
All visitor lookups then happen locally. The analytics application never sends
|
||||
an individual visitor address to MaxMind.
|
||||
|
||||
If the MMDB is missing, stale or unreadable, inspect the timer and journal,
|
||||
check the file mode and run the updater directly if required:
|
||||
|
||||
```sh
|
||||
systemctl status geoipupdate.timer geoipupdate.service
|
||||
journalctl -u geoipupdate.service --since today
|
||||
sudo stat -c '%U:%G %a %s %y %n' \
|
||||
/etc/GeoIP.conf /var/lib/GeoIP/GeoLite2-Country.mmdb
|
||||
sudo geoipupdate
|
||||
sudo -u hamradio-analytics test -r \
|
||||
/var/lib/GeoIP/GeoLite2-Country.mmdb
|
||||
```
|
||||
|
||||
When `mmdblookup` is installed, a lookup of a neutral public test address can
|
||||
confirm database readability without using any visitor address:
|
||||
|
||||
```sh
|
||||
sudo -u hamradio-analytics mmdblookup \
|
||||
--file /var/lib/GeoIP/GeoLite2-Country.mmdb \
|
||||
--ip 1.1.1.1 country iso_code
|
||||
```
|
||||
|
||||
An unreadable Country MMDB or GoAccess without MMDB support makes `--check`
|
||||
fail with exit code 2.
|
||||
|
||||
### Let's Encrypt
|
||||
|
||||
Let's Encrypt supplies the TLS certificate for `stats.hamradioonline.de`. Its
|
||||
state is below `/etc/letsencrypt/live/stats.hamradioonline.de/`. The ACME
|
||||
webroot is `/var/lib/letsencrypt`, and the final Nginx vhost permanently serves
|
||||
`/.well-known/acme-challenge/` over IPv4 port 80 so webroot renewal continues
|
||||
to work.
|
||||
|
||||
Use the Snap client explicitly as `/snap/bin/certbot` for this vhost.
|
||||
Certificate issuance and a renewal dry-run have been confirmed. The server
|
||||
currently also has an APT Certbot installation and both renewal timers. That
|
||||
duplication is a separate server-maintenance issue; do not change either
|
||||
installation as part of analytics maintenance.
|
||||
|
||||
ACME account data, private keys and certificate state live only on the server
|
||||
and in a future protected backup. Validate renewal with:
|
||||
|
||||
```sh
|
||||
sudo /snap/bin/certbot renew --dry-run \
|
||||
--cert-name stats.hamradioonline.de
|
||||
```
|
||||
|
||||
### GitHub
|
||||
|
||||
GitHub supplies the source repository consumed by the website deployment. It
|
||||
is not an analytics processor and receives no individual analytics request or
|
||||
visit data. The deployment credential remains in
|
||||
`/etc/kst4contest-website.env`; only the server-side deploy process needs it.
|
||||
|
||||
### No external analytics platform
|
||||
|
||||
Statistics are generated locally on the project server. There is no Google
|
||||
Analytics, Matomo Cloud service, external tracking script or transfer of
|
||||
individual visits to an analytics provider.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Start with the safe verification sequence above. Keep secrets out of commands
|
||||
and captured output, and do not weaken file modes merely to make a check pass.
|
||||
|
||||
### `GoAccess JSON report has no geo_location panel`
|
||||
|
||||
The expected GoAccess 1.8.1 key is `geolocation`. An installed older generator
|
||||
which expects `geo_location` must be replaced with the reviewed repository
|
||||
version.
|
||||
|
||||
### Missing Virtual Hosts panel
|
||||
|
||||
The combined report requires the JSON key `vhosts` and must start GoAccess with
|
||||
`--enable-panel=VIRTUAL_HOSTS`. `virtual_hosts` is not a valid replacement.
|
||||
Site reports do not require this panel.
|
||||
|
||||
The names `geo_location` and `virtual_hosts` are allowed in repository tests
|
||||
only as deliberately invalid negative cases.
|
||||
|
||||
### Missing `.1` rotation
|
||||
|
||||
This is normal before the first rotation and whenever no previous rotation is
|
||||
present. The current analytics log remains required.
|
||||
|
||||
### `.gz` rotations on a GoAccess build without Zlib
|
||||
|
||||
This is normal. Regular operation does not read `.gz` files. Do not configure
|
||||
a compressed or rotated file as `analyticsLog`; any exceptional historical
|
||||
import must be planned separately.
|
||||
|
||||
### MMDB missing or unreadable
|
||||
|
||||
`--check` must fail. Verify `/etc/GeoIP.conf`, the GeoIP updater units,
|
||||
`/var/lib/GeoIP/GeoLite2-Country.mmdb` and the service account's read access.
|
||||
Run `geoipupdate` and a neutral local lookup as described above when needed.
|
||||
|
||||
### Analytics log missing or unreadable
|
||||
|
||||
Compare the registry path with the additional `access_log` directive in the
|
||||
site vhost. Check the active Nginx configuration and verify
|
||||
`www-data:hamradio-analytics 0640`. Test Nginx before reloading it. Do not
|
||||
replace or repurpose the normal operational access log.
|
||||
|
||||
```sh
|
||||
sudo nginx -t
|
||||
sudo nginx -T
|
||||
sudo stat -c '%U:%G %a %s %y %n' \
|
||||
/var/log/nginx/kst4contest-analytics.log
|
||||
sudo -u hamradio-analytics test -r \
|
||||
/var/log/nginx/kst4contest-analytics.log
|
||||
```
|
||||
|
||||
### Public counter returns `404`
|
||||
|
||||
This is expected before the first successful generation. Afterwards inspect
|
||||
the service journal, public-output directory, `0644` file mode, Nginx include
|
||||
and exact alias path. A successful response is `200 application/json`.
|
||||
|
||||
### Statistics vhost returns `401` or `404`
|
||||
|
||||
`401` without credentials is correct. With the valid Basic Auth login, `/`
|
||||
must redirect to `/combined/`. A `404` after authentication usually means that
|
||||
the generator has not produced `report.html`, the URL and report directory do
|
||||
not match, or Nginx cannot traverse/read the report path.
|
||||
|
||||
### Certificate error
|
||||
|
||||
Check the DNS A record, certificate paths, active Nginx vhost, ACME webroot and
|
||||
Snap Certbot renewal. Immediately after an Nginx reload, wait briefly and
|
||||
retry if observations conflict, then inspect `nginx -T`, the Nginx journal and
|
||||
worker start times.
|
||||
|
||||
### Generator lock error
|
||||
|
||||
Exit code 3 means that the production lock could not be acquired. Check the
|
||||
service, timer and running processes. Do not remove the lock blindly. Remove it
|
||||
only after confirming that no generator process is active and that the lock is
|
||||
genuinely stale.
|
||||
|
||||
### Counter identity mismatch
|
||||
|
||||
If `activatedOn` or the hostname differs from existing counter state, stop.
|
||||
Do not repair this by deleting the state. Establish the cause and approve a
|
||||
migration or reset explicitly.
|
||||
|
||||
### Exit code 0 but no output
|
||||
|
||||
Check the installed generator size, owner, mode and SHA-256 before anything
|
||||
else. Also look for `Analytics generation completed` in the journal and inspect
|
||||
the expected output files.
|
||||
|
||||
A known operator error is copying a shell prompt or continuation marker `>`
|
||||
with a command. Bash interprets a stray `>` as output redirection. This can
|
||||
truncate `generate-reports.js` to zero bytes and create empty files whose names
|
||||
look like command options. A zero-byte JavaScript file still passes
|
||||
`node --check` and exits with code 0 while doing nothing.
|
||||
|
||||
Identify such artifacts precisely before removing them. Reinstall the
|
||||
generator from the reviewed checkout with the documented owner and mode,
|
||||
compare its size and SHA-256, then repeat `--check` and `--dry-run`. Do not make
|
||||
one-off artifact names or old checksums part of the permanent procedure.
|
||||
|
||||
## Monitoring
|
||||
|
||||
The systemd timer is active, and service results and errors are visible through
|
||||
systemd and the journal. No separate alerting channel has been confirmed.
|
||||
Routine checks should confirm:
|
||||
|
||||
- the timer is active and waiting with a future trigger;
|
||||
- the last service result is `success` and `ExecMainStatus=0`;
|
||||
- the journal contains `Analytics generation completed`;
|
||||
- report and counter timestamps continue to advance;
|
||||
- the public total changes plausibly;
|
||||
- repeated runs do not multiply the existing history.
|
||||
|
||||
Central alerting can be handled later as part of the general server operations
|
||||
plan.
|
||||
|
||||
## Backup and recovery
|
||||
|
||||
There is currently no comprehensive automated server backup plan. The
|
||||
analytics installation must be included explicitly when that server-wide
|
||||
backup and recovery design is implemented. Creating that backup system is not
|
||||
part of this repository change.
|
||||
|
||||
### Data to include
|
||||
|
||||
At minimum, the later plan must cover:
|
||||
|
||||
- `/var/lib/hamradioonline-analytics/public-counter-state.json`;
|
||||
- `/var/lib/hamradioonline-analytics/db`;
|
||||
- `/etc/hamradioonline-analytics`;
|
||||
- the installed systemd units;
|
||||
- Nginx analytics and vhost configuration;
|
||||
- `/etc/logrotate.d/hamradioonline-analytics`;
|
||||
- `/etc/nginx/htpasswd/hamradioonline-analytics`;
|
||||
- Certbot and Let's Encrypt account/certificate state;
|
||||
- `/etc/GeoIP.conf`;
|
||||
- the website deployment configuration and credential storage.
|
||||
|
||||
Handle the Basic Auth hash, MaxMind Account ID and License Key, GitHub deploy
|
||||
token, ACME account data and private TLS keys as secrets. Never copy them into
|
||||
Git, public documentation, logs or ordinary support bundles.
|
||||
|
||||
The generator is recoverable from GitHub, and GeoLite2-Country can be fetched
|
||||
again with `geoipupdate`. HTML/JSON reports can be rebuilt when the GoAccess
|
||||
databases or sufficient raw logs remain. The public JSON can be rebuilt from
|
||||
the counter state.
|
||||
|
||||
The accumulated public total is not fully recoverable without
|
||||
`public-counter-state.json`. Older detailed aggregates are not recoverable
|
||||
without the GoAccess databases, and historical raw requests disappear after
|
||||
the 14-day rotation window.
|
||||
|
||||
Do not let backups extend the published raw-log retention by accident. Either
|
||||
exclude analytics raw logs from durable backups or enforce the same confirmed
|
||||
retention limit in backup storage. Counter state and anonymised/aggregated
|
||||
GoAccess state can be governed separately.
|
||||
|
||||
### Recovery order
|
||||
|
||||
1. Install the operating-system packages and external dependencies.
|
||||
2. Recreate the service account and group relationships.
|
||||
3. Create the directories with the documented owners and modes.
|
||||
4. Install the generator and non-secret configuration.
|
||||
5. Restore secrets and certificate state from protected backup storage.
|
||||
6. Restore GeoLite2-Country or download it again.
|
||||
7. Restore the GoAccess databases and public counter state.
|
||||
8. Validate Nginx, systemd and Logrotate configuration.
|
||||
9. Run the generator with `--check`.
|
||||
10. Run `--dry-run` and verify that production state remains unchanged.
|
||||
11. Start one productive run through the systemd service.
|
||||
12. Verify reports, public counter and Basic Auth.
|
||||
13. Enable the timer only after every preceding check succeeds.
|
||||
|
||||
## Outstanding operational checks
|
||||
|
||||
The first real rotation of the dedicated analytics log still needs explicit
|
||||
observation. This is not a current service blocker. After rotation, confirm:
|
||||
|
||||
- a new current log exists;
|
||||
- `.1` exists and remains uncompressed;
|
||||
- both files retain the expected owners and modes;
|
||||
- `hamradio-analytics` can read both files;
|
||||
- the next generator run succeeds;
|
||||
- values do not show duplicate counting;
|
||||
- the older rotation is compressed on the following cycle as intended.
|
||||
|
||||
The general server backup/recovery implementation and any central alerting
|
||||
remain separate future operations tasks.
|
||||
|
||||
The repository contains no password, password hash, MaxMind credential,
|
||||
deployment token, server IP address, TLS private key or private backup
|
||||
destination. Keep it that way.
|
||||
|
||||
Reference in New Issue
Block a user