Instance maintenance

The admin panel is currently available in Russian only; menu items and button labels in this guide are translated.

All commands go through tracker-ctl, which install.sh places at /usr/local/bin/tracker-ctl. It works from any directory and relies on INOTRACK_HOME (/opt/inotrack by default; docker-compose.yml and .env live there too).

tracker-ctl status                  # docker compose ps
tracker-ctl logs [service...]       # docker compose logs -f --tail=200
tracker-ctl update                  # new version + migrations + rollback on failure
tracker-ctl backup                  # Postgres + ClickHouse + .env + storage, local/S3
tracker-ctl backup --to s3://bucket/prefix   # + offsite copy, verification, rotation (see "Offsite backups")
tracker-ctl backup keygen           # generate BACKUP_ENCRYPTION_KEY (store it off the server!)
tracker-ctl backup schedule daily [--at 03:00] [--to s3://...]   # daily timer
tracker-ctl backup schedule off     # remove the schedule (same as --remove)
tracker-ctl backup schedule         # cron from BACKUP_SCHEDULE_CRON
tracker-ctl restore --from <path|s3://bucket/prefix[/archive]>   # restore from a backup
tracker-ctl domain add|rm <domain>  # tracker domain pool + Caddy re-created
tracker-ctl license info|refresh    # local license status / forced activation-heartbeat
tracker-ctl doctor                  # diagnostics: DNS/TLS/DB/buffer/queue/disk
tracker-ctl uninstall [--purge] [--yes] [--no-backup]   # remove the instance

status / logs

status wraps docker compose ps, logs wraps docker compose logs -f --tail=200 (you can name specific services: tracker-ctl logs tracker-core workers).

doctor

Diagnostics in one command, without reading logs by hand. It checks:

that docker/docker compose are present;
required variables in .env;
the state of the compose containers;
that Postgres/ClickHouse/Redis are reachable;
DNS and TLS of every domain in the pool (panel + tracker domains);
connectivity to the license server;
how fresh the latest backup is;
free disk space;
click buffer lag and the length of the outgoing postback queue (via metrics-probe inside the tracker-core container).

There are no flags — it runs all checks at once and prints a list of what is wrong.

tracker-ctl doctor

ClickHouse: version and the daily report summary

The product works with any ClickHouse version that starts at all — there is no hard minimum. The standard install brings up its own ClickHouse, pinned to 24.8; if you connect a ClickHouse you already run instead (INOTRACK_CH_ADDRS in .env), the version can be older.

The only thing that depends on the version is the daily report summary (inotrack.stats_daily / stats_daily_recent, which speeds up reports over long periods): it is built on REFRESHABLE MATERIALIZED VIEW, which appeared in ClickHouse 23.12. On an older version:

migrations apply in full, the tracker starts and works normally — click intake, postbacks, finance and the panel do not depend on the summary;
on startup the central/worker log shows a WARN saying that this ClickHouse version does not support REFRESHABLE MATERIALIZED VIEW and the daily report summary will not be created;
reports over long periods (a year or more) are computed directly from the raw tables — the numbers are correct, but on large volumes noticeably slower than with the summary (the report engine checks that the summary is ready before every use and never serves partially filled data).

Upgrading to 23.12+ later requires nothing manual: the next central/worker restart applies the skipped part of the migration and fills the summary.

⚠️ Upgrade of the summary's money logic (migration 201_stats_daily_decision.sql). This migration recreates stats_daily/stats_daily_recent with a new SELECT (money according to the network's current decision, not the raw rows) and empties both tables (TRUNCATE) — the old data was computed with the previous formula and, until refilled, would diverge from the raw path. After the migration is applied, the tracker itself starts a one-off SYSTEM REFRESH VIEW in the background and does not block startup; until the summary is filled, reports over long periods automatically fall back to the raw tables — the numbers are correct, just slower until the rebuild finishes (usually minutes).

A separate trap is a rolling deploy across several central/worker replicas: the cache of the summary's readiness (5 minutes) is local to each process, not shared. A replica that has already restarted after the upgrade and sees an empty (not yet filled) summary may wrongly cache it as "healthy" for up to 5 minutes before the summary is actually filled — reports for a UTC period served by THAT replica during that window will show revenue near zero with non-zero spend (the same consequence as the "cold start" in the previous paragraph, but caused by one replica's stale local cache rather than by the summary being empty). The recommended upgrade order in production with several replicas is stop → migrate → start (the standard tracker-ctl update sequence), not a rolling restart one replica at a time: that way the migration and the one-off summary fill finish before any replica starts serving traffic and caching the summary state.

Backup and restore

tracker-ctl backup collects into one .tar.gz: a Postgres dump (always full), a dump of the ClickHouse tables (Native format — see "Incremental ClickHouse backups" below), a copy of .env, the storage volume (uploaded landers and files) and the license directory (machine-id + the daemon's cache, see "Instance fingerprint" below). Optionally it uploads the archive to S3-compatible storage (BACKUP_S3_* in .env) and deletes old backups per BACKUP_RETENTION_DAYS (default 14 days). The schedule is set separately:

tracker-ctl backup schedule daily --to s3://bucket/prefix   # systemd timer, 03:00 (+ up to 30 min)
tracker-ctl backup schedule daily --at 02:30                 # time is the server's local time
tracker-ctl backup schedule            # cron from BACKUP_SCHEDULE_CRON (default "0 3 * * *")
tracker-ctl backup schedule off        # remove any kind of schedule (same as --remove)

daily installs the systemd timer inotrack-backup.timer (journal: journalctl -u inotrack-backup.service; a run missed because the server was off is caught up after boot); on hosts without systemd it adds a cron line (log: BACKUP_LOCAL_DIR/backup-cron.log). Each option first removes the others — there are never two schedules at once. install.sh sets up cron from BACKUP_SCHEDULE_CRON during install; a schedule set via backup schedule daily is left alone by a reinstall.

Offsite backups (S3-compatible storage)

The local BACKUP_LOCAL_DIR lives on the same server: if the disk or the server dies, the backups go together with the data. An offsite copy goes to object storage at another provider:

tracker-ctl backup keygen                              # once; SAVE the key off the server
tracker-ctl backup --to s3://my-bucket/inotrack        # backup + upload + verification + rotation
tracker-ctl backup schedule daily --to s3://my-bucket/inotrack
tracker-ctl restore --from s3://my-bucket/inotrack     # the newest archive in the bucket
tracker-ctl restore --from s3://my-bucket/inotrack/inotrack-backup-incr-20260102-030000.tar.gz

The bucket and prefix go in the URL; the endpoint and access keys go in .env (there are no keys on the command line: it is visible in ps and ends up in the timer unit):

Storage	`BACKUP_S3_ENDPOINT`	`BACKUP_S3_REGION`
AWS S3	`s3.<region>.amazonaws.com` (or empty — derived from the region)	`eu-central-1` etc.
Cloudflare R2	`<account_id>.r2.cloudflarestorage.com`	`auto`
Yandex Object Storage	`storage.yandexcloud.net`	`ru-central1`
Selectel	`s3.<pool>.storage.selcloud.ru`	the pool, e.g. `ru-1`
MinIO	`minio.example.com:9000` (`BACKUP_S3_USE_SSL=0` for http)	`us-east-1`

BACKUP_S3_ACCESS_KEY/BACKUP_S3_SECRET_KEY are keys with read, write, list and delete rights on the bucket. Addressing, BACKUP_S3_URL_STYLE: auto (virtual-host for AWS, path for the rest), path or virtual; a bucket with a dot in its name is always addressed path-style (the provider's wildcard certificate does not cover it).

Encryption. With --to, an archive without BACKUP_ENCRYPTION_KEY is not uploaded (only with an explicit --allow-unencrypted): your whole business goes to a third-party provider. Get a key with tracker-ctl backup keygen (it does not overwrite a key that is already set) or openssl rand -hex 32. The key lives only in the .env of the server, which is exactly the thing that can die: save it in a password manager right away, otherwise the offsite copy is useless. To restore on a new server, put the key and BACKUP_S3_* into its .env before restore.
Integrity check. During upload the storage itself verifies Content-MD5 and the signed SHA-256 of the body (an archive corrupted in transit is rejected); the archive's SHA-256 is written to the object metadata. After the upload, HEAD checks the size and the ETag — the MD5 the storage computed over the bytes it wrote. Where the ETag is not an MD5 (SSE-KMS, SSE-C), the object is downloaded in full and the SHA-256 is recomputed automatically; --verify-download always does this (twice the traffic). An object that fails the check is deleted from the bucket and backup exits with an error (non-zero code, visible in the timer's journal). restore verifies the SHA-256 when downloading.
Chain in the bucket. An incremental is uploaded only if its parent is already stored under the same prefix; otherwise (the first --to into a new prefix, a failed previous upload) a full is made instead of an incremental — the offsite copy is always restorable on its own.
Rotation. After a successful upload, the bucket keeps the last BACKUP_S3_KEEP archives of this instance (0 — never delete; unset — 14 for --to, 0 for uploads via BACKUP_S3_* without --to, so that an update does not suddenly start deleting old copies). An incremental without its full is useless, so rotation does not delete fulls and incrementals that a kept incremental depends on (there may end up being more than N). Every archive is tagged with the instance (metadata x-amz-meta-inotrack-instance, a hash of the INOTRACK_PUBLIC_URL host): rotation does not touch archives of other servers in the same bucket/prefix or archives without the tag. Only objects of the form <prefix>/inotrack-backup-*.tar.gz (and their .manifest.json) are deleted; other files and nested "directories" are left alone. It is still better to give several servers separate prefixes: restore --from s3://bucket/prefix without an archive name, when the prefix holds archives of several instances, takes the newest archive of THIS instance (the tag from INOTRACK_PUBLIC_URL in .env), and if there is none, it refuses and asks you to name the archive explicitly (the name is shown before confirmation). Local archives are cleaned up by BACKUP_RETENTION_DAYS, independently of the bucket.
Without --to, tracker-ctl backup still uploads to BACKUP_S3_BUCKET under BACKUP_S3_PREFIX (default backups) if BACKUP_S3_ENDPOINT and BACKUP_S3_BUCKET are set; an unencrypted archive is uploaded in this mode with a warning, as before.
The upload is a single PUT, without multipart: the S3 limit for a single PUT is 5 GB. If a full archive is getting close to it, reduce the ClickHouse TTL (clickhouse/ttl-policy.sql in the install directory).

Backup growth at 1-10M clicks/day. The five large ClickHouse tables (clicks, conversions, costs, blocked, funnel_steps) are dumped in full once a week, and on the other nights only as a window since the previous backup (details in "Incremental ClickHouse backups" below). Postgres, storage and the small ClickHouse tables are dumped in full on every run. It is still worth keeping the disk for BACKUP_LOCAL_DIR separate from clickhouse-data and watching it (tracker-ctl doctor, the InotrackDiskSpaceLow alert): the weekly full dump of clicks at 10M/day weighs hundreds of gigabytes by the end of the retention period; you can shrink it with the TTL (clickhouse/ttl-policy.sql).

Restore:

tracker-ctl restore --from /opt/inotrack/backups/inotrack-backup-full-20260101-030000.tar.gz
tracker-ctl restore --from s3://my-bucket/inotrack          # the newest archive in the bucket
tracker-ctl restore --from-s3 --from backups/inotrack-backup-incr-....tar.gz   # key in BACKUP_S3_BUCKET
tracker-ctl restore --from <archive> --new-instance   # cloning the instance, see below

--from can point to either a full or an incremental archive — if it is an incremental, restore finds and applies the whole chain itself (the full + intermediate incrementals) up to the given link, see below.

⚠️ The .env from the backup archive is not applied automatically — its secrets may differ (a different instance or an earlier generation). The file is placed alongside as .env.restored for manual comparison; the active .env is not touched.

Restore overwrites the current Postgres/ClickHouse/storage with the backup contents — the command asks for confirmation unless --yes is passed.

Restoring ClickHouse and refreshable MVs

The daily summary (stats_daily_mv, stats_daily_recent_mv) is a REFRESHABLE MATERIALIZED VIEW. On the pinned ClickHouse version 24.8, any operation on them (creation, RESTORE, SYSTEM REFRESH VIEW) requires the setting allow_experimental_refreshable_materialized_view = 1, otherwise the server answers Code: 344 (SUPPORT_IS_DISABLED) (since 24.10 the feature is no longer experimental; the setting name is still accepted and breaks nothing).

tracker-ctl restore sets the setting itself: it checks whether the server knows it (system.settings) and passes it to every restore call of clickhouse-client. On ClickHouse older than 23.12 (no refreshable MVs at all) the setting is not passed.
Migrations (tracker-ctl migrate, central/worker startup) set it in the CREATE MATERIALIZED VIEW ... SETTINGS allow_experimental_refreshable_materialized_view = 1 itself — a clean install does not need a separate setting.
Manually (native ClickHouse BACKUP/RESTORE, replaying DDL from SHOW CREATE and so on) — set it in the session before restoring:
```
SET allow_experimental_refreshable_materialized_view = 1;
RESTORE DATABASE inotrack FROM Disk('backups', 'inotrack.zip');
```
With clickhouse-client, as an argument: clickhouse-client --allow_experimental_refreshable_materialized_view=1 --query "RESTORE ...". If objects are created one by one, the order is: first the tables (the sources clicks, conversions and the targets stats_daily, stats_daily_recent, stats_hourly), then their data, and only then the views (stats_hourly_mv, stats_daily_mv, stats_daily_recent_mv): a view with TO needs an existing target table, and a regular MV created before the sources are loaded would add aggregates on top of the restored ones. After that, SYSTEM REFRESH VIEW inotrack.stats_daily_mv (and stats_daily_recent_mv) recomputes the summary right away instead of waiting for the schedule.

tracker-ctl restore does not restore the schema: the dump contains only data, the tables and views are created by tracker-ctl migrate, so the command keeps the "tables → data → MV recompute" order by itself.

Incremental ClickHouse backups

At 1–10M clicks/day (≈300M rows/month, click TTL 13 months) a full daily dump of the largest ClickHouse tables grows without bound in time, IO and disk. That is why not everything is dumped in full:

Always full, on every run: Postgres, the storage volume, the license directory and the small/aggregate ClickHouse tables (stats_hourly, landing_events, smartlink_clicks, stats_daily, stats_daily_recent) — they are either small or do not grow in proportion to traffic.
Incrementally, by a time window — the five largest event tables: clicks, conversions, costs, blocked, funnel_steps. The window is built on the ts column (event time) for all of them except costs — there the window is on synced_at (when the row was actually written), not on day: a resync of a cost source can recompute spend for a day a month back, and such a correction must get into the next incremental instead of being lost until the next full. The window overlaps the previous backup's watermark by 2 hours — a margin for delivery delays of the click buffer/postback queue; re-inserting rows that already made it into the previous backup is safe (clicks, conversions and costs are ReplacingMergeTree, and ClickHouse collapses duplicates on merge/FINAL; for blocked and funnel_steps see the warning below).

Mode:

tracker-ctl backup                 # auto: full once a week, incremental otherwise
tracker-ctl backup --full          # force a full
tracker-ctl backup --incremental   # force an incremental (with no history it still makes a full)

Without a flag (which is what the cron schedule runs) the command decides itself: if the newest full is older than 7 days (or there has been none yet) it makes a full, otherwise an incremental from the latest backup in the chain. The schedule (BACKUP_SCHEDULE_CRON) sets how OFTEN backup runs (usually daily); which of those runs is a full is decided by the command — the result is a full once a week + an incremental daily, with no extra settings.

Disk estimate: at 5M clicks/day an incremental dump of the big five holds roughly one day of data (plus the 2-hour overlap), not the whole 13-month history — two orders of magnitude less than a full dump of the same tables. A full dump is needed only once a week; over a week the total backup volume (1 full + 6 incrementals) is noticeably smaller than the seven daily full dumps taken before.

The manifest (manifest.json inside the archive + an unencrypted sidecar <archive>.manifest.json next to it) stores: the backup type, a link to the parent in the chain (parent_archive), the window [watermark_from, watermark_to), the list of tables dumped by window, and the sha256 of the dump of each of the big five.

Restoring a chain: tracker-ctl restore --from <any archive of the chain> follows parent_archive back to the full, unpacks every link, verifies the checksums and the continuity of the chain (matching parent_id and no time gap between one link's watermark_to and the next link's watermark_from) — and fails with an explicit error if an archive is missing somewhere in the chain (deleted by retention or by hand) or a manifest is damaged, instead of silently restoring incomplete data. Then: Postgres/storage/license/ small CH tables are taken from the last link of the chain (they are full in every backup anyway); the big five are restored through the whole chain in order (the full, then each incremental — with a plain INSERT, no TRUNCATE between links), and ReplacingMergeTree/FINAL collapse the overlapping duplicates. Archives without manifest.json (taken before incremental backups existed) are still restored as before — as a self-contained full.

⚠️ blocked and funnel_steps are a plain MergeTree (no version column), not ReplacingMergeTree: unlike clicks/conversions/costs, re-inserting overlapping rows MAY leave a small number of duplicates within the two-hour overlap after restoring a chain of several links. For these two tables (diagnostic logs of blocks and funnel steps, not money) this is an accepted trade-off — just like the ReplacingMergeTree behaviour described above (clicks/conversions/costs), where temporary duplication is also possible until a merge/FINAL.

Retention (BACKUP_RETENTION_DAYS) deletes backups older than the given age by file date, but never deletes a full that a not-yet-deleted incremental still depends on through the chain — otherwise the chain would break silently (a full is made once a week and usually lives longer than the retention horizon, and every incremental of that week counts from it).

Backup encryption

The archive holds a full Postgres/ClickHouse dump (offers, payouts, advertisers, traffic domains, finance, user password hashes) and the license machine-id (see "Instance fingerprint" below). A leak of ONE archive file is your entire business, whoever ends up with it: on the server's disk, in an S3 bucket, in an email you copied it to by hand.

Set BACKUP_ENCRYPTION_KEY in .env — the archive will be encrypted with AES-256-GCM before it is written to disk (streamed, in frames; so encryption does not need free memory equal to the archive size — it works on dumps of tens of gigabytes too):

openssl rand -hex 32   # put the result into .env as BACKUP_ENCRYPTION_KEY

Keep this key SEPARATE from the backups — in a password manager, your organization's vault, anywhere, but not next to the archives (not in the same directory, not in the same S3 bucket, not in the same email). The point of encrypting a backup is the assumption that the backup medium itself may be compromised (a stolen disk, a breached bucket); if the key lies in the same place, that assumption buys nothing. For its part, tracker-ctl does not put BACKUP_ENCRYPTION_KEY into the archive: the copy of .env in the archive (env.backup, see below) has this key REPLACED with a placeholder — keeping the key itself safe is entirely the operator's responsibility. A lost key is a lost backup: the archive cannot be decrypted without the key (that is the point of encryption), and tracker-ctl restore will then fail with a clear error rather than silently corrupt data. Check periodically that the key is still there and that restoring with it actually works (tracker-ctl restore --from <fresh archive> --new-instance on a test host is a safe way to check without touching production).

If BACKUP_ENCRYPTION_KEY is not set, tracker-ctl backup does not refuse to make a backup — it runs as before, unencrypted, but warns loudly about it on every run (in the console and in backup-cron.log if the backup runs on schedule). Refusing to back up because there is no encryption key would be worse: an instance that simply had not set up a key yet after a tracker-ctl update would be left with no backups at all — for disaster recovery that is a worse risk than an unencrypted archive (the same risk all backups of this product had before BACKUP_ENCRYPTION_KEY existed).

Restore detects the archive format from the file contents, not from its extension or name — tracker-ctl restore handles both old archives taken before you set a key and new encrypted ones. An encrypted archive needs BACKUP_ENCRYPTION_KEY in the target instance's .env (the same key the archive was encrypted with — when moving to a new server, copy it into the new host's .env in advance, before tracker-ctl restore). A wrong key is an explicit error BEFORE unpacking starts, not corrupted data on disk.

Restore and the license (machine-id) — default behaviour and `--new-instance`

The backup carries the license directory (machine-id, see "Instance fingerprint" below), and restore restores it by default — the target instance gets THE SAME license fingerprint the instance had when the copy was taken. This is a deliberate choice for the one scenario tracker-ctl restore exists for — disaster recovery (a dead disk, moving to new hardware): without restoring machine-id, the license server would see a restore from backup as a new install and charge an extra activation against the limit on every restore, even though physically it is the same customer.

The product cannot automatically tell disaster recovery from cloning an instance (a second parallel copy of the product from the same backup) — both look the same: the archive is deployed on another host. If you really need a clone (for example, a staging copy for testing) rather than a replacement of the original, pass the --new-instance flag — it skips restoring the license directory, and on the next start the daemon generates its own machine-id (a new fingerprint; a separate activation will be needed). Without this flag the clone will present itself to the license server as the same machine as the original — the server will treat that as license circumvention.

The rule is simple: moving/restoring an existing instance — a plain restore, pass nothing; making a second copy next to a running original — always --new-instance. Both restore and restore --new-instance print an explicit message about what happened to machine-id — the behaviour does not depend on the command guessing the operator's intent.

Adding/removing a tracker domain

From the panel (the main way)

"Tracker domains" section → "Add domain", then point the domain's A/CNAME record at this server (or use "Connect via Cloudflare" if a token is set). Nothing else is needed:

docker compose (Caddy). The Caddyfile has an https:// block with tls { on_demand }. On the first TLS handshake for a new name, Caddy asks the core GET http://tracker-core:8081/internal/tls/ask?domain=<name> and issues a certificate only if the domain is in the pool and enabled (answer 200; otherwise 404 — unknown names get no certificate). The endpoint exists only on the admin port 8081 and is not published outside. Installs older than this version need a fresh Caddyfile — tracker-ctl update brings it.

k3s (Traefik + cert-manager). Traefik has no on-demand TLS, so the core maintains the Ingress inotrack-pool-domains itself (env INOTRACK_K8S_POOL_INGRESS on the tracker-core Deployment): one rule and a separate tls block per active pool domain → Service tracker-core:8080. It is updated right after a domain is added/edited/removed in the panel and again on every health-check tick. cert-manager creates a Certificate per domain — a domain that fails validation does not break its neighbours' certificates. Permissions: the tracker-core ServiceAccount + the inotrack-domains-ingress Role (only ingresses in its own namespace), as in the tracker-core manifest. Settings:

env	default	purpose
`INOTRACK_K8S_POOL_INGRESS`	empty (off)	name of the pool Ingress
`INOTRACK_K8S_POOL_ISSUER`	`letsencrypt-http01`	ClusterIssuer. HTTP-01 works for any domain pointing at the cluster; DNS-01 (`letsencrypt-inotrack`) — only for zones in the Cloudflare account that issuer is configured with
`INOTRACK_K8S_POOL_TLS_OPTIONS`	`traefik-direct@kubernetescrd`	Traefik TLSOption; `-` — no annotation. Without it the cluster's origin lock breaks the handshake
`INOTRACK_K8S_POOL_SERVICE` / `_SERVICE_PORT`	`tracker-core` / `8080`	where to proxy
`INOTRACK_K8S_POOL_INGRESS_CLASS`	`traefik`	ingressClassName

Domains that already have their own static Ingress also work in the pool: both routes lead to tracker-core:8080, the certificate is just issued twice.

The panel shows "not served by this server" only AFTER the first health check, when https://<domain>/healthz was answered by something other than this tracker (PointsToTracker=false). Such a domain cannot be made primary (the button is disabled, the core answers 409). Before the first check there is no badge — there is nothing to judge by; run "Check now".

Links for smartlinks, landers, funnels and flows are built from the pool's primary domain (GET /domains/tracking-base, the tracking_url field in API responses); INOTRACK_PUBLIC_URL is only a fallback while there is no primary domain.

Statically, by command

tracker-ctl domain add track2.example.com
tracker-ctl domain rm track2.example.com

It edits TRACKER_DOMAINS in .env and re-creates the Caddy container with the new list (a brief restart of the TLS front end; other services are not touched) — no manual config edits are needed; Caddy gets the TLS certificate automatically on the first request to the domain. After add, don't forget to point the domain's A record at this server. This is for installs with an old Caddyfile without the on-demand block and for domains outside the panel's pool.

You cannot remove the last domain from TRACKER_DOMAINS — add a new one first, otherwise the static Caddy block would be left with no domain at all.

Updating

tracker-ctl update                 # show versions, ask for confirmation, update
tracker-ctl update --yes           # no question (scripts; without a terminal --yes is required)
tracker-ctl update --yes --backup  # tracker-ctl backup first; if it fails, the update does not start
tracker-ctl update --check         # check only: exit code 0 — up to date, 10 — a newer one exists

It asks the license server for the channel's version (INOTRACK_UPDATE_CHANNEL: stable by default or beta), writes it to INOTRACK_VERSION, pulls the images, runs the migrations inside the new image and waits until tracker-core is healthy. A failure at any step means an automatic rollback to the previous version (INOTRACK_VERSION in .env is restored, the old images are already on disk). update renews an expired license token on the host by itself.

What update deliberately does not do:

it does not downgrade. If 1.3.0 is installed and the channel has 1.2.0 (you switched beta → stable), it reports this and exits: schema migrations do not go backwards;
it does not jump over min_supported. If the installed version is below the minimum for a direct upgrade, update refuses and names an intermediate version: install it (INOTRACK_VERSION=<intermediate> in .env, docker compose pull && up -d, migrations — tracker-ctl migrate), then run tracker-ctl update again.

Releases and channels

Versions are announced by the license server (the versions feed), not by the registry: an instance installs only what the server named for its channel. The image is always pinned to a specific version — there are no floating tags in .env, and docker-compose.yml does not start without INOTRACK_VERSION.

What	What it is	Who gets it
`X.Y.Z`	a release version	the `beta` channel first, `stable` after it has been tested
`edge-<sha>`	a pre-release build	install.sh takes the latest one only while neither `stable` nor `beta` has been published
`:stable`, `:edge`	floating `tracker-ctl` (and `tracker`) tags in the registry	only install.sh, to fetch `tracker-ctl` before the license is activated; they never end up in `.env`
`tracker-l<id>:X.Y.Z`	the shared `X.Y.Z` + a layer with the license watermark	licenses with stamping (`INOTRACK_IMAGE` is set by install.sh and `update`)

The stable channel is never downgraded, and beta is never below stable. Each release has its own min_supported (if none is set — X.0.0 of the same major version).

After a successful update, tracker-ctl on the host is replaced with the binary from the new image (the previous one stays next to it as tracker-ctl.prev; to roll back — mv tracker-ctl.prev tracker-ctl).

Licensee watermark in the image

Every license with stamping (new ones by default) gets its own image registry.inotrack.run/inotrack/tracker-l<id>: the tracker binary carries <licensee_id>:<license_id>:<hmac> (visible in docker run --rm --entrypoint /usr/local/bin/tracker <image> version and in the panel's /version as build_stamp; it is sent in the heartbeat). The license server flags someone else's stamp in a heartbeat as build_stamp_leaked on the source license. Only the tracker is stamped; the panel (admin) and tracker-ctl are shared.

A license issued between releases is installed on the shared image; as soon as its personal image is ready, tracker-ctl update moves the instance to it, and the shared image is closed to that license.

An install made before the first release (stable/beta empty) pins edge-<sha> with a loud warning; as soon as stable is published, tracker-ctl update (or auto-update) moves such an instance onto it.

Auto-update

INOTRACK_AUTO_UPDATE in .env (or install.sh --auto-update ...); after editing it, run tracker-ctl update schedule:

Mode	What it does once a day in `INOTRACK_UPDATE_WINDOW` (default `04:00`, local time, + up to 15 min)
`off`	nothing
`notify` (default)	`update --check --notify`: writes to the journal and, once per version, to Telegram that a new one is out
`install`	`update --yes --backup --notify`: backup, update, automatic rollback on failure; the result goes to Telegram

This installs the systemd timer inotrack-update.timer (journal: journalctl -u inotrack-update); on hosts without systemd, a cron line (log: backups/update-cron.log). To remove: tracker-ctl update schedule --remove. Telegram uses the bot from INOTRACK_TELEGRAM_TOKEN and the numeric chat INOTRACK_TELEGRAM_CHAT; if they are not set, the result goes only to the journal.

The current version and "version X available" are shown in the admin panel: a badge in the sidebar and "Settings → About". The core checks with the license server once an hour.

Manual rollback

Automatic rollback covers a failure of the update itself. If a problem surfaces later:

tracker-ctl backup                       # snapshot of the current state
sed -i 's/^INOTRACK_VERSION=.*/INOTRACK_VERSION=1.1.0/' /opt/inotrack/.env
cd /opt/inotrack && docker compose pull && docker compose up -d

If there were schema migrations between the versions, rolling back the image does not undo them: restore the backup taken before the update (tracker-ctl restore --from ...; the install mode takes it by itself). While you investigate, set INOTRACK_AUTO_UPDATE=off and run tracker-ctl update schedule, otherwise the timer will update the instance again at night.

License

tracker-ctl license info       # status of the locally stored token
tracker-ctl license refresh    # forced activation/heartbeat to the license server

license info shows: until when the local token is valid, when the last successful contact with the license server was, the length of the offline grace period (72 hours by default) and the deadline until which the instance can stay offline without degradation, the enabled features and limits.

Exactly what goes to the license server on each heartbeat (and what is deliberately not included — offers, creatives, traffic domains, finance) — see the FAQ, section "What telemetry does the instance send to the license server?".

Instance fingerprint (machine-id) — don't lose this directory

The license is bound to a "fingerprint" (fingerprint): sha256 of a persistent machine-id + the list of tracker domains. machine-id is not the host's /etc/machine-id but a separate file next to the license cache; its path is INOTRACK_LICENSE_CACHE_PATH (/var/lib/inotrack/license/state.json by default, so machine-id lives in /var/lib/inotrack/license/).

In docker-compose.yml this directory is a bind mount of the host path /var/lib/inotrack/license (the only one of the core's volumes that is NOT a named docker volume, unlike geoip/storage/wal) — on purpose, so that tracker-ctl running directly on the host and the daemon in the containers (tracker-core, workers) read the same file and produce the same fingerprint. If you lose this directory (recreate the host without moving /var/lib/inotrack/license and without tracker-ctl restore), the next start generates a NEW id — the license server will see it as license cloning or an extra activation, not as a continuation of the same install. tracker-ctl backup/restore include this directory automatically (see "Backup and restore" above, the part about --new-instance) — a manual move is needed only if you move the disk/files around tracker-ctl (for example, rsync of the whole install directory): in that case move /var/lib/inotrack/license by hand together with the rest of the data.

⚠️ The backup archive contains machine-id — a secret and an install identifier in itself. Whoever gets the archive file gets someone else's license fingerprint. Set BACKUP_ENCRYPTION_KEY (see "Backup encryption" above) — without it the archive stays a plain .tar.gz with 0600 permissions (only the host's file ACL), both locally and when uploaded to S3 (BACKUP_S3_*) — the SigV4 signature protects the transfer channel and key-based access to the bucket, but not the object's contents once it is there. Even with encryption, restrict access to BACKUP_LOCAL_DIR and to the BACKUP_S3_BUCKET bucket as strictly as to the license itself — encryption protects against a leak of the MEDIUM, it does not replace access control over it.

In Kubernetes the same thing is solved differently: machine-id is not a file on the pod's disk (with HPA at 2-8 tracker-core replicas a regular PersistentVolume does not work — ReadWriteOnce mounts to one pod only, the rest hang in Pending), but a Secret mounted read-only into all tracker-core/workers pods with identical contents (the inotrack-license.machine_id key) — read-only Secret volumes are by design available to any number of replicas without StorageClass requirements, and the tracker never tries to overwrite an existing non-empty file.

Removing the instance

tracker-ctl uninstall               # interactive confirmation, backup before removal,
                                     # data (Postgres/ClickHouse/Redis/storage volumes) stays
tracker-ctl uninstall --purge       # + deletes the data volumes and the whole install directory
tracker-ctl uninstall --no-backup   # don't take a backup before removal
tracker-ctl uninstall --yes         # don't ask for confirmation

Without --purge the instance can be brought back with the same .env — the data stays on disk. --purge deletes data irreversibly. Without --yes, uninstall (with or without --purge) asks you to type the instance name or yes to confirm.

User management

The first administrator is created by the installer from INOTRACK_ADMIN_EMAIL/_PASSWORD (this happens once, while the users table is empty). The owner (the admin role) adds the rest of the staff the standard way, by invitation — without editing the DB and without a restart (only the admin role can do this):

# invite (owner's session cookie — see the quickstart, appendix "the same path through the API", step 0)
curl -s -b cookies.txt -X POST "$TRACKER/api/internal/auth/users/invite" \
  -H 'Content-Type: application/json' -d '{"email":"[email protected]","role":"buyer"}'
# the response has a token, returned ONLY ONCE. Link for the invitee: https://<panel>/invite?token=<token>
# (valid for 7 days, the invitee sets their own password via the link — public POST /invite/accept)

curl -s -b cookies.txt "$TRACKER/api/internal/auth/users"                       # list
curl -s -b cookies.txt -X PATCH "$TRACKER/api/internal/auth/users/<id>/role" \
  -H 'Content-Type: application/json' -d '{"role":"viewer"}'                    # change role
curl -s -b cookies.txt -X POST "$TRACKER/api/internal/auth/users/<id>/disable"  # disable
curl -s -b cookies.txt -X POST "$TRACKER/api/internal/auth/users/<id>/enable"   # restore access

Disabling forces re-login immediately (it revokes all of the user's sessions), rather than just forbidding the next sign-in. You cannot demote or disable the last active admin — 409 conflict: the product does not let you lock the team out without an owner.

The same is available in the panel, in the "Users" section (/users): the staff list, invitations, role changes, disabling and restoring access, resetting 2FA. The curl calls above are for automation.

Roles: admin — full access and team management; buyer sees only their own flows/sources (flows.owner_user_id); viewer — the same, but without the right to change anything.

Administrator lost their 2FA device

Two standard ways (the detailed rationale is in the FAQ):

A second admin resets the first one's 2FA, without a restart: POST /api/internal/auth/users/<id>/2fa/reset (under the second admin's session, not for themselves).
A single administrator — the emergency variable INOTRACK_AUTH_RESET_2FA=email:label in .env; it needs a process restart (that is, physical access to the server). The label is any non-empty string unique to the incident (a date/ticket number): without it, the next ordinary restart would wipe 2FA again, after it had already been set up anew. Right after signing in, set up 2FA again and remove the variable — it does not clear itself.

POST /2fa/disable is still unconditionally 403 for the admin role (2FA is mandatory for the owner) — this is not the same as the ways above; you cannot remove it from yourself with a password or a recovery code, only by one of the two ways above. For buyer/viewer, self-service works as usual (recovery code → sign in → 2fa/disable with the password → 2fa/setup again).

Environment variables

All of them are in /opt/inotrack/.env (permissions 600). There are about a hundred variables across all modules — below is not the full list but what an instance owner really needs to know: what the installer fills in itself, what to check for your configuration, and what not to touch at all.

Filled in by the installer — usually not edited by hand

install.sh, interactively or via flags (install guide), writes to .env by itself: domains, the first administrator's e-mail/password (INOTRACK_ADMIN_EMAIL/_PASSWORD, read once while the users table is empty), the license key (INOTRACK_LICENSE_KEY), Postgres/ClickHouse/Redis passwords, the JWT secret, the instance role (INOTRACK_ROLE: central/edge), the time zone and base currency (INOTRACK_TZ, INOTRACK_BASE_CURRENCY), ports (INOTRACK_HTTP_ADDR :8080, INOTRACK_ADMIN_ADDR :8081), REDIS_MAXMEMORY (25% of host RAM, at least 1gb — computed only on the first install; on a reinstall over the same directory an already set value is kept, see "Redis growth estimate" in the install guide).

⚠️ One thing is worth checking by hand after install: INOTRACK_PUBLIC_URL — the domain substituted into tracking links. If by mistake it points to the panel domain rather than a tracker domain, links will go to the wrong place while the install looks formally "green" (see the common problems in the install guide).

Change for your configuration — if you use the corresponding feature

Password strength requirements for new passwords (applies when a password is set — a password change, accepting an invitation, bootstrapping the first admin; it does not affect sign-in itself): INOTRACK_PASSWORD_MIN_LEN (default 8 characters), INOTRACK_PASSWORD_ALLOW_COMMON=1 — turn off the check against the list of predictable passwords (a deliberate weakening for a test environment/trusted network). On a clean install a weak INOTRACK_ADMIN_PASSWORD does not block creating the first administrator (otherwise the product would lock out its own customer from the very first start) — instead an ERROR is written to the log on startup, demanding that the password be changed right after the first sign-in.
Emergency access recovery for an administrator who lost their 2FA device: INOTRACK_AUTH_RESET_2FA=email:incident-label — see "Administrator lost their 2FA device" above and the details in the FAQ. When unset it does nothing on startup; it is set only for the duration of an incident and removed right after.
Restricting panel access by IP: INOTRACK_ADMIN_IP_ALLOWLIST (IP/CIDR, comma-separated) — closes the entire admin perimeter, including the sign-in form itself. Unset or empty — no restriction (default). The list is read only from the process environment, never from the database or the UI: this is deliberate, so that a mistake in a range is fixed by editing the configuration and restarting, rather than requiring a sign-in to an app that no longer lets you in. Before enabling it, check that your own address is in the list and that it is static. The public API with Bearer tokens is deliberately not covered by this restriction.
Running behind Cloudflare/another proxy: INOTRACK_TRUSTED_PROXIES (CIDR, comma-separated) — which addresses to trust for X-Forwarded-For/X-Real-IP; INOTRACK_CLOUDFLARE_IPS — the same for CF-Connecting-IP. Without this the client IP is taken from an untrusted source — a hole for spoofing geo and bypassing the postback IP allowlist (postbacks).
Cloud backups: BACKUP_S3_* (endpoint/bucket/keys/prefix/addressing, BACKUP_S3_KEEP — how many latest archives to keep in the bucket), BACKUP_RETENTION_DAYS (default 14), BACKUP_SCHEDULE_CRON (default 0 3 * * *, set from .env.example at install; tracker-ctl backup schedule fails with a clear error if the line in .env is empty).
Updates: INOTRACK_UPDATE_CHANNEL (stable by default, beta if you deliberately want fresher builds), INOTRACK_AUTO_UPDATE (off/notify/install, default notify), INOTRACK_UPDATE_WINDOW (default 04:00), INOTRACK_TELEGRAM_CHAT — the chat for notifications; see "Updating" above.
Report SQL sandbox (API guide): INOTRACK_CH_SANDBOX_PASS — without it the /api/v1/reports/sql endpoint is off.
Demo traffic for testing: INOTRACK_SANDBOX=1 — only on a non-production instance.
Push subscriptions, domain manager, ad account integrations, crypto payouts: each module has its own key for encrypting secrets in the DB — INOTRACK_PUSH_ENC_KEY, INOTRACK_DOMAINS_ENC_KEY, INOTRACK_INTEG_ENC_KEY, INOTRACK_EXPORT_ENC_KEY, INOTRACK_RANKTRACKER_ENC_KEY, and separately INOTRACK_FINANCE_TRONGRID_KEY/_ETHERSCAN_KEY/_BSCSCAN_KEY for receiving crypto payouts (TRC20/ERC20/BEP20). Without its module's key the corresponding feature simply does not store secrets — not an error if you don't use it. The push service additionally has INOTRACK_PUSH_CONTACT_MAILTO (optional) — the contact e-mail in the VAPID JWT (RFC 8292), and the domain manager has INOTRACK_DOMAINS_CHECK_INTERVAL_SEC (how often a domain is health-checked; see the FAQ on the fact that failover in this module today means monitoring and the Cloudflare connection, not automatic switching of real traffic).
Telegram alerts: INOTRACK_TELEGRAM_TOKEN/_CHAT.
TTL of the click cache for postback matching: INOTRACK_CLICK_CACHE_TTL (default 30m, Redis key click:<clickid>) — 0 turns the cache off (matching only via ClickHouse, colder). It affects the Redis footprint — see "Redis growth estimate" in the install guide.
Automatic GeoIP database updates: they work out of the box with no settings at all — by default (none of the variables below set) the product downloads the free DB-IP Lite database itself (CC BY 4.0 license, attribution is already shown in the panel, "Reports" section) every INOTRACK_GEOIP_UPDATE_INTERVAL_HOURS (default 24h), checks the file's integrity and atomically swaps the working database, without restarting the process. A failed download (network unavailable, the source returned something that is not .mmdb) writes an ERROR to the log and keeps the previous/empty geo — click intake never stops under any circumstances. A completely empty database (first start/auto-update off) is visible: one WARN on startup, the gauge inotrack_geoip_loaded (0/1) and tracker-ctl doctor.
- INOTRACK_GEOIP_CITY_URL/_CITY_V6_URL/_ASN_URL — override the source (your own mirror or a direct link to a paid-license file). _CITY_V6_URL is needed only for a source with separate IPv4/IPv6 files (DB-IP); for a source with a single combined City file (including MaxMind) don't set it at all.
- INOTRACK_GEOIP_MAXMIND_LICENSE_KEY — an optional MaxMind GeoLite2 license key (more accurate than DB-IP Lite). If it is set and none of the three URLs above is set explicitly, the resolver builds the official MaxMind permalink URLs itself and unpacks their tar.gz. ⚠️ The vendor has no MaxMind key of its own — this path has been tested only with synthetic data, not with a live download from download.maxmind.com; if you use it, be sure to check tracker-ctl doctor and real IP resolution after the first start. The Kubernetes manifests do not include this key yet (just like INOTRACK_TELEGRAM_TOKEN — an external credential needed by a minority of instances); for a Kubernetes install with paid MaxMind, add it to your secrets the same way as INOTRACK_LICENSE_KEY.
- INOTRACK_GEOIP_AUTOUPDATE=off — turns off background downloading entirely (air-gapped/ regulated environments where outgoing traffic to GitHub/MaxMind is undesirable) — then .mmdb files are placed into INOTRACK_GEOIP_PATH by hand.

⚠️ One variable needs attention at first setup, even if anti-fraud/first_click don't seem a priority: INOTRACK_IP_HASH_SALT — the salt of the IP hash in anti-fraud and identity matching. If it is unset, a new random one is generated on every process restart, and matching visitors by IP hash across restarts (including first_click) stops working. Set it once at install and don't change it.

Internal tuning — don't touch without a clear reason

The click intake buffer (INOTRACK_CLICK_FLUSH_MS, _SHUTDOWN_FLUSH_MS, _RING_SIZE, _BATCH_SIZE, _OVERFLOW_QUEUE), on-disk paths (INOTRACK_WAL_PATH, _STORAGE_PATH, _GEOIP_PATH), the log level (INOTRACK_LOG_LEVEL, default info), metrics exposure (INOTRACK_METRICS, default 1). The defaults are sized for production load and verified by load tests; change them only if tracker-ctl doctor or the metrics clearly show a shortage.

⚠️ Separately — INOTRACK_CLICK_CONFIRM_INSERT (default true): this is not only about performance but about whether clicks get lost silently. The ClickHouse connection is asynchronous (async_insert=1), and without confirmation the server may report "ok" before physically writing the batch — if ClickHouse crashes at exactly that moment, the batch disappears without a trace, bypassing both ClickHouse and the WAL (details and measurements: FAQ, the section on ClickHouse being unavailable). =false is a deliberate operator choice for throughput above typical load (~500–640 thousand inserts/s instead of ~130–150 thousand/s), not something worth turning off by default.

Browser check difficulty (guard)

INOTRACK_GUARD_POW_BITS — the proof-of-work difficulty of the guard challenge, the number of leading zero bits (2^N attempts in the browser on average). Optional, default 14. The allowed range is 8–24: a value below 8 is raised to 8, above 24 lowered to 24, a non-number means the default 14; in each of these cases a WARN is written to the log. The value is read lazily — on the first challenge/cookie check, not at process start, so the WARN appears in the log only after the first request that hits the challenge (an instance without such traffic won't show it). Changing it requires a restart.

Installation — the install guide. Honest answers to questions about reliability — the FAQ.