# Monitoring and troubleshooting This page covers health checks, logs and known symptoms with their fixes. ## Health checks Both images declare a Docker health check. Read the result with: ``` docker compose ps ``` ### Backup container The backup container is healthy while its heartbeat file is younger than `HEARTBEAT_MAX_AGE_SECONDS` (180 seconds). The file is `HEARTBEAT_FILE`, by default `/tmp/telegram-archive.heartbeat`. A missing file counts as unhealthy. Only the `schedule` command writes the heartbeat. It writes it every 30 seconds and starts before it connects to Telegram. A long first backup therefore stays healthy. Docker runs the check every 60 seconds, with a 300-second start period and 3 retries. > [!WARNING] > > **A healthy container can still fail every backup** > > The heartbeat proves that the scheduler's event loop runs. It does not prove that Telegram is reachable, that the database works or that a backup succeeded. A container can be healthy while every run fails. To see whether backups succeed, read the logs and [Archive status](<#archive-status>). Two setups read as unhealthy even when nothing is wrong: - A container that runs the one-shot `backup` command instead of `schedule` writes no heartbeat. Docker marks it unhealthy about three minutes after the 300-second start period ends, once three checks in a row have failed. - A read-only root filesystem without a tmpfs at `/tmp` leaves the scheduler nowhere to write the heartbeat. The container stays unhealthy for good. Every 30 seconds, the backup log repeats `Could not write heartbeat: OSError`. A `/tmp` that exists but is not writable by uid 1000 logs `PermissionError` instead. The stock compose file mounts that tmpfs. ### Viewer container The viewer checks `HEALTHCHECK_URL`, by default `http://127.0.0.1:8000/api/health`, every 30 seconds, with a 30-second start period and 3 retries. The endpoint needs no login. It answers `{"status":"ok","database":"connected"}`, or HTTP 503 with `{"status":"degraded","database":"unreachable"}` when a database query fails. From the host running the stock compose file: ``` curl http://127.0.0.1:8000/api/health ``` ### Check from a script `/api/health` only says whether the database answers. For backup state, call `/api/status`, the endpoint behind the viewer's Archive status page. It needs the master login's session cookie: ``` curl -fsS -c cookies.txt -H 'Content-Type: application/json' \ -d '{"username":"admin","password":"choose-a-long-password"}' \ http://127.0.0.1:8000/api/login curl -fsS -b cookies.txt http://127.0.0.1:8000/api/status ``` The answer holds: - `backup.last_run` and `backup.in_progress` - `stats_calculated_at` - `listeners`: one entry per account, with `active`, `active_since`, `last_update_at`, `deletions_blocked` and `last_rate_limit_at`. See [Checking that it runs]() for what they mean and how to alert on them. - `media`: the media counts - `database.backend` and `database.size_bytes` It carries no chat ids, titles or text. An alert on `backup.last_run` older than two schedule ticks catches a backup that stopped working while the container stays healthy. Log in once and keep the cookie. Each login opens a new session, and the session and rate limits in [Sessions]() apply. ### From the command line `telegram-archive status` gives the same answer without the viewer and without a login. It reads the database directly, prints a short summary, and exits 1 when no backup has run, the last backup did not finish, or `SCHEDULE` has missed a run. Add `--json` for the full answer as JSON. Run it inside the backup container, which has the database settings and the scheduler's time zone: ``` docker compose exec telegram-backup python -m telegram_archive status ``` See [`status`]() for the output and what each exit code means. ## Logs Follow the backup: ``` docker compose logs -f telegram-backup ``` Follow the viewer: ``` docker compose logs -f telegram-viewer ``` The stock compose file caps each container's log at 10 MB, in 3 rotated files. ### Log level `LOG_LEVEL` sets the level for the backup, the scheduler and every CLI command except `migrate`. The default is `INFO`. | Value | Shows | | --- | --- | | `DEBUG` | Everything, including per-step detail | | `INFO` | Progress, run summaries and warnings | | `WARNING` | Warnings and errors. `WARN` is accepted too. | | `ERROR` | Errors only | | `CRITICAL` | Fatal errors only | An unknown value silently becomes `INFO`. The viewer ignores `LOG_LEVEL` and always logs at `INFO`. The Telethon and httpx loggers are always set to `WARNING`, so request URLs never reach the log. `DB_ECHO=true` logs every SQL statement. Only the word `true`, in any case, turns it on; `1`, `yes` and `on` do not. ### What the logs contain By default, the logs never contain chat ids, message text, account names or phone numbers. Set `LOG_CHAT_TITLES=true` to add a cleaned chat title to the two lines that report progress through each chat: ``` [3/40] Backing up [Archived 1/5] ``` The backup appends the title to the end of each line. For a private chat it writes `private chat` instead of the name. The same setting also names the channel in the listener's warning about a replay dropped as too long. It adds the cleaned title and the channel id, as the archive stores it: ``` Telegram dropped a channel's replay as too long: "Demo News" (id -1001234567890) (pts 40→400); ... ``` When Telegram's answer does not include the channel, the title reads `untitled chat` and the id is still there. `FLOOD_WAIT_LOG_THRESHOLD` hides short Telegram rate-limit pauses, called FloodWaits. The default is 10 seconds. While it fetches message history, the backup does not log a FloodWait shorter than this the first time it occurs. Set it to `0` to log every one. ### Specific log lines Other pages document the lines their features write: - The transcription drain summary: [Voice transcription](). - The webhook failure line: [Event webhook](). - The line the viewer logs when it rejects an update from the backup: [Live updates and notifications](). ## Other signals to watch ### Archive status The master login's **Archive status**, in the viewer's main menu, shows the state of the backup and its parts, under one line that says whether everything runs. Its sections are listed under [Archive status](). A growing `gave up` count means files failed `MEDIA_MAX_DOWNLOAD_ATTEMPTS` times, 5 by default. The backup also logs this warning at the end of each run: ``` N media file(s) permanently skipped after 5 failed download attempts (raise MEDIA_MAX_DOWNLOAD_ATTEMPTS to retry them) ``` ### Messages that failed When the backup cannot process a message, it keeps the chat's progress marker before that message and retries it on the next run. After two failed runs, it moves past the message. It records the id in the `metadata` table under the key `message_failures_`. For accounts other than the first, the key ends in `_account_`. The record keeps the newest 500 ids and an exact total. The ids never go to the logs. ### Statistics ``` docker compose exec telegram-backup python -m telegram_archive stats ``` This prints cached figures. They are all zero until the first backup has completed. ## Troubleshooting table | Symptom | Cause | Fix | | --- | --- | --- | | `Permission denied` writing to `/data` | Both containers run as uid 1000 and cannot write a directory another user owns. | `sudo chown -R 1000:1000 data`. See [Create the data directory](). | | The backup container restarts over and over after printing help | The image's default command has no subcommand, so it prints help and exits. | Pass a command, normally `schedule`, as the stock compose file does. | | `Session not authorized` | Telegram no longer accepts the session. See [Log in to Telegram](). | Stop the backup service, run `auth` again, start the service. | | `No module named telegram_archive` | Images up to and including 8.16.1 know the module only as `src`. See [the module rename](). | Run the same command with `python -m src`, or move both image pins to 8.17.0 or later. | | The viewer answers 503 `Viewer authentication is not configured` | No login method is set, so the viewer serves no data by design. See [Logins, viewer accounts and share links](). | Set `VIEWER_USERNAME` and `VIEWER_PASSWORD`, or `AUTH_PROXY_HEADER`, or `ALLOW_ANONYMOUS_VIEWER=true`, in the viewer's environment. | | The viewer shows no chats while the backup has data | The two containers resolve different databases, usually because `DATABASE_DIR` or `DATABASE_PATH` is set only in `.env`. See [SQLite and PostgreSQL](). | Use `DB_PATH`, or add the same variable to the viewer's `environment:` block. Both containers need identical database settings. | | The viewer shows errors right after an upgrade, or on an empty PostgreSQL database | The viewer never migrates, and the backup has not finished migrating yet. See [Upgrading](). | Start the backup container and wait. Check with `docker compose exec telegram-backup alembic current`. | | A variable set in `.env` has no effect on the viewer | The viewer service has no `env_file` and only receives an allowlist of variables. See [Run with Docker](). | Add the variable to the viewer's `environment:` block in `docker-compose.yml`. | | `ERROR: unrecognised database configuration` at start | `DB_TYPE` is not `sqlite`, `postgresql` or `postgres`, or `DATABASE_URL` uses an unsupported scheme such as `postgresql+psycopg2://`. See [SQLite and PostgreSQL](). | Fix the value. The container refuses to start until you do. | | `table chats already exists` when running `migrate` after a pip install | Another command created the schema before `migrate` ran, without recording a schema version. See [Install from PyPI](). | If the archive holds nothing yet, delete the database file and run `telegram-archive migrate` before any other command. | | `database is locked` on a slow disk | SQLite waits `DATABASE_TIMEOUT` seconds (60) for a lock, then gives up. See [SQLite and PostgreSQL](). | Raise `DATABASE_TIMEOUT` in both containers. It has no effect on PostgreSQL. | | Startup fails with `invalid literal for int()` | A chat id list, such as `CHAT_IDS` or any other `*_CHAT_IDS` except `EVENT_WEBHOOK_CHAT_IDS`, contains something that is not an integer. The error does not name the variable. A bad `EVENT_WEBHOOK_CHAT_IDS` only turns the webhook off with a warning. See [Choosing chats](). | Check every chat id list for stray text, quotes or spaces between commas. | | Startup fails with `Invalid boolean value for X` | Variable `X` holds a word the parser does not accept. | Use `true`, `false`, `1`, `0`, `yes`, `no`, `on` or `off`. | | The viewer crashes at start | `AUTH_SESSION_DAYS`, `MAX_WS_CONNECTIONS` or `MAX_WS_SUBSCRIPTIONS_PER_CONNECTION` is not an integer, or `DELETION_MODE` or a `MASS_OPERATION_*` value is invalid. See [Environment variables](). | Read the traceback in the viewer log and fix the value. `DELETION_MODE` must be `soft` or `hard`. `MASS_OPERATION_THRESHOLD` and `MASS_OPERATION_WINDOW_SECONDS` must be integers of at least 1. `MASS_OPERATION_BUFFER_DELAY` must be a number. | | Bot chats are missing | `bots` is not in the default `CHAT_TYPES`. See [Choosing chats](). | Add `bots` to `CHAT_TYPES`. | | Chats disappeared from new backups after adding an include list | Include lists are allow-lists: they restrict capture to the listed chats. See [Choosing chats](). | List every chat you want, or use per-type include lists together with `CHAT_TYPES`. | | Media is missing in the viewer | The viewer shows the reason under the file. See [Why media is missing in the viewer](). | Relax the limit or filter, or wait for the next backup run. | | A media file opens nowhere but the viewer, transcripts fail with `file_missing`, or the backup log warns that the run met media files whose link into media/\_shared points at nothing | The row points at a link into `media/_shared` whose file is gone, or at a chat folder that never held the file. The viewer finds some of these through its legacy folder fallback, other readers do not. See [A missing shared file](). | Run `telegram-archive check-media`, then `check-media --repair`. It restores files from copies on disk and marks the rest to download again at the next backup run. The transcription drain sends a file again once it is back, however many times it failed as missing. | | Media turned back on is not fetched for older messages | With `DOWNLOAD_MEDIA=false` or `SKIP_MEDIA_CHAT_IDS`, archived messages have no media record, and the cursor has moved past them. See [Media downloads](). | Only new messages get media. To keep the option of fetching later, narrow media with the type, MIME or size filters instead. | | Video thumbnails are missing on a native install | Video thumbnails need `ffmpeg`. Without it, thumbnails are skipped and no error appears. See [Install from PyPI](). | Install `ffmpeg` so it is on the `PATH`. | | Times are shown in the wrong zone | The viewer uses `VIEWER_TIMEZONE`, default `Europe/Madrid`. An unknown name falls back to UTC with a warning. See [Using the viewer](). | Set `VIEWER_TIMEZONE` to a tz database name, such as `America/New_York`. | | Backups run at an unexpected hour | `SCHEDULE` runs in the container's time zone, which is UTC unless you set `TZ`. `VIEWER_TIMEZONE` does not affect it. See [Schedule and backup tuning](). | Write the cron expression in UTC, or set `TZ` on the backup service, for example `TZ=Europe/Berlin`. | | The container is healthy but nothing is backed up | The heartbeat only proves the process is alive. See [Backup container](<#backup-container>). | Read `docker compose logs telegram-backup` and [Archive status](<#archive-status>). | | Long `FloodWait` pauses, and some chats are only finished on the next run | Telegram rate limits. A wait longer than `MAX_FLOOD_WAIT_SECONDS` (3600) is not waited out, and that chat or file is retried next run. See [Your first backup](). | Nothing to fix. Let the runs catch up. | | New messages only appear after a backup run | The [real-time listener]() is off, or not running. A `.env` or compose file copied before 9.2.0 sets `ENABLE_LISTENER=false`. `telegram-archive status` names an account whose listener is not running. | Set `ENABLE_LISTENER=true`, or remove the line. Read the backup log for the listener's errors. | | Live updates never arrive on SQLite | The backup cannot reach the viewer, or the two containers do not share the push secret. Outside compose, `VIEWER_PORT` defaults to 8080. See [Live updates and notifications](). | Outside compose, set `VIEWER_HOST` and `VIEWER_PORT=8000`. Mount the same database directory in both containers, or set `INTERNAL_PUSH_SECRET` on both. | | No notifications | Notifications need the [real-time listener](). The default `basic` mode only notifies for the open chat in a hidden tab. `full` needs the viewer served over HTTPS. See [Live updates and notifications](). | Turn on the listener, then set `PUSH_NOTIFICATIONS=full` behind HTTPS for notifications with the browser closed. | | The event webhook never fires | The webhook needs the listener, and each event needs its `LISTEN_*` switch. Startup logs a warning such as `EVENT_WEBHOOK_ENABLED has no effect: ENABLE_LISTENER=false`. See [Event webhook](). | Read the startup warnings and fix what they name. A bad `EVENT_WEBHOOK_*` value also turns the webhook off with one warning. | | Transcripts stay queued | Pressing Transcribe in the viewer only queues the file. The backup sends queued files on its next drain: every `TRANSCRIPTION_DRAIN_INTERVAL_MINUTES` under `schedule`, and at the end of each successful run. A drain stops when the server is unreachable or rejects a request. A 404 often means `TRANSCRIPTION_URL` ends in `/v1`. See [Voice transcription](). | Wait for the next drain, check the drain summary line, and remove `/v1` from the URL. | | A share token still works after its expiry | Expiry only blocks new logins with the token. Sessions it already created stay valid for `AUTH_SESSION_DAYS`. See [Logins, viewer accounts and share links](). | Revoke or delete the token. That ends every session it created. | | The login or transcript rate limit locks out everyone behind a reverse proxy | Without `TRUST_PROXY_HEADERS=true`, every client shares the proxy's address and one rate limit. See [Sessions]() and [Exposing the viewer safely](). | Set `TRUST_PROXY_HEADERS=true`, but only if the proxy overwrites `X-Forwarded-For`. | | The sidebar message search finds nothing and says it needs the full-text index | The backup has not started once on 8.5 or later, so migration 028 has not built the full-text index. See [SQLite and PostgreSQL](). | Start the backup container once and wait for it to migrate. | | Search stays without the full-text index after the backup has migrated | The SQLite build has no FTS5. Without the index, the sidebar search answers nothing, and the search inside a chat matches message text only, without transcripts. See [SQLite and PostgreSQL](). | Use a SQLite build with FTS5, or PostgreSQL. | | `missing from the archive disk` in an open tab after `reclassify-round-videos` | The tab holds old media addresses. See [Import and maintenance tasks](). | Reload the page. | | An excluded chat still shows in the viewer | Excluded chats keep their archived data by default. See [Choosing chats](). | Set `EXCLUDE_DELETE_EXISTING=true` to delete it on the next run. This cannot be undone. | | Telegram logged the account out after a one-off command | A second process used the same session file while `schedule` ran. See [Log in to Telegram](). | Log in again. From then on, stop the backup service before running commands that connect to Telegram. | | `duplicate key value violates unique constraint "reactions_pkey"` on PostgreSQL | The reactions id sequence is out of step with the table, often after a restore. See [Duplicate key errors on reactions](). | The backup resets the sequence and retries on its own. If the error persists, run the statement on that page. | ## Reporting a bug Open a bug report at [github.com/GeiserX/Telegram-Archive/issues](). Questions and feature requests have their own forms. Before you file a bug, have these ready: - The version tag of both images, as pinned in `docker-compose.yml` or shown by `docker compose images`. For a pip install, the output of `pip show telegram-archive`. - The log lines around the failure, from `docker compose logs telegram-backup` or `telegram-viewer`. - Whether you run SQLite or PostgreSQL. - Your operating system, Docker version or Python version. The logs carry no chat content, so you can paste them as they are. If you turned on `LOG_CHAT_TITLES`, remove the chat titles and ids first. An issue with no activity for 14 days is marked stale and closed 14 days later. Reply to it to keep it open. A security problem is not a bug report. Do not open an issue for it. See [Reporting a vulnerability]().