Playbooks
Step-by-step responses to the incidents seen in production, from symptom to fix.
Each playbook goes symptom → checks → likely causes → fix. Commands assume the dc alias from Debugging.
Flow diagram · placeholder
Entry point: "what does the user see?" branching to each playbook.
Checks
- Database row:
processing_status,processing_error,manifest_url(queries). - The lock: which video holds it, and its TTL.
- The worker log for that video ID.
Likely causes
| What you see | Cause | Fix |
|---|---|---|
queued, lock empty | Nothing restarted the queue | Run start_next_video_if_idle() |
queued, lock held by another video | Waiting its turn | Wait; check the other video is progressing |
| An in-progress status, no recent log lines | Worker interrupted | Restart the worker (recovery marks it failed), then re-queue |
failed with an error | The pipeline failed | Read the error, fix the cause, re-queue |
| An error in the log between stages | A Celery-level failure | See the 2026-10 incident |
A visible thumbnail proves nothing: it is stored at upload time, not by the pipeline.
- Read
processing_errorand grep the worker log for the video ID. ffprobeerrors mean the file is unreadable or not a video: ask for a different file.- FFmpeg or MinIO errors may be transient: re-queue once.
Pipeline stopped during '…'names the stage and task that raised.- "Interrupted … by a worker restart": re-queue.
| Symptom | Check | Fix |
|---|---|---|
503 with Retry-After | Count tus:active:* keys | The cap (8) is full; slots expire after 2 hours, or finish/cancel the active uploads |
| "Invalid or expired token" | The admin's session | Sign in again, then upload |
| Upload completes but "processing didn't start" | dc logs api around that time for the post-finish hook | Check the tus_uploads row; if a video row exists, re-queue it |
| Fails immediately | dc logs tusd, dc logs api | tusd or API down, or the hook secret differs between them |
dc ps: is every serviceUp? Start anything that is not:dc up -d <service>.curl -s https://api.vod.spokhrel.dev/health/from outside, andcurl -s http://127.0.0.1:8000/health/on the server. If the second works but the first does not, the problem is Caddy or DNS.dc logs caddy: certificate or config errors.dc logs api: a startup error naming a missing setting meansprod.envis incomplete.free -handdmesg -T | grep -i oom: was a container killed for memory?
curl -sIthe master playlist under/storage/...(Debugging).- 404 with an S3
NoSuchKeyerror: the object is missing; checkmanifest_urlmatches what is in the bucket, or re-process. - 403 / Access Denied: the bucket lost public read; re-apply it (Server setup).
- Works with curl but not in the browser: check the frontend's
NEXT_PUBLIC_API_URLin Vercel.
df -h /, thendocker system df.- Worker leftovers:
dc exec worker du -sh /app/tmp/video_processing. Directories for videos that are not processing can be removed; recreating the worker clears them all. - Container logs: check
/var/lib/docker/containers/*/*-json.log; configure rotation in/etc/docker/daemon.jsonif they are large. - Old images:
docker image prune. - MinIO data: originals of soft-deleted videos are still stored.
A known frontend session-restore bug. Clear the site's local storage (or sign out, if possible) and reload. Tracked in Improvements.
Expected if it was processing during the deploy: the worker's startup recovery fails interrupted videos. Re-queue it. Next time, check the lock before deploying (Deploy).