Likely questions
Questions an interviewer is likely to ask about VOD, with answers grounded in the code.
Architecture
A Python process is a poor pipe for gigabytes: it ties up workers for the length of the upload and cannot resume. tusd streams chunks straight to MinIO; the API only answers small hook calls. (ADR 0001)
post-finish creates a queued row → start_next_video_if_idle() takes a Redis lock → a Celery chain probes, builds the storyboard, transcodes renditions in a chord, segments, writes the master playlist, uploads to MinIO and finalizes → the lock passes to the next video. (Upload to playback)
HLS splits each rendition into short segments listed in playlists. The player measures throughput while downloading segments and picks the rendition for the next segment, so quality drops before the buffer runs out instead of stalling. Segments are plain HTTP files, so they cache well.
On one small server, parallel FFmpeg pipelines compete for CPU and memory. Even with one worker process, uncoordinated chains interleave so every video finishes late. Serialising pipelines keeps resource use predictable and finishes videos in order. (ADR 0002)
A group of tasks plus a callback that runs after all of them finish, with their results. The seven transcodes form the group; on_transcode_complete aggregates which renditions succeeded before segmenting.
Reliability
No callback runs, so on startup a worker_init handler purges leftover messages, clears the lock, marks in-progress videos failed and starts the next queued one. A failed video is retried by setting it back to queued.
The lock is taken with SET NX (atomic), and prepare_video refuses any video not exactly queued. The lock is released with an owner-checked Lua script, so a late callback cannot release another video's slot.
The stuck videos incident: a link= callback on the chain landed on the chord and crashed every pipeline with a TypeError inside Celery. Found from the database state and worker logs, reproduced locally, mechanism confirmed by inspecting Celery's built messages, fixed by making completion a chain step and making failures visible.
Today: the failure handler records the error, and [queue] log lines show every start, finish and failure. Next: alert on any video in a processing state longer than a threshold, and an end-to-end pipeline test in CI.
Scale and performance
Playback bandwidth: every segment is served from one server. A CDN in front of /storage is the first change, and it is easy because segments are already immutable and cache-friendly. (Walkthrough)
Decode once and encode all renditions in one FFmpeg process writing HLS directly; force aligned keyframes; encode audio once; upload segments in parallel; then hardware encoders or chunked parallel encoding across machines. (Improvements)
Replace single-worker recovery with per-video ownership (a heartbeat the worker renews), move intermediate files to shared or object storage so stages can run on different machines, and allow N pipelines at once with a counting semaphore instead of a single lock.
Security
Each admin route depends on get_current_admin_user, which verifies the access token and loads the user from the database; the role comes from the row, not the token, so demotion is immediate. Frontend guards are only UX.
They are called by tusd over the private Docker network with a shared secret compared in constant time, and Caddy returns 404 for /internal/* from the internet. The only public internal route is the status bridge, which needs an admin token.
Short-lived access tokens need only a signature check per request; refresh tokens are stored as SHA-256 hashes so sessions can be revoked (logout, password reset) and a database leak does not expose usable tokens.