Media Converter Service
A Kubernetes-based HLS media conversion pipeline driven by AWS SQS. A file lands on S3, a queue message fires, a Kubernetes job spins up, and the result — multi-resolution video streams, multi-bitrate audio tiers, thumbnails, and master playlists — is back on S3 with no manual steps. The service handles multiple audio tracks per video, auto-detects available GPU codecs, and scales worker nodes to zero when the queue is empty.
Challenges
One input, hundreds of output files
A single video produces five video resolutions, five audio bitrate tiers per audio track, thumbnails, previews, and a master playlist. Organizing, generating, and uploading all of that cleanly — and cleaning up stale files from previous runs — required careful pipeline design.
Multiple audio streams per video
Some videos contain several audio tracks in different languages. The system had to detect all tracks automatically using FFprobe, convert each one separately, and build a master m3u8 playlist that correctly references every track with language tags and quality levels.
Queue reliability and duplicate prevention
SQS delivers messages at least once, not exactly once. Before creating a Kubernetes job, the manager checks for an already-running job for the same file. If one exists, it pushes the message back with a delay instead of spawning a duplicate.
Codec detection across heterogeneous hardware
Not every node has GPU acceleration. The system detects available codecs at runtime — NVIDIA NVENC, Intel Quick Sync, AMD, or software x264 — and picks the best available option automatically, with a clean fallback chain.
Our approach
Queue polling and deduplication
The job manager polls SQS with long polling. When a message arrives, it checks for an already-running Kubernetes job for the same S3 object. Duplicates are pushed back to the queue with a delay instead of spawning a second conversion.
Spin up a dedicated Kubernetes job
A new pod is created from a template with defined CPU and memory limits, node affinity for ARM instances, and ephemeral volumes for temp files, input, output, and logs. The pod downloads the source file from S3 to a local volume.
Convert video and audio
FFmpeg converts video to five HLS resolutions with 4-second segments using the best available codec. Each audio track is converted to five bitrate tiers independently. Audio-only files get a multi-bitrate HLS set with AAC encoding and loudness normalization.
Build master playlist and extra outputs
A single m3u8 master file references all video resolutions and all audio tracks with language tags. The system also extracts a thumbnail, generates a low-res preview, and keeps MP3 and WAV copies of audio tracks.
Upload, clean up, and scale down
All outputs are uploaded to S3 in parallel. Stale files from previous runs are cleaned up. The autoscaler monitors queue depth and scales worker nodes back to zero when jobs are done — no idle compute cost between batches.
Result
Fully automated pipeline from S3 upload to production-ready HLS. Zero idle compute cost between batches. Supports 5 video resolutions and 5 audio bitrate tiers per file.
By the numbers
5 (240p–1080p)
Video resolutions per output
5 per track
Audio bitrate tiers
zero (scales to 0)
Idle compute cost
0
Manual steps in pipeline
Tech stack
We want to hear your thoughts.
our CTO Kyrylo Osadchuk, will reply within 24 hours. No SDR funnel.
Start a conversation
We want to hear your thoughts
Re: Media Converter Service