Back to projects

Streaming service

X Tube

Event-driven pipeline for media processing with Go, SQS, S3, FFmpeg, and Kafka.

View processing service

Technical vision

Event-driven media processing

The flow separates ingestion, processing, and delivery using S3, SQS, Go workers, FFmpeg, and Kafka.

ImpactS3 → SQS → Go → S3 → Kafka

asynchronous processing and progress flow

Ingestion

Objects in S3 trigger SQS messages to start asynchronous processing.

Processing

Go workers run FFmpeg and produce chunks and artifacts.

Progress

Kafka events are published after each processed chunk upload is confirmed.

Execution flow

Main flow

Event-driven media processing

The flow separates ingestion, processing, and delivery using S3, SQS, Go workers, FFmpeg, and Kafka.

UPLOAD -> CLAIM JOB -> FFMPEG -> ARTIFACTS -> EVENT PUBLISHED

  1. 01

    Media enters S3

    Processing starts through SQS instead of being tied to a synchronous request.

  2. 02

    Worker processes with FFmpeg

    The job generates chunks and artifacts while keeping the message active during long processing.

  3. 03

    Upload confirms progress

    After upload to S3, Kafka events communicate processing progress.

Engineering decisions

Architecture

Explicit async pipeline

S3, SQS, Go, FFmpeg, S3, and Kafka define clear stages between ingestion and delivery.

Delete only after success

The message is removed from the queue only after processing completes successfully.

Progress confirmed by upload

Kafka events are emitted after the upload of each processed chunk is confirmed.

Risk controls

Reliability

Long job loss

Visibility timeout renewal prevents in-progress jobs from being lost during long processing.

Observable operation

Prometheus, health checks, and structured logs support operational visibility.

Stack

Processing

  • Go
  • FFmpeg

Messaging

  • SQS
  • Kafka

Storage and observability

  • S3
  • Prometheus
  • health checks
  • structured logs

Outcome

Decoupled, observable media delivery

The project demonstrates event-driven processing with progress confirmation and controls for long-running jobs.

asynchronous processing and progress flow
S3 → SQS → Go → S3 → Kafka

Next project

GGCompress

Go engine for processing large files in parallel while keeping memory controlled and integrity verifiable.

Next project