PocketKafka: a Kafka-compatible broker in one dependency-free Go binary

01Problem

• Standing up Kafka for a small industrial integration means operating a JVM broker plus a separate schema registry, a REST proxy, and an MQTT bridge for sensor telemetry.
• Every extra component adds memory, image size, and one more service to monitor, which hurts most on constrained or edge hosts.

02Constraints

• No third-party Go modules. Everything from protocol encoding to the storage engine is written against the standard library, so the dependency graph stays empty.
• Existing Kafka clients had to keep working, which rules out inventing a new protocol and forces conformance to the Kafka wire format.
• A single static binary is the deployment unit, so the web UI assets are embedded rather than served from a sidecar.

03Architecture and data flow

• The broker listens on 9092 for the Kafka wire protocol, with an internal listener on 29092 for container-to-container traffic.
• Three sidecars run inside the same process: an embedded web UI on 8080, a Confluent-compatible schema registry on 8081, and an HTTP REST proxy on 8082.
• An MQTT 3.1.1 bridge on 1883 accepts IoT telemetry and lands it on Kafka topics, and its traffic is exposed in the same web UI live tail as the Kafka messages.
• Storage, consumer-group coordination, and the schema registry are separate internal packages (internal/storage, internal/coordinator, internal/schemaregistry) rather than one shared blob.
• Prometheus metrics and Kubernetes-style health probes are served from the web listener.

04Engineering decisions

• A purpose-built log storage engine was written instead of embedding an existing library, because a zero-dependency build was the point of the project.
• Retention, log compaction, idempotence, and tiered storage each got a dedicated implementation and its own test file, so the guarantees are pinned by tests rather than asserted in prose.
• Tiered storage offloads cold segments to an S3 or MinIO endpoint and ships disabled by default, so the default deployment stays self-contained.
• Consumer-group session, rebalance, and heartbeat timeouts are configuration rather than constants, and offsets can be kept in memory or on disk.
• The admin CLI (cmd/kctl) supports JSON output so topic and consumer-group operations can be scripted in CI.

05Testing and verification

• Fifteen Go test files cover protocol encoding and compression, record batches, partition behaviour, log compaction, idempotence, tiered storage, SASL and SCRAM, the REST and MQTT gateways, and web auth.
• End-to-end coverage lives in internal/e2e as separate basic and feature suites.
• CI runs go build ./..., go vet ./... and go test ./... on every push and pull request, and recent runs on the master branch are green.

06What is proven today

• The build is genuinely dependency-free: go.mod declares only the module path and the Go version.
• The implemented surface is currently green under vet and the full test suite.
• What is not claimed: no throughput, latency, memory, or image-size measurement is published, so none appears here. The README also documents a TLS listener that has no counterpart in the sample configuration, and that claim is left out too.

Source and references