kafgres

A PostgreSQL extension that embeds a Kafka broker. Unmodified Kafka clients (librdkafka, the Java client, Sarama, kafka-python, kcat, the kafka-*.sh tooling) connect to Postgres on port 9092 and cannot tell the difference. Topics, partitions, offsets, consumer groups, and the log itself live in the database.

The goal is transactional coupling between the database and the event stream:

BEGIN;
  INSERT INTO orders (id, customer_id, total) VALUES (...);
  SELECT kafgres_produce('order-events', 'OrderCreated',
                         jsonb_build_object(...)::text);
COMMIT;

Both commit or neither does. There is no outbox table, no Debezium, no Connect cluster, and no dual-write inconsistency window. See docs/producing.md for the produce paths and when to use each.

Status

The following all work today:

Unmodified clients produce, consume, and rebalance: librdkafka and kcat, the Java client and its console tools, Sarama, and kafka-python.
A default-config Java KafkaProducer with idempotence on (the default since Kafka 3.0) works with no overrides.
Admin APIs: kafka-topics.sh, kafka-configs.sh, kafka-consumer-groups.sh, kafka-acls.sh, transactions, share groups, and the KIP-848 consumer protocol.
Retention reclaims disk, and cleanup.policy=compact topics are compacted on both storage engines.
Clients authenticate with SCRAM-SHA-256 against Postgres roles or an mTLS certificate, with ACLs in a SQL-managed table.
Consumers truncate rather than read divergent data when an async replica is promoted, verified against a real physical standby.
The conformance suite drives four real clients against kafgres and a reference Kafka and diffs the observable results; it runs in CI on every pull request, and every intended difference is catalogued in docs/conformance.md.
The segment engine (the default) passes the same conformance suite, survives kill -9 with every acknowledged record intact, and replicates its log to a standby out of band. By default every acknowledged record also survives a host crash. On an i9-13900 with power-loss-protected NVMe it produced about 2.4x the table engine's throughput with both durability settings relaxed. The strict defaults cost 3 to 25%: 806 MB/s relaxed against 605 MB/s strict for 1 KiB records from 4 producers (the full table is in