spark

data-engineering-zoomcamp

+162

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

apache/spark

+75

Apache Spark - A unified analytics engine for large-scale data processing

Spark-X2.6

+29

Spark-x2.6 open model series. Pushing the Limits of Agentic Capabilities in On-Device Models

sqlglot

+23

Python SQL Parser and Transpiler

lucebox

+17

LLM speculative inference server for heterogeneous hardware & consumer GPUs

splink

+16

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

delta-io/delta

+11

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

datafusion-comet

+9

Apache DataFusion Comet Spark Accelerator

apache/paimon

+8

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.