data-pipelines

DataFlow

+98

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

apache/airflow

+51

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

unstructured

+45

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

dagster

+36

An orchestration platform for the development, production, and observation of data assets.

feldera

+12

The Feldera Incremental Computation Engine

dolphinscheduler

+11

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

data-engineering-wiki

+13

The best place to learn data engineering. Built and maintained by the data engineering community.

bruin

+6

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

kedro

+6

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.