Getting Started | Deployment | Documentation and Support | Blog | License

Pathway Live Data Framework is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

Pathway Live Data Framework comes with an easy-to-use Python API, allowing you to seamlessly integrate your favorite Python ML libraries. Pathway Live Data Framework code is versatile and robust: you can use it in both development and production environments, handling both batch and streaming data effectively. The same code can be used for local development, CI/CD tests, running batch jobs, handling stream replays, and processing data streams.

Pathway Live Data Framework is powered by a scalable Rust engine based on Differential Dataflow and performs incremental computation. Your Pathway Live Data Framework code, despite being written in Python, is run by the Rust engine, enabling multithreading, multiprocessing, and distributed computations. All the pipeline is kept in memory and can be easily deployed with Docker and Kubernetes.

You can install Pathway Live Data Framework with pip:

pip install -U pathway

For any questions, you will find the community and team behind the project on Discord.

Ready to see what Pathway Live Data Framework can do?

Try one of our easy-to-run examples!

Available in both notebook and docker formats, these ready-to-launch examples can be launched in just a few clicks. Pick one and start your hands-on experience with Pathway Live Data Framework today!

With its unified engine for batch and streaming and its full Python compatibility, Pathway Live Data Framework makes data processing as easy as possible. It's the ideal solution for a wide range of data processing pipelines, including: