The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
Updated
Jun 29, 2024 - Python
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
task management & automation tool
Example end to end data engineering project.
Smarter data pipelines for audio.
Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.
Code review for data in dbt
Streaming reactive and dataflow graphs in Python
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes
Fluent data pipelines for python and your shell
A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.
Code for "Efficient Data Processing in Spark" Course
Tools for ASR Corpus Generation from Online Video
Watchmen Platform is a low code data platform for data pipeline, meta data management , analysis, and quality management
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Compose multimodal datasets 🎹
Build and deploy a serverless data pipeline on AWS with no effort.
Data pipelines from re-usable components
Add a description, image, and links to the data-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the data-pipeline topic, visit your repo's landing page and select "manage topics."