Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

Apache Spark 3 - Structured Streaming Course Material

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

A data engineering project (Twitter monitor app)

For a series of posts on Amazon MSK, Amazon EKS, and Amazon EMR

Final Project for IoT: Big Data Processing and Analytics class. Analyzing U.S nationwide temperature from IoT sensors in real-time

Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO

Counting Tweets Per User in Real-Time

Benchmarks for data processing systems: Pathway, Spark, Flink, Kafka Streams

Toolbox for building Generative AI applications on top of Apache Spark.

Apache Spark Guide

Repository of notebooks and related collateral used in the Databricks Demo Hub, showing how to use Databricks, Delta Lake, MLflow, and more.

Udacity Data Streaming Nanodegree Program

Video processing (webcam) in real time using Kafka and Spark.

Slides, Code, Images and Meme related to the course Technologies for Advanced Programming

A real time analytics dashboard to analyze the trending hashtags and @ mentions at any location using kafka and spark streaming.

Solution for IBM Data Engineer Professional Certificate

A structured streaming was applied to the robot data from ROS-Gazebo simulation environment using Apache Spark. Data is collected in Kafka, analyzed by Apache Spark and stored in Cassandra.

This is a data processing pipeline that implements an End-to-End Real-Time Geospatial Analytics and Visualization multi-component full-stack solution, using Apache Spark Structured Streaming, Apache Kafka, MongoDB Change Streams, Node.js, React, Uber's Deck.gl and React-Vis, and using the Massachusetts Bay Transportation Authority's (MBTA) APIs …

Spark stream consumption commit in kafka consumer group