Skip to content

Additional Code Samples for Hazelcast Jet. For the development version samples use master branch

License

Notifications You must be signed in to change notification settings

jbartok/hazelcast-jet-code-samples

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hazelcast Jet Code Samples

A repository of code samples for Hazelcast Jet. The samples show you how to use the Pipeline API to solve a range of use cases, how to integrate Jet with other systems and how to connect to various data sources (both from a Hazelcast IMDG and 3rd-party systems). There is also a folder with samples using the Core API.

Stream Aggregation

  • apply a sliding window
  • perform basic aggregation (counting)
  • print the results on the console
  • like the above, plus:
  • add a second-level aggregation stage to find top/bottom N results
  • apply a session window
  • use a custom Core API processor as the event source
  • perform a composite aggregate operation (apply two aggregate functions in parallel).
  • print the results on the console
  • use the SourceBuilder to create a mock source of trade events from a stock market
  • apply a tumbling window, configure to emit early results
  • aggregate by summing a derived value
  • present the results in a live GUI chart
  • use SourceBuilder to create a mock source of trade events from a stock market
  • simple rolling aggregation (summing the price)
  • keep updating the target map with the current values of aggregation
  • present the results in a live GUI chart

Batch Aggregation

  • use an IMap as the data source
  • stateless transforms to clean up the input (flatMap + filter)
  • perform basic aggregation (counting)
  • print a table of the most frequent words on the console
  • serialize a small dataset to use as side input
  • fork a pipeline stage into two downstream stages
  • stateless transformations to clean up input
  • count distinct items
  • group by key, then group by secondary key
  • aggregate to a map of (secondary key -> result)
  • hash-join the forked stages
  • open an interactive GUI to try out the results

Joins

  • co-group three bounded data streams on a common key
  • for each distinct key, emit the co-grouped items in a 3-tuple of lists
  • store the results in an IMap and check they are as expected
  • use the Event Journal of an IMap as a streaming source
  • apply a sliding window
  • co-group three unbounded data streams on a common key
  • print the results on the console

Hash Join

Data Enrichment

  • the sample is in the enrichUsingIMap() method
  • use the Event Journal of an IMap as a streaming data source
  • apply the mapUsingIMap transform to fetch the enriching data from another IMap
  • enrich from two IMaps in two mapUsingIMap steps
  • print the results on the console
  • the sample is in the enrichUsingReplicatedMap() method
  • use the Event Journal of an IMap as a streaming data source
  • apply the mapUsingReplicatedMap transform to fetch the enriching data from another IMap
  • enrich from two ReplicatedMaps in two mapUsingReplicatedMap steps
  • print the results on the console
  • the sample is in the enrichUsingAsyncService() method
  • prepare a mock data server: a gRPC-based network service
  • use the Event Journal of an IMap as a streaming data source
  • enrich the unbounded data stream by making async gRPC calls to the mock service
  • print the results on the console
  • the sample is in the enrichUsingHashJoin() method
  • use the Event Journal of an IMap as a streaming data source
  • use a directory of files as a batch data source
  • hash-join an unbounded stream with two batch streams in one step
  • print the results on the console

Configuration

Job Management

Integration with Hazelcast IMDG

Integration with Other Systems

Custom Sources and Sinks

  • Custom Source:
    • start an Undertow HTTP server that collects basic JVM stats
    • construct a custom Jet source based on Java 11 HTTP client
    • apply a sliding window
    • compute linear trend of the JVM metric provided by the HTTP server
    • present the results in a live GUI chart
  • Custom Sink
    • construct a custom Hazelcast ITopic sink
  • Custom Side Input for Data Enrichment
    • look at the enrichUsingReplicatedMap() method

Integration with Frameworks

Spring Framework

Use a pre-trained TensorFlow model to enrich a stream of movie reviews. The model classifies natural-language text by the sentiment it expresses.

Integration with Deployment Environments

Code samples that show how to use the low-level Core DAG API directly. See the dedicated README for more details.

License

Hazelcast is available under the Apache 2 License. Please see the Licensing section for more information.

Copyright

Copyright (c) 2008-2019, Hazelcast, Inc. All Rights Reserved.

Visit www.hazelcast.com for more info.

About

Additional Code Samples for Hazelcast Jet. For the development version samples use master branch

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Java 99.3%
  • Other 0.7%