About

Implementation of a map reduce system inspired by MapReduce: Simplified Data Processing on Large Clusters.

Dependencies

Protocol buffer compiler and protoc-gen-go

Running

# Run the master
./dev/run.sh --http.port=9000 --grpc.port=8001 --workspace-folder="./tmp"

# Run N workers
./dev/run.sh --grpc.port=8010 --master.addr=":8001" --worker.memory=100Mi --workspace-folder="./tmp/worker_1"

curl -X POST localhost:8002/task \
-H 'Content-Type: application/json' \
--data-binary @- << EOF
{
  "file": "./dev/input_word_count.txt",
  "numberOfPartitions": 3,
  "numberOfMapTasks": 3,
  "numberOfReduceTasks": 1,
  "requests": {
     "memory": "64Mi"
  },
  "scriptBase64": "$(cat word_count.js | base64 -w 0)"
}
EOF