Skip to content
This repository has been archived by the owner on Mar 1, 2021. It is now read-only.
/ smag-mvp Public archive

Social Record - Distributed scraping and analysis pipeline for a range of social media platforms

License

Notifications You must be signed in to change notification settings

codeuniversity/smag-mvp

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Social Record

Distributed scraping and analysis pipeline for a range of social media platforms

Shields

Contributors Commits

GitHub tag (latest SemVer) CircleCI

GitHub go.mod Go version

Table of content

About

The goal of this project is to raise awareness about data privacy. The mean to do so is a tool to scrape, combine and analyze public data from multiple social media sources.
The results will be available via an API, used for some kind of art exhibition.

Architectural overview

You can find an more detailed overview here.
Open it in draw.io and have a look at the different tabs "High level overview", "Distributed Scraper" and "Face Search".

Further reading

Detailed documentation

part docs contact
Api api/README.md @jo-fr
Frontend frontend/README.md @lukas-menzel
Postgres DB db/README.md @alexmorten

Wanna contribute?

If you want to join us raising awareness for data privacy have a look into CONTRIBUTING.md

List of contributors

Github handle Real name Instagram profile Twitter profile
@1Jo1 Josef Grieb josef_grieb josefgrieb
@Urhengulas Johann Hemmann Urhengulas Johann
@alexmorten Alexander Martin no profile :( no profile :(
@jo-fr Jonathan Freiberger jonifreiberger Jonathan
@m-lukas Lukas Müller lmglukas Lukas Müller
@lukas-menzel Lukas Menzel lukasmenzel Lukas Menzel
@SpringHawk Martin Zaubitzer / /

Deployment

The deployment of this project to kubernetes happens in codeuniversity/smag-deploy (this is a private repo!)

Getting started

Requirements

depency version
go v1.13 (go modules)
docker v19.x
docker-compose v1.24.x

Preparation

If this is your first time running this:

  1. Add 127.0.0.1 my-kafka and 127.0.0.1 minio to your /etc/hosts file
  2. Choose a <user_name> for your platform of choice <instagram|twitter> as a starting point and run
    $ go run cli/main/main.go <instagram|twitter> <user_name>

Scraper

Run the instagram- or twitter-scraper in docker:

$ make run-<platform_name>