Skip to content

Latest commit

 

History

History
34 lines (21 loc) · 2.04 KB

README.rst

File metadata and controls

34 lines (21 loc) · 2.04 KB

Tadpoles.com Scraper

OLD BROKEN RUBBISH REPLACED

This repo has been broken for several years--my apologies to all denizens of the Interwebs who found this repo and were horribly confused.

The repo previously used dockerized Selinium browser automation, with a hideous login flow that didn't work for 90% of users.

Over the years, many unfortunate people have emailed me in a panic a day or two before their tadpoles accounts were about to be disabled. Probably tending to an exploding diaper of my own, I was never really able to help.

I have replaced this garbage with three janky little scripts that work, but require some annoying fiddling. Some code was cribbed from other Github repos by more able minds; attribution leaves something to be desired. Just search for Tadpoles on Github and you will find their stuff.

How to Use

  • Create an SQS Queue and S3 bucket to hold your images.
  • Grep this repo for 'TADPOLES_' and define the required environment variables. S3 bucket name, SQS queue url, etc.
  • Install requests and boto3.
  • Pour yourself a beverage
  • This step sucks: using mitmproxy or similar, log into Tadpoles and get the Cookie header from one of your http requests once the auth flow completes.
  • Set the value of the Cookie as an environment variable. Also set it to the secret named TADPOLES_COOKIE_AWS_SECRET_NAME environment variable (or pro tip: copy/paste the secret into your checkout of the script and be done with it).
  • Run the get_events.py script. It will output zillions of JSON objects to stdout. Pipe this into a file and proceed to the next step.
  • Run the extent_queue.py script. This just pumps all this JSON objects into the TADPOLES_SQS_QUEUE_URL queue in your AWS accounts (note: create that thing first).
  • Create a Lambda function and set the fetcher.py code as the body of the function.
  • Add the lambda function as a handler for your SQS queue.
  • Watch an episode of the Simpons and your photos should be sitting in S3 when the script is done.