Meliscraper

Meliscraper is a web scraper for Mercado Livre best sellers. This isn't a very complex project, but it is my first most complete project. I tried to implement simple things that I never used before, such as linters, set up CORS and a better way to deal with logs. I also decide to use two databases: PostgreSQL for relational data and MongoDB for application logs.

Table of Contents

• Technologies
• Usage
  • Requirements
  • Installing
  • Configuring
    • .env
    • Migrations
  • Starting
  • Routes
    

Technologies

Usage

Requirements:

• NodeJS v.18.x.x or higher

Installing:

Yarn:

$ yarn

NPM:

$ npm i

Configuring

Since the application uses two different databases, you'll need to create a PostgreSQL database and a MongoDB database as well.

.env

Rename the .env.example file to .env and update the variables with your settings.

key	description	default
PORT	Port number where the app will run.	3000
DATABASE_URL	Your PostgreSQL database connection URL.	postgresql://<username>:<password>@localhost:5432/<database>
LOGS_DATABASE_URL	Your MongoDB database connection URL.	mongodb://localhost:27017/<database>

Migrations

Run the following script to create database migrations:

$ npm run prisma:migrate

Starting

$ npm start

Project will start at http://localhost:3000/

Routes

route	HTTP method	params	description
/documentation	GET	-	Swagger documentation.
/scrapings	POST	-	Performs a new scraping.
/scrapings/{date}	GET	:date	Searches scraping by date, e.g. YYYY-MM-DD
/categories	GET	-	List all categories.
/categories/{slug}/products	GET	:slug	A list of all products related a single category.

⬆ Back to the top