Scrapes Tagalog, Cebuano, Hiligaynon, and Ilocano words from Pinoy Dictionary
The scraped words are then printed into an output file in a UTF-8-encoded JSON format.
If all you need is to get a word list (with or without definitions) for your app, you may just go to the Scraped Data folder.
Note: The Ilocano dictionary is a list of English words with Ilocano definitions.
We need to have Python 3, BeautifulSoup and Requests installed on your machine. If you already have these, then you should be able to run the scraper files without any additional setup.
Please ensure that you are running Python 3 or newer and NOT Python 2. The scraper uses functionality in argparse that is only available on Python 3.
To install BeautifulSoup and Requests, we need Python's pip package.
According to pip's documentation, you can install pip by running:
sudo python -m ensurepip --upgrade
According to BeautifulSoup's implementation, you can install BeautifulSoup by running:
pip install beautifulsoup4
If this doesn't work you may have to try
pip install beautifulsoup4 --user
or
sudo pip install beautifulsoup4
According to Requests' implementation, you can install Requests by running:
python -m pip install requests
Run the following command:
python scraper.py [--dictionaries <dictionaries>] [--output_file <file_name>]
The dictionaries value that you should pass in should be a comma-separated string containing any of the following strings:
"tagalog", "cebuano", "hiligaynon", "ilocano"
For example, if you wanted to scrape the data for Tagalog, Cebuano, and Hiligaynon, you would use --dictionaries tagalog,cebuano,hiligaynon.
If this argument is not specified, words from all dictionaries will be scraped.
The file_name value that you should pass in should be the file that you would like to print the words in.
For example, if you wanted to print the Tagalog words into a file named tagalog_dictionary.json, you would run the command python scraper.py --dictionaries tagalog --output_file tagalog_dictionary.json.
It is advised to specify this argument; if it is not specified, the value will be printed out on the console (or wherever the console output may be redirected to).