Beautify the Bot-UA dataset

[0x48piraj]

The current implementation includes a list of patterns in no specific order and hence makes it harder for contributors to collaborate & submit new entries). The dataset can be improved by a bit of beautification (for now).

[
      /bot/,/spider/,/crawl/,                            // GENERAL TERMS
      /APIs-Google/,/AdsBot/,/Googlebot/,                // GOOGLE ROBOTS
      /mediapartners/,/Google Favicon/,
      /FeedFetcher/,/Google-Read-Aloud/,
      /DuplexWeb-Google/,/googleweblight/,
      /bing/,/yandex/,/baidu/,/duckduck/,/yahoo/,        // OTHER ENGINES
      /ecosia/,/ia_archiver/,
      /facebook/,/instagram/,/pinterest/,/reddit/,       // SOCIAL MEDIA
      /slack/,/twitter/,/whatsapp/,/youtube/,
      /semrush/,                                         // OTHER
    ]

The current list is jumbled and catches false positives as well.

['/index/', '/wbsearchbot/', '/proximic/', '/panscient/', '/Googlebot/', '/psbot/', '/whatsapp/', '/findxbot/', '/archive/', '/Voyager/', '/pinterest/', '/ccbot/', '/bibnum/', '/Linguee Bot/', '/libwww/', '/spider/', '/wget/', '/archive.org_bot/', '/Mediapartners-Google/', '/Twitterbot/', '/APIs-Google/', '/Livelapbot/', '/msn/', '/Adidxbot/', '/europarchive.org/', '/ip-web-crawler.com/', '/Facebot/', '/CrystalSemanticsBot/', '/bingbot/', '/fluffy/', '/Python-urllib/', '/360Spider/', '/lipperhey/', '/binlar/', '/buzzbot/', '/postrank/', '/urlappendbot/', '/ngbot/', '/Lipperhey SEO Service/', '/curl/', '/grub.org/', '/Googlebot-Mobile/', '/lssrocketcrawler/', '/scaper/', '/acoonbot/', '/siteexplorer.info/', '/yeti/', '/purebot/', '/indexer/', '/Domain Re-Animator Bot/', '/httrack/', '/heritrix/', '/AdsBot/', '/blexbot/', '/yacybot/', '/aihitbot/', '/FAST-WebCrawler/', '/msnbot/', '/sistrix crawler/', '/xovibot/', '/ia_archiver/', '/discobot/', '/woriobot/', '/Google favicon/', '/cXensebot/', '/DuplexWeb-Google/', '/baidu/', '/IOI/', '/findthatfile/', '/sogou/', '/AdvBot/', '/GingerCrawler/', '/drupact/', '/spbot/', '/wocbot/', '/facebook/', '/gnam gnam spider/', '/googleweblight/', '/instagram/', '/y!j-asr/', '/lb-spider/', '/careerbot/', '/antibot/', '/smtbot/', '/Aboundex/', '/bibnum.bnf/', '/teoma/', '/changedetection/', '/gslfbot/', '/jyxobot/', '/biglotron/', '/blekkobot/', '/tagoobot/', '/niki-bot/', '/toplistbot/', '/google/', '/yoozBot/', '/Commons-HttpClient/', '/citeseerxbot/', '/UsineNouvelleCrawler/', '/turnitinbot/', '/rogerbot/', '/yandexbot/', '/wotbox/', '/bot/', '/grub/', '/duckduck/', '/voilabot/', '/mediapartners/', '/TweetmemeBot/', '/ichiro/', '/CC Metadata Scaper/', '/webcrawler/', '/brainobot/', '/WeSEE:Search/', '/ips-agent/', '/g00g1e/', '/it2media-domain-crawler/', '/OrangeBot/', '/nutch/', '/netresearchserver/', '/BUbiNG/', '/http/', '/webmon /', '/intelium_bot/', '/web-archive-net.com.bot/', '/ecosia/', '/nerdybot/', '/Applebot/', '/sitebot/', '/DuckDuckBot/', '/integromedb/', '/FAST Enterprise Crawler/', '/backlinkcrawler/', '/openindexspider/', '/java/', '/reddit/', '/seekbot/', '/Googlebot-Image/', '/SemanticScholarBot/', '/linkdex/', '/webcompanycrawler/', '/scribdbot/', '/CyberPatrol/', '/lssbot/', '/siteexplorer/', '/httpunit/', '/semrush/', '/ltx71/', '/exabot/', '/slurp/', '/ezooms/', '/g00g1e.net/', '/AISearchBot/', '/ADmantX/', '/arabot/', '/Qwantify/', '/scrape/', '/mlbot/', '/fr-crawler/', '/headless/', '/Google-Read-Aloud/', '/speedy/', '/MegaIndex/', '/yanga/', '/SimpleCrawler/', '/baiduspider/', '/phpcrawl/', '/dotbot/', '/RetrevoPageAnalyzer/', '/twitter/', '/ec2linkfinder/', '/youtube/', '/bing/', '/msrbot/', '/FeedFetcher/', '/yandex/', '/domaincrawler/', '/coccoc/', '/summify/', '/GrapeshotCrawler/', '/gigablast/', '/webmon/', '/content crawler spider/', '/yahoo/', '/twengabot/', '/WeSEE/', '/memorybot/', '/SemrushBot/', '/MJ12bot/', '/seznambot/', '/googlebot/', '/bnf.fr_bot/', '/Google Favicon/', '/Mail.RU_Bot/', '/page2rss/', '/facebookexternalhit/', '/AddThis/', '/NerdByNature.Bot/', '/seokicks-robot/', '/crawl/', '/convera/', '/slack/', '/elisabot/', '/A6-Indexer/', '/crawler4j/', '/edisterbot/', '/ahrefsbot/', '/findlink/', '/InterfaxScanBot/']