Add documentation for GET_HOST_CATEGORIES function #6
Conversation
| # `GET_HOST_CATEGORIES` function | ||
|
|
||
| ## Description | ||
| We are happy to release the largest open-source dataset of website categories, featuring 147 million hosts and 31 million domains, making it the most extensive open-source data available in this area. |
There was a problem hiding this comment.
WDYT about keeping the language more neutral for the technical reference docs? For a blog post style announcement we can also create a new page under the guides dir.
There was a problem hiding this comment.
I was also not sure about that, feel free to remove this sentence
| ### Raw Data | ||
| The raw data for the classifications is stored in the `httparchive.urls.categories` table. This table consists of pre-classified hostnames with their corresponding categories. The categories follow a hierarchical structure, providing both specific subcategories and broader parent categories. | ||
|
|
||
| Please consider the limitations of our method regarding some hosts discussed [here](https://github.com/HTTPArchive/httparchive.org/issues/868). Thus, while this data can be accessed directly, we highly recommend using the `GET_HOST_CATEGORIES` function due to the handling of hashed subdomains. If your analysis requires working with domains (e.g., `google.com` instead of `maps.google.com`), accessing the raw data directly is also appropriate. |
There was a problem hiding this comment.
We also list some of these hosts in the Limitations section below, so do we need to link to the GH issue?
There was a problem hiding this comment.
I think linking to the issue makes sense for anyone interested in discovering the limitations more
|
|
||
|
|
||
| ## Methodology | ||
| The classification of hostnames is performed using Chrome's Topics API (currently `chrome5`) by [Nurullah Demir](https://ndemir.com) and [Yohan Beugin](https://yohan.beugin.org/) using [this repository](https://github.com/yohhaan/httparchive-topics-classification). The taxonomy used is version 2, which consists of 469 categories as defined in the [Topics API taxonomy v2](https://github.com/patcg-individual-drafts/topics/blob/a116e9e404dd96c7793fe6542c53e9a9d93cb75b/taxonomy_v2.md). |
There was a problem hiding this comment.
Could we include more info about how recently the chrome5 model was trained?
There was a problem hiding this comment.
Yes, it makes sense. It's currently the latest model shipped in Chrome for topics.
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
Co-authored-by: Rick Viscomi <rviscomi@users.noreply.github.com>
|
Thanks, I'll merge and open follow-up PRs for any updated. |
This pull request adds documentation for the
GET_HOST_CATEGORIESfunction, including its description, usage, and methodology.