Missing data since ~2021-03-22

[crflynn]

No description provided.

[crflynn]

It looks like the Google BigQuery table for downloads are missing partitions for the following dates

• 2021-03-22 through 2021-03-26
• 2021-03-29 onwards

This corresponds to the recent missing data shown on pypistats.org visualizations here: downloads

So this must be an issue upstream related to either

• https://github.com/pypa/linehaul-cloud-function
• https://github.com/pypa/linehaul
• or whatever triggers the linehaul cloud function

Will do some research upstream...

[crflynn]

The downloads table appears to have stopped updating, however there is another table called file_downloads which appears to have the same schema/partition and has been updating.

[crflynn]

Looks like file_downloads is newer and stable: https://twitter.com/sethmlarson/status/1347236470688542721

[crflynn]

Currently (slowly) backfilling data since Jan 4.

[crflynn]

We're caught up, at least since Jan 4. Based on the lack of weekly cadence in older data it looks like the data producer has been broken/unreliable for a while. :(

According to linehaul maintainers it looks like the newer implementation is better and the data is at least somewhat recoverable if upstream processing fails.

[jewettaij]

Thanks crflynn!