Make appstore more robust

[MorrisJobke]

Over time the appstore gets more and more load. As of now it has some caching system in front of the /api/v1/ routes.

That works up to a given point when the caches are about to expire and then the load raises in waves.

Also those are basically pretty static assets. The most problematic URLs are /api/v1/apps.json (2.36 MB gzipped and 5.61 MB uncompressed) and /api/v1/categories.json (22.3 kB gzipped and 114.3 kB uncompressed).

The idea here would be to not do a transparent caching here but go over to statically generated resources that the webserver can serve directly without regularly checking if an update of the cache is required.

Basically the JSON files need to be written on every app release upload and when admins change anything on apps.

Proposed change

• remove those two routes from the Django backend
• add logic to generate those responses and write them to files inside the static/ directory
• add aliases for those two URLs to the files inside static/ and document it in https://nextcloudappstore.readthedocs.io/en/latest/prodinstall.html#configuring-the-web-server
• hook the generator logic into app uploads and changes by admins to the app releases and app itself

Opinions on that? cc @rullzer @blizzz @gary-kim @ChristophWurst

[ChristophWurst]

Isn't it less error-prone to have a simple HTTP caching layer in front of the rarely changing JSON responses instead of having to deal with concurrent IO in the python code base?

That works up to a given point when the caches are about to expire and then the load raises in waves.

Why is that? Shouldn't the cache just need a few requests to repopulate and the app store is free from that load again?

[MorrisJobke]

Isn't it less error-prone to have a simple HTTP caching layer in front of the rarely changing JSON responses instead of having to deal with concurrent IO in the python code base?

Why concurrent? Like the case when two apps are uploaded at the very same time? Then the DB is the source of truth anyways and the JSON might be outdated for a bit. But on the next non-concurrent update of the file everything is fine again.

That works up to a given point when the caches are about to expire and then the load raises in waves.

Why is that? Shouldn't the cache just need a few requests to repopulate and the app store is free from that load again?

Theoretically yes. The Nginx cache is configured like that, but it somehow still doesn't reduce the load in a way that is acceptable. Too many requests are reaching the API endpoint, even if many seem to be cached.

[ChristophWurst]

Why concurrent? Like the case when two apps are uploaded at the very same time? Then the DB is the source of truth anyways and the JSON might be outdated for a bit. But on the next non-concurrent update of the file everything is fine again.

Yes, exactly that. But also concurrent in a sense that wile one process writes the file, another request may read it.

[MorrisJobke]

But also concurrent in a sense that wile one process writes the file, another request may read it.

Write and move should fix that, as move is an atomic operation on posix systems AFAIK.

[bigcat88]

Also those are basically pretty static assets. The most problematic URLs are /api/v1/apps.json (2.36 MB gzipped and 5.61 MB uncompressed) and /api/v1/categories.json (22.3 kB gzipped and 114.3 kB uncompressed).

Current gzip file sizes are more than twice that size, with all CPU cores at 100% utilization at all times.

Let's see this week if these changes helped: #1147

[bigcat88]

At the moment the performance situation looks much better than it was before, so I'll close it for now.