[RFC] Problems on deploying the projects that has many files ( > 10000 )

[styner32]

We had an issue on builder and deployer due to projects that has a lot of files.

We have fixed the builder by increasing the limit on number of opening files. But it started hanging on deployer due to uploading many files to S3. Because we upload files one each right now. So we had to just remove messages from queue.

Short term solutions

1. Returns an error when it takes more than X minutes to upload files. We just stop uploading and send a error message to users that deployment fails due to too many files.
2. Do batch upload when uploading to S3.
3. Add monitoring for remaining jobs in queue.

Possible solutions

1. Don't upload node_modules
2. Limit file extensions can upload
3. Limit number of files on CLI

Please advise any good solution issues.

[chuyeow]

We should try to make each deployment not affect other deployments, so ideally:

1. Each deployment is handled by a new worker (which uploads files in its own goroutine/process instead of the current "Global Upload Lock").
2. A project should have only 0 or 1 ongoing deployment (including rollbacks, since those are essentially a deployment).

In practice, I think we can only run at most N workers at a time because resources are finite, so all N workers can potentially be hung up on uploading to S3. So we should apply limits (e.g. number of files, timing out) as suggested.