Batching

[cathyzbn]

Describe your changes

Enable batching in modal functions and class methods.
Interface Examples:

@app.function()
@modal.batch(batch_max_size=4, batch_linger_ms=1000)
async def batch_squared(x):
    return [x_i ** 2 for x_i in x]

@app.cls()
class Batch:
    @modal.batch(batch_max_size=4, batch_linger_ms=1000)
    async def batch_squared(self, x):
        return [x_i ** 2 for x_i in x]

Backward/forward compatibility checks

---

Check these boxes or delete any item (or this section) if not relevant for this PR.

• Client+Server: this change is compatible with old servers
• Client forward compatibility: this change ensures client can accept data intended for later versions of itself

Note on protobuf: protobuf message changes in one place may have impact to
multiple entities (client, server, worker, database). See points above.

---

Changelog

[cathyzbn]

For parameter names, @batch and batch_max_size / batch_linger_ms may be redundant, but max_size seems unintuitive especially for ML users who are accustomed to using batch_size?

[mwaskom]

Does this mean we can't do batching with webhook functions?

[mwaskom]

@Batch and batch_max_size / batch_linger_ms may be redundant, but max_size seems unintuitive especially for ML users who are accustomed to using batch_size?

I think it should be intuitive in the context of a batch decorator with a small number of parameters. If the decorator had a lot of parameters maybe the link would be less clear.

[mwaskom]

Where is "linger_ms" coming from by the way? At one point I thought we had max_size and max_wait which I thought nicely contrasted the two parameters that you trade off. "linger" isn't really evoking anything specific for me relating to batching, but maybe I'm missing a reference point.

[cathyzbn]

Does this mean we can't do batching with webhook functions?

Yes, we can't batch the webhook functions but we can do this. I think most of the use cases would probably involve declaring a class for model + make batched inference as one of its methods

@app.function()
@modal.batch(batch_max_size=4, batch_linger_ms=1000)
async def batched_function_async(x):
    return [x_i**2 for x_i in x]

@app.function()
@modal.web_endpoint()
async def f(x: int):
    output = await batched_function_async.remote.aio(x)
    return output

[mwaskom]

What's the technical reason that we can't decorate an endpoint function or method with the batching decorator? I think users will naively expect those to compose.

[gongy]

We're looking into getting it to work with web endpoints, but the short summary is that each web endpoints invocation has ASGI streams of incoming/outgoing data. It's difficult to batch four streams into a single request/response pattern unless the user defines their request handler explicitly.

[cathyzbn]

Split into: https://github.com/modal-labs/modal-client/pull/2064/files
https://github.com/modal-labs/modal-client/pull/2065/files