wip: GRPC server

[john-z-yang]

Turns out our grpc version is too old, bumps grpcio version to 1.65.4.
Run devenv sync to install the new dependency, then run pip install -e ../sentry-protos/py to install the code gen from our sentry protos branch (worker-pull).

You can start the server with
sentry run kafka-task-grpc-server
It doesn't do anything yet, but it's useful to check if imports are successful.

John Sept: 19:
I decided to spend the entire day today to get rid of the PendingTask class that the PendingTaskStore has.

The result of this is that PendingTaskStore and the consumer will use a protobuf data structure directly instead of the python dataclass that wraps a protobuf data structure.

The main reason being that in order to use grpc we cannot pass PendingTask between the services (because it's a python class that wraps a protobuf type, plus some additional data), and instead of creating yet another Protobuf definition, I just added the fields into the existing protobuf message.

The new protobuf messages are structured like the following (name are probably going to change):

Work: An abstract unit of work, which only contains information like retry policy, the task ID, and the function parameters. Theses are the messages that are on Kafka.

Task: The actual physical task for the worker, which contains a Work, but also things like status, topic, partition, recieved_at. These are the messages the consumer stores into the PendingTaskStore.

This leaves us with a single python data structure
PendingTask: The Django model, used only in PendingTaskStore and it's just has all the fields a Task has, but flattened.

Now that PendingTaskStore is just taking a protobuf message (Task), this makes implementing the grpc server relatively simple.

def GetTask(self, request: GetTaskRequest, context) -> GetTaskResponse:
    # task is already a GRPC message
    task = self.pending_task_store.get_pending_task(
        request.partition if request.partition else None,
        request.topic if request.topic else None,
    )
    return GetTaskResponse(task=task)

TODO:

• Implement the functions defined in ConsumerServicer (we need to make sure to use the PendingTaskStore abstraction here)
• Have the worker fetch the tasks via grpc by calling gRPC stub in TaskService
• Test different failure modes

[john-z-yang]

We might want to move this to pending_task_store.py because it is using PendingTasks which is the Django model which should be kept in pending_task_store.py. Also the only thing that calls this is pending_task_store.py.

Alternatively, we can make this function accept a protobuf Task instead of PendingTasks

[john-z-yang]

Removed the return value here because it's not clear to me what we're trying to accomplish with the return value (nothing uses it)