What is a deployment unit?

[marcellanz]

If an image is deployed on k8s, and I deploy 10 of them, resource wise what is the footprint of such a deployment? There is copy on write in linux, right? How much of a image is duplicated at runtime? especially of the parts that are not writeable or never written to.
I'm asking because I see how much of resources a small 7 service deployment at Akkaserverless (AS) is taking, and I think we have to anticipate for that in our implementation too.

On AS: 7 services => 10GB memory used, ~20-25%CPU burning
AS is advocating 1 Service to be 1 deployment unit; and therefore 1 image deployed in a pod.

context:
https://discuss.lightbend.com/t/resource-usage-costs-7-services-10gb-memory-used-20-25-cpu-burning/8370

[marcellanz]

Resourcewise, one issue AS has is, that they advise to deploy each service separately, where each such service maps 1:1 to a gRPC Service: https://developer.lightbend.com/docs/akka-serverless/designing/programming-model.html#_what_is_an_akka_serverless_service
This has the implication, that for every service, a container is running and also a service proxy sidecar. We would have the same, probably with a smaller sidecar image and runtime size.

There are alternatives to that model.

• One is, to see a gRPC service as immutable, so once released in an image, it will never change except there is a new version of it. When we then take multiple services per "service pod" they share a language runtime and also only one sidecar proxy.
• Other is to have one service proxy servicing multiple user function pods.

In general, the k8s pods scheduling model probably does not fit well when it comes to have user-functions implemented in languages that bring a heavy environment with them. Having the sidecar being multiple hundreds of MBytes even doesn't help here and so we have to have an eye on that for out BEAM based proxy.

[sleipnir]

If an image is deployed on k8s, and I deploy 10 of them, resource wise what is the footprint of such a deployment? There is copy on write in linux, right? How much of a image is duplicated at runtime? especially of the parts that are not writeable or never written to.
I'm asking because I see how much of resources a small 7 service deployment at Akkaserverless (AS) is taking, and I think we have to anticipate for that in our implementation too.

On AS: 7 services => 10GB memory used, ~20-25%CPU burning
AS is advocating 1 Service to be 1 deployment unit; and therefore 1 image deployed in a pod.

context:
https://discuss.lightbend.com/t/resource-usage-costs-7-services-10gb-memory-used-20-25-cpu-burning/8370

Very good discussion.
The model proposed by LB is based on scalability per service, which makes a lot of sense to me. Having a pod with many services would mean the impossibility of scaling the services individually, and therefore, in addition to the technical difficulties, it must also imply some difficulty in consolidating the Billing of these services.

[marcellanz]

How do you see our options to reduce the resources imposed by each sidecar per service-pod?

[sleipnir]

1. Smallest possible image size.
2. Caching strategies.
3. Keep Process state as small as possible in memory.
4. Use BeamJIT.

[sleipnir]

I've thought of a proxy model for multiple pods before and I think this would bring certain difficulties. Today the proxy is co-located with the user function this lessens the network footprint and also security concerns, just to remember our proxy creates user services at runtime and this is not a security concern as we know it does in the scope of localhost, if we had to handle this via real network we would have a medium/severe security hole in our proxy that we would have to resolve.

What I've seen in the past of LB's proxy is that it suffered from a problem any GraalVM application will suffer from, which is memory deallocation, GraalVM takes a long time to return used memory to the system (obviously because allocating and deallocating memory blocks costs a lot) and that along with the fact that object-oriented applications consume more memory causes the effect you mentioned.
In our case I believe our memory footprint will be smaller at most times with certain sharp spikes at some peak times. Another point is that I think our biggest problem is the CPU consumption which should be high overall.
I think we have to start doing benchmarks (we can use stateless service for now) to better understand the behavior of our proxy and see what points we have to attack.

A related point I've been thinking about (startup time) about implementing is caching the compiled services of a user function, since a proxy would compile a version of a service it wouldn't be necessary to do the work of compiling the same version of a service in another instance. So if you scale a service from 1 to N the compilation cost would be paid on the first service.

[sleipnir]

It's important to try to understand what we're competing with as well, our Serveless model differs from the Lambda model. I also have doubts about whether lambdas functions don't have similar spikes either, the difference is that they scale to 0 when they don't have input requests.

[sleipnir]

Let's not forget that in Kubernetes you impose certain limits that the proxy will need to respect, so if the proxy is using 100% of the cpu given to it, this cpu can be just a slice of the total cpu of a Kubernetes Node. So I consider this less of a problem than memory usage which can be much more serious because if we use too much memory we can get the pod to be brutally murdered by Kubernetes which is not good.

[marcellanz]

@sleipnir agree on all of that. I wanted to have a common understanding that we chase the right setting to be aware of what resources we have to allocate and therefore set on. Awareness is for me more important at the moment than that we already fight this topic, where clearly other topics are more important.

[sleipnir]

Yes I agree, as we have full control over the pod deployment we can try to find the best estimate for the cluster and configure the Kubernetes limits as we see fit. But it would be interesting to allow the user to customize these points by themselves, advanced users can help us configure the system according to their needs.

[sleipnir]

This together with the BEAM settings I mentioned should make our proxy at least more flexible and with a better profile than LB.