Connection Leak when Circuit Breaker Hystrix Command timeout

[gao288]

My Setup: JerseyClient (2.39) + ApacheConnector + PoolingHttpClientConnectionManager (4.5.x)

Hi I am trying to use Netflix Hystrix Circuit Breaker to wrap the JerseyClient call. In my testing, I found out that when hystrix command is enforced with timeout, and under high TPS, I can see connection leased pool slowly accumulating. I found this behavior got greatly reduced when disable circuit breaker timeout. I have called response.close() in the finally block, so as a caller, I am closing the response and consume the entity in response. Then I suspect this behavior is caused by JerseyClient not releasing the connection when circuit breaker interrupt the thread when hystrix timeout.

In Hystrix's official documentation, it is mentioned that:

Please note that there's no way to force the latent thread to stop work - the best Hystrix can do on the JVM is to throw it an InterruptedException. If the work wrapped by Hystrix does not respect InterruptedExceptions, the thread in the Hystrix thread pool will continue its work, though the client already received a TimeoutException. This behavior can saturate the Hystrix thread pool, though the load is 'correctly shed'. Most Java HTTP client libraries do not interpret InterruptedExceptions. So make sure to correctly configure connection and read/write timeouts on the HTTP clients.

I am curious why Jersey Client cannot handle such exceptions and create potential risks of leaking connections. What is the best practice when using against circuit breaker or other multi threading scenario that could potentially interrupt the network call.

[jansupol]

Can you share a test case? There are three technologies used: Jersey, Apache & Hystrix and we can hardly know what is going on there from your description.

[gao288]

Hi I wasn't able to reproduce this locally, but do see a relation of connection leakage when increased InterruptedException happening.

I would like to ask whether JerseyClient could handle any exception properly (such as interrupted exception) so that connection can be released even not calling response.close() or readEntity.