Memory leak in version 1.22.0

[rausnitz]

Describe the issue

Memory usage spiked when we deployed our application with postgres-nio 1.22.0.

Vapor version

4.102.1

Operating system and version

Linux

Swift version

Swift 5.10

Steps to reproduce

Run a Swift backend application that uses postgres-nio and compare the memory usage when on 1.21.5 and 1.22.0.

Outcome

Memory usage steadily increases after starting an application that uses postgres-nio 1.22.0.

Additional notes

We observed this on an instance that runs recurring jobs but does not receive web traffic, so memory usage tends to be stable over time.

In the attached screenshot, the first release to include postgres-nio 1.22.0 was on July 26. The sudden drops in memory usage indicate when new releases went live, restarting the app. The graph stabilizes on July 30 when we reverted postgres-nio to 1.21.5.

Edit: I'm not sure if the memory usage spikes in 1.21.6 or 1.22.0.

[MahdiBM]

Repro (No need to even do a query):

@main
struct EntryPoint {

    static func main() async throws {
        let config = PostgresClient.Configuration(...)
        let client = PostgresClient(configuration: config)

        try await withThrowingTaskGroup(of: Void.self) { taskGroup in
            taskGroup.addTask {
                await client.run()
            }

            taskGroup.addTask {
                for idx in (0..<100_000) {
                    if idx % 10 == 0 {
                        print(idx)
                    }
                    try await Task.sleep(for: .milliseconds(10))
                    try await client.withConnection { conn in
                        /// No need to even do a query
                    }
                }
            }

            try await taskGroup.waitForAll()
        }
    }
}

[MahdiBM]

No need to even be able to create a proper connection?:

@main
struct EntryPoint {
    static func main() async throws {
        let config = PostgresClient.Configuration(
            host: "CORRECT-HOST",
            port: 5432,
            username: "USER",
            password: "BAD_PASSWORD",
            database: "DB",
            tls: .disable
        )

        let client = PostgresClient(configuration: config)

        try await withThrowingTaskGroup(of: Void.self) { taskGroup in
            taskGroup.addTask {
                await client.run()
            }

            taskGroup.addTask {
                for idx in (0..<100_000) {
                    if idx % 10 == 0 {
                        print(idx)
                    }
                    try await Task.sleep(for: .milliseconds(10))
                    try await client.withConnection { conn in

                    }
                }
            }

            try await taskGroup.waitForAll()
        }
    }
}

[petrpavlik]

I'll just add that memory leak instruments in Xcode is reporting a leak when profiling my backend code.

[fabianfett]

Okay, found the source of this leak:

With 1.22.0 we started to forward write promises:

postgres-nio/Sources/PostgresNIO/Connection/PostgresConnection.swift

Line 545 in d18b137

private func write<T>(_ task: HandlerTask, cascadingFailureTo promise: EventLoopPromise<T>) {

However in the success case we never forward them to the wire in the PostgresChannelHandler.

postgres-nio/Sources/PostgresNIO/New/PostgresChannelHandler.swift

Line 199 in d18b137

func write(context: ChannelHandlerContext, data: NIOAny, promise: EventLoopPromise<Void>?) {

[petrpavlik]

Would it maybe make sense to revert the commit that has caused the leak before the fix PR is approved and merged in?

[LukasCZ]

Any update on this? This likely affects vast majority of web servers written in Vapor, and it's been broken for three weeks now. When can we expect it to be fixed and merged in? Or should I pin to the previous version of postgres-nio for now?

[fabianfett]

@LukasCZ thanks for pinging. Will prepare a revert pr for now.

[fabianfett]

PR is up: #501

[fabianfett]

Released as 1.22.1

[LukasCZ]

Thank you @fabianfett 🙏