Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Query Hangs if Connection is Closed #487

Merged
merged 25 commits into from
Jun 24, 2024

Conversation

MahdiBM
Copy link
Contributor

@MahdiBM MahdiBM commented Jun 19, 2024

See the test for how to trigger this.

@MahdiBM
Copy link
Contributor Author

MahdiBM commented Jun 19, 2024

Cancelled the integration tests CI manually, since they wouldn't have finished.

Side note @gwynne: probably good to have a time limit in CIs.

@gwynne
Copy link
Member

gwynne commented Jun 19, 2024

Side note @gwynne: probably good to have a time limit in CIs.

The default timeout is 1 hour, IIRC.

@MahdiBM MahdiBM changed the title Query Hangs if Connection is Closed Mid-Query [Fix] Query Hangs if Connection is Closed Mid-Query Jun 19, 2024
Copy link

codecov bot commented Jun 19, 2024

Codecov Report

Attention: Patch coverage is 75.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 55.28%. Comparing base (7b621c1) to head (ceaa688).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #487      +/-   ##
==========================================
+ Coverage   54.98%   55.28%   +0.30%     
==========================================
  Files         127      127              
  Lines       10155    10161       +6     
==========================================
+ Hits         5584     5618      +34     
+ Misses       4571     4543      -28     
Files Coverage Δ
...es/PostgresNIO/Connection/PostgresConnection.swift 48.57% <75.00%> (+6.32%) ⬆️

... and 3 files with indirect coverage changes

Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch! We really need to make sure we cover all cases by adding unit tests for each of them!

Tests/IntegrationTests/PSQLIntegrationTests.swift Outdated Show resolved Hide resolved
Tests/IntegrationTests/PSQLIntegrationTests.swift Outdated Show resolved Hide resolved
Sources/PostgresNIO/Connection/PostgresConnection.swift Outdated Show resolved Hide resolved
Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry used wrong button before :)

@fabianfett
Copy link
Collaborator

the test I added as an example hangs 100% of time for me.

@MahdiBM
Copy link
Contributor Author

MahdiBM commented Jun 20, 2024

the test I added as an example hangs 100% of time for me.

Hmm you're right. I thought i tested that properly and it turned out like i said, but apparently not.
I did try to remove the loop like you did with just closing the channel beforehand, but i thought that didn't work out to get the hang.

I was likely testing with my fixes in there, and not reverted.

@MahdiBM MahdiBM marked this pull request as draft June 20, 2024 15:17
@MahdiBM MahdiBM requested a review from fabianfett June 20, 2024 15:50
@MahdiBM MahdiBM marked this pull request as ready for review June 20, 2024 15:50
@MahdiBM
Copy link
Contributor Author

MahdiBM commented Jun 20, 2024

ahh hmm CI is failing with Fatal error: SWIFT TASK CONTINUATION MISUSE: listen(_:) tried to resume its continuation more than once, throwing ioOnClosedChannel! need to investigate.

@MahdiBM
Copy link
Contributor Author

MahdiBM commented Jun 20, 2024

I reckon I didn't add a unit test for "each" modified function. Should I?

@MahdiBM MahdiBM changed the title [Fix] Query Hangs if Connection is Closed Mid-Query [Fix] Query Hangs if Connection is Closed Jun 20, 2024
@fabianfett
Copy link
Collaborator

I reckon I didn't add a unit test for "each" modified function. Should I?

Yes, please!

}
}

func testListenFailsIfConnectionIsClosedMidway() async throws {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What connection state do you want to test here? In which connection state does this happen? Can we mock this explicitly and not by accident?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is consistent, it only waits 3 seconds to make sure the listener has started listening.

It tests that if there is a listen in process, and then the connection is closed, the listener sequence needs to fail.

Copy link
Collaborator

@fabianfett fabianfett Jun 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah! in that case we should wait for the client messages on the channel instead of using a timer.

See here how it is done:

func testSimpleListen() async throws {
let (connection, channel) = try await self.makeTestConnectionWithAsyncTestingChannel()
try await withThrowingTaskGroup(of: Void.self) { taskGroup in
taskGroup.addTask {
let events = try await connection.listen("foo")
for try await event in events {
XCTAssertEqual(event.payload, "wooohooo")
break
}
}
let listenMessage = try await channel.waitForUnpreparedRequest()
XCTAssertEqual(listenMessage.parse.query, #"LISTEN "foo";"#)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clues, I did this in the new test.
We no longer need to change that fatalError("Invalid state: \(self.state)"), it appears that was triggered as a result of my bad test.

@fabianfett i think i no longer need to make another PR out of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fabianfett I think this PR's changes are good, but I noticed this results in a fatal error regardless of my changes:

    func testAddListenerFailsIfConnectionIsClosed() async throws {
        let (connection, channel) = try await self.makeTestConnectionWithAsyncTestingChannel()

        connection.addListener(channel: "example") { context, notification in
            XCTFail("Did not expect to receive")
        }

        try await connection.close()

        XCTAssertEqual(channel.isActive, false)
    }
Screenshot 2024-06-23 at 1 16 11 AM

@MahdiBM MahdiBM marked this pull request as draft June 22, 2024 10:36
@MahdiBM MahdiBM marked this pull request as ready for review June 22, 2024 10:56
@MahdiBM MahdiBM marked this pull request as draft June 22, 2024 10:57
@MahdiBM MahdiBM marked this pull request as ready for review June 22, 2024 21:47

try await channel.writeInbound(PostgresBackendMessage.notification(.init(backendPID: 12, channel: "foo", payload: "wooohooo")))

try await connection.close()
Copy link
Contributor Author

@MahdiBM MahdiBM Jun 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using .close() instead of .closeGracefully() here because .closeGracefully() will wait it out till infinity and beyond, apparently because it's "graceful" and the listener is still listening.

Copy link
Collaborator

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great patch! Thanks for pushing through!

@fabianfett fabianfett merged commit f55caa7 into vapor:main Jun 24, 2024
11 checks passed
@fabianfett fabianfett added the semver-patch No public API change. label Jun 24, 2024
fabianfett added a commit to fabianfett/postgres-nio that referenced this pull request Aug 20, 2024
fabianfett added a commit that referenced this pull request Aug 20, 2024
MahdiBM added a commit to MahdiBM/postgres-nio that referenced this pull request Sep 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver-patch No public API change.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants