Race condition while converting from Slick DatabasePublisher

[jendakol]

I ran into an interesting issue.
I'm creating an abstraction over Slick where methods should return fs2.Stream thus I need to convert its DatabasePublisher to Stream somehow - fs2-reactive-streams is an ideal candidate. However, in unit tests I've discovered that some items are missing when querying multiple tables and then merging (joining) the streams - e.g. I have 20 tables containing 1000 items in total but the result contains 995-1000 items (briefly, few items are missing).
The problem is not in the Slick itself, because i tried more approaches (serializing into Seq, converting to Monix Observable, converting to fs2.Stream through an FS2 Queue) and those were always fine (30 tries) - the only two incomplete were "converting to fs2.Stream from Slick DatabasePublisher" and "converting to fs2.Stream through Monix Observable" (incomplete in cca 10% cases).
The sad thing is I didn't succeed with proving the bug in test with skipping the Slick (just generating random data), thus I can provide you my code but it contains many irrelevant/boring stuff -> it looks like a problem in cooperation Slick <-> fs2-reactive-streams to me! 😢
I will do a further investigation (it's my main task in work right now) but.. do you have any idea?

P.S. I use fs2 0.10.6 and therefore fs2-reactive-streams 0.5.1. I need a last stable version of fs2 because newer version throws me into never ending dependency hell. Is it possible that I ran into some bug that has been already fixed in newer version? (I will try to use it tomorrow if it's even possible for me...)

[jendakol]

OK, it works with version 0.8.0 (and transitive fs2 1.0.0-M5). But I still need to get it working with latest stable... :-(

[zainab-ali]

Thanks for raising this @jendakol. Perplexingly, there haven't been any bug fixes between version 0.5.1 and the current version - just the upgrades to fs2 1.0.0 milestones.

It could be that the bug, whatever it was, was in fs2 itself, in which case you'll have a hard time upgrading.

[jendakol]

Well I ended up with doing the conversion to Stream by myself (sorry...) through async Queue. It's a little bit slower but doesn't lose anything 😃 I can provide you my code if you want to do some testing - I already spent much more time than I could on it 😐

Once I'll be able to use the newest versions, I'll get back to your library.
Thx.