Fonts & images are not being loaded intermittently

[khitrenovich]

We see that fonts & images are not being loaded intermittently in our CI builds. This had happened once in a while all the time, but it had started happening frequently on Wed 6/23 around 10am EST. For example, for one of our projects we a constantly seeing 1-2 screenshots out of 3 that have no fonts loaded. As a result, we have "visual changes on master builds" notifications almost hourly, and Percy PR checks are always red because of the fonts mismatch. This renders Percy pretty much useless for given project.

We on 1.0-beta version of Percy CLI. We upgraded from b52 to b54/b55, then downgraded back to b52 - with the same issues.
Any help will be greatly appreciated.

[Robdel12]

Hey @khitrenovich! Could you link to a build or two (or snapshot IDs) where this is happening? This usually isn't an SDK issue, but it would be good to rule that out for sure (and having a build will help :D)

[khitrenovich]

@Robdel12 You can check 11163677 and 11167161, for example. Pretty much every build of this project has a visual diff - either fonts are being changed / image glyphs disappeared, or things are back to normal.

[Robdel12]

Thanks for that! Looks like it is an infrastructure issue -- all the assets are correctly captured by the SDK. I spoke to some folks that work on the infrastructure and we've uncovered an issue over the past few days and have rolled out a (hopeful fix) over the past hour.

I'll close this issue since it's not an SDK bug, BUT not closing the discussion or triaging here. Very curious to hear how this continues to play out (and the infra folks are aware of this gh issue)

[khitrenovich]

@Robdel12 Thank you, we'll keep an eye and let you know how the things are going!

[khitrenovich]

@Robdel12 It still happens (see build 11170338), although it's hard to tell the frequency yet.

[warreneblackwell]

@Robdel12 It is still happening frequently. See latest builds: 11199388, 11196297, 11195134

[Robdel12]

Hm, this is interesting -- it looks like some of these are actually missing assets being captured by the SDK. Would you be able to add --verbose to the CI run and pass logs over for a build that has missing assets? Those logs will have responses for those assets / hints as to why they're not being captured.

[warreneblackwell]

@Robdel12 - Please see attached logs for build 11249225.

percyLog.txt

[Robdel12]

Thanks for that! The assets that are missing in the snapshot aren't in the logs for asset discovery either. So that means the network idle timeout might be too quick for the these assets to be requested/discovered in asset discovery. I'd try setting the network-idle-timeout to 250ms to see if these assets will be discovered: https://docs.percy.io/docs/cli-configuration#discovery

version: 2
snapshot:
  # your current config here
discovery:
  network-idle-timeout: 250 # ms (up to 500ish is good)

This tells Percy's asset discovery to wait for 250ms for zero requests to be made by the captured DOM snapshot before closing asset discovery (and moving onto the next snapshot). 150 seems a bit too quick and there's assets that aren't being discovered reliably

[vdineva]

I am seeing the same assets loading issues on a different project. I have tried setting the network-idle-timeout to 500ms and it's currently set to 5000ms - but it did not help in my case. The only difference between the 2 projects is that the project I'm working on is on the free trial version. The only thing stopping us from moving to a paid version is this unstable behavior we're experiencing with Percy

[warreneblackwell]

@Robdel12 - setting network-idle-timeout: 250 does not appear to have helped. Some additional info: resources that do not show up are referred from the CSS (whereas others are direct and always show up).

[warreneblackwell]

@Robdel12 - Any further suggestion or insight? We really would like to resolve this ASAP, or the product is not very useful for us.

[Robdel12]

It's possible #469 helps with this. I would also play with the network idle timeout (up to 750ms -- we validate/clamp on that now).

[khitrenovich]

@warreneblackwell @vdineva Try out beta-64 - it fixed similar issue with asset loading for us.

[warreneblackwell]

Thanks Anton!

[Robdel12]

Yea, I think those were the needed fixes that we've been feeling/seeing since our refactor in beta.53 and then upgrading chrome Chrome 87 to 92 in asset discovery really exposed the bug.