Desktop performance score lower in CLI than when testing in the browser

[amannn]

I'm using @lhci/cli to test the desktop viewport of a web app I'm developing. I'm using the emulatedFormFactor: 'desktop' setting.

When running in lhci, I get this score:

However when I run the test with the same exact URL in Chrome, I get this score:

I have no idea what is causing this.

Note that the mobile performance score of the same website is 100 in lhci – exactly the same as in Google Chrome.

Do you have any pointers? The app is private, therefore I sadly can't share a reproduction at this point. Thank you for your help!

Versions:

• @lhci/cli: 0.4.4 (although I got the same result when trying out 0.5.0)
• Chrome: Version 84.0.4147.135 (Official Build) (64-bit)
• OS: macOS 10.15.4 (19E287)
• Chrome Lighthouse: 6.0.0
• @lhci/cli Lighthouse: 6.0.0

[patrickhulce]

Thanks for filing @amannn!

When you're running from the CLI (like LHCI does) form factor and throttling are controlled separately unlike Chrome DevTools which handles this behind the scenes for you (see desktop throttling profile).

module.exports = {
  ci: {
    collect: {
      settings: {
        emulatedFormFactor: 'desktop',
        throttling: {rttMs: 40, throughputKbps: 10 * 1024, cpuSlowdownMultiplier: 1},
      }
  }
}

We'd welcome adding this example config to the docs if you're up for a PR :)

[amannn]

Oh, I see! Thanks for your quick reply – now this works as expected 🙂 .

Would it make sense for lighthouse-ci to include an option to set the device like lighthouse does it in the Chrome DevTools?

The option would control both the form factor and the throttling.

[patrickhulce]

Would it make sense for lighthouse-ci to include an option to set the device like lighthouse does it in the Chrome DevTools?

We don't really want to drift from the official Lighthouse programmatic API. It's being considered over there for 7.0 if you want to chime in GoogleChrome/lighthouse#10910

[amannn]

Right, I see. I'll open a PR to add the desktop settings to the docs as you proposed above.

[ankityadav4]

@patrickhulce I am trying Lighthouse CLI and facing similar issue with huge difference on Lighthouse UI and Lighthouse CLI.

I am running below command:
lighthouse --output html --output-path C:\0508.html --skip-audits bf-cache --preset=desktop --only-categories performance --throttling.rttMs 40 --throttling.throughputKbp 10240 --throttling.cpuSlowdownMultiplier 1

I am getting a differnece of at least 20-25 in Lh score.

What am i doing wrong?

[patrickhulce]

That's not really enough information to explain why. If you can share the full reports, we might be able to provide more assistance. (depending on metric values, 20-25 isn't that unexpected variance)

[ankityadav4]

@patrickhulce Please find the attached reports. Do you have any pointers?
Lighthouse Report.pdf
LighthouseReport-UI.pdf

[adamraine]

@ankityadav4 as patrick said the variance you are experiencing is not unexpected. There are a variety of reasons that the performance score of a page could change between runs.

The page content is slightly different between the two reports, this could affect your performance score. Please read our variance docs for more help.

[adamraine]

I'm going to lock this issue as it is very old. If you have a similar issue, please read our variance docs before filing a new issue.