Reconfigure emulation flags #10910

paulirish · 2020-06-04T19:35:55Z

TLDR: The emulation flags are awkward and there's implicit assumptions made. We should fix that and support our varied usecases with first-class settings. Key idea: get rid of emulatedFormFactor=none.

DevTools, Calibre, and WPT are three lighthouse clients that handle emulation previous to lighthouse running. In DevTools' case, only screen emulation is applied, and lighthouse is expected to apply its own network emulation. And calibre's case, both emulations are applied and the intent is for a lighthouse to not apply double emulation anywhere. LH on WPT supports a few approaches and everyone's been confused at some point.

We introduced the illustrious internalDisableDeviceScreenEmulation flag (#9377) to solve this. (However even with this, LH will still overwrite any previously set userAgentOverride). But if emulatedFormFactor (#6098) is none and external UA emulation is applied, you don't get correct scoring.

This is messy and I think that there's an opportunity to completely redefine this configuration to be more clear.

Goals

no surprises wrt mobile or desktop scoring being used
a more first-class config for this external-emulation setup
continue to support the no-emulation-lh-is-running-against-a-real-phone scenario

Starting Proposal

Rename emulatedFormFactor to formFactor. This is the key flag and the value must be either mobile or desktop, none (née provided) is no longer a valid option. Our perf metrics and 3 SEO metrics change their behavior based on mobile v desktop, so it seems important to make this an explicit user contract.
Add two more flags: disableScreenEmulation and disableNetworkEmulation.
Remove internalDisableDeviceScreenEmulation. Throw if this or emulatedFormFactor are used.
Internally, we drop the guessed TestedAsMobileDevice bool and just use the formFactor enum of 'desktop'|'mobile'.

The combination of these allows us to support the below user stories with clear expectations. It also prevents the case of a using incorrect scoring accidentally.

User stories

I believe we have 4 user stories to support (right?)

Typical LH (we apply emulation): formFactor is set.
DevTools: formFactor is set via the mobile/desktop radio, and screenEmulation is false.
Calibre and WPT: formFactor is set, and both screenEmulation: false, throttlingMethod: 'provided'. emulatedUserAgent is set as desired.
LH on mobile device: --no-screenEmulation and --throttling.cpuSlowdownMultiplier=1. (--formFactor=mobile is the default already)

Extra thoughts and questions

Given this situation and our different desktop scoring, it definitely raises the priority of communicating mobile/desktop run near the score. Add visual indicator to score to warn when it was run with non-default settings #8178 Report needs visible indicator of mobile or desktop run #9379 ☂️Report changes and goals for v6 #9438
Does the CLI default to --formFactor=mobile or do we make it (annoyingly) required?
- the TestedAsMobileDevice logic correctly handles real-mobile-device, but doesn't handle external UA emulation. We could detect there's external UA emulation being applied and throw if we don't see configuration that matches. Or with the proposal implemented, we may just log a warning to let them know we see it.
- docs: Testing on a mobile device still works correctly in v6, works in v6 even with emulated-form-factor=none. We could also detect a potential double-emulation scenario here and warn to make sure flags are correctly setup.
core: add settings.internalDisableDeviceScreenEmulation #9377 (review) has some discussion around custom UA/metrics/touch/etc. I think the above proposal lays good groundwork for allowing that configurability.
Throttling is related, but I haven't yet considered if we should also reconfigure those settings to better align with these. Probably not, but we could optionally detect and warn on unexpected emulation/throttling pairings.
- Desktop throttling preset.. Going through all this I see that clients(devtools): use the same desktop throttling as lightrider #10322 made it so devtools/desktop use the same throttling constants as lr-desktop, but CLI/desktop doesn't. It probly should? Will file as a separate issue.
I was a lil concerned the LH v6 on WPT might have broke something. But looks like Set --emulated-form-factor=desktop for LH tests without device emulation catchpoint/WebPageTest.agent#297 which was merged last week fixes things (judging by Joseph's last comment) - Update: there's a bug but only for the scorecalc handoff, actual scoring is all good.

References

Probably related emulation-y issues: Report Score not matching the Calculator Score (webpagetest.org) - desktop #10836 LHCI fails to run puppeteer script lighthouse-ci#321 Set --emulated-form-factor=desktop for LH tests without device emulation catchpoint/WebPageTest.agent#297
lol: Remove disableDeviceEmulation flag before v5 #7044
Mobile run (with external screen & UA emulation), scored as desktop: https://googlechrome.github.io/lighthouse/viewer/?gist=701d46b57b8252011a25025b2442ca81

The text was updated successfully, but these errors were encountered:

paulirish · 2020-06-04T19:38:20Z

cc @wildlyinaccurate I have a feeling you'd have been much happier with the above proposal, so you wouldn't have had to do all the investigation in catchpoint/WebPageTest.agent#297. But curious for your thoughts. :)

also cc @benschwarz @alekseykulikov @mattzeunert

connorjclark · 2020-06-04T21:54:08Z

Related: I think we may need something like setEmulation cb in the Lighthouse runner API so that we can reset the emulation correctly for programmatic usages. See #10716 (comment)

patrickhulce · 2020-06-05T01:16:59Z

Rename emulatedFormFactor to formFactor. This is the key flag and the value must be either mobile or desktop, none (née provided) is no longer a valid option.

Sounds great, love it 👍

Add disableScreenEmulation

Basically required to do the above, SGTM 👍

Remove internalDisableDeviceScreenEmulation. Throw if this or emulatedFormFactor are used.

Sounds like good housekeeping given the above 👍

Internally, we drop the guessed TestedAsMobileDevice bool and just use the formFactor enum of 'desktop'|'mobile'.

Feels a little weird that formFactor is at this point just acting more like scoringMode: 'easy' | 'hard' instead of what the form factor actually was. Not sure I completely buy into dropping TestedAsMobileDevice for double checking it matched formFactor

Or with the proposal implemented, we may just log a warning to let them know we see it.

Speak of the 👿 Yeah let's do that instead :)

Add disableNetworkEmulation

This is the one I don't understand :) Sounds like it's just returning us to the old days of duplicate flags controlling the same thing.

brendankenny · 2020-06-15T21:36:52Z

Old comment from #9377 (review)

If we were designing our flags from scratch we definitely wouldn't want to overlap with emulatedFormFactor like this :/

The combo with provided makes emulatedFormFactor actually emulatedUAString, and if you're just trying to get your settings right, it's not clear why you'd want emulatedViewportMethod vs emulatedFormFactor and what each one does (and is Lighthouse providing provided or am I?).

See below for one suggestion to simplify this, but what if we just went all the way and added free-form emulation variables for UA string, metrics, and touchEnabled (in the style of throttling)? emulatedFormFactor would stay a set of presets that most would use, but it would be possible to override. This would also unlock what PaulKinlan is looking for re: feature phones.

Not sure if I completely agree with this, but just splitting up all our emulation options so that they don't have interactions (unexpected or otherwise) anymore does have a certain appeal. Most people will still use the presets, and the people heavily tweaking will want to heavily tweak anyways.

paulirish · 2020-06-15T21:45:42Z

discussed. plan:

lets expose screen emu properties (width, height, dpr) to config.
let's expose UA to config. Allow custom user-agent via --extra-headers #8756
determine disable* in relation to these.
perhaps in the calibre/wpt "provided" throttling setup, we should have more useful text in runtime settings. do they want to set a string or .... ? HTML Report renderer incorrect labels for custom throttling & device #7053

updated:

probably dont need disableScreenEmulation and instead can pass false as screenEmulation config value
probably need to determine inconsistencies between formFactor and emulation state. (testing as mobile but it looks like desktop)
emulatedUserAgent is also falseable
--preset=desktop
extends: 'lighthouse:mobile', and extends: 'lighthouse:desktop', @patrickhulce
- perhaps we drop lighthouse-default as it's a mobile thing and too many default assumptions?

benschwarz · 2020-06-15T23:17:47Z

Thanks for the ping @paulirish ✌️

lets expose screen emu properties (width, height, dpr) to config.
let's expose UA to config.
determine disable* in relation to these.

SGTM.

perhaps in the calibre/wpt "provided" throttling setup, we should have more useful text in runtime settings. do they want to set a string or .... ?

Yeah, for completeness I think it'd be good to be able to set some properties that explain the test environment more deeply. I raised this on #7053 but never took it anywhere.

paulirish · 2020-06-16T01:03:38Z

oh word! k we'll break off my number 4 into #7053. perfect.

amannn · 2020-09-10T09:10:17Z

Thanks for your great work on lighthouse!

I've experienced some different scores when running lighthouse on CI than when running locally in regards to the total blocking time.

After some research, I think this is due to the cpuSlowdownMultiplier option as it's relative to the available CPU power. My CI runner already is a bit less capable, therefore I was seeing lower numbers there than when testing locally on my MacBook Pro.

I was wondering if there's a way to normalize this behaviour across environments, so the score would be the same, regardless of the machine? Obviously, there would be some minimum CPU requirements, but ideally from there on the result would be the same. I'm not sure if this is somehow possible practically?

patrickhulce · 2020-09-10T15:12:02Z

I think you'd be interested in the saga of #9085 and the resulting calibration documentation @amannn :)

tl;dr - we tried to do this, but it's very, very difficult and even benchmarks built by dedicated benchmark companies can't predict machine performance well enough to normalize Lighthouse scores, so a single contributor here using 20% of their time won't be able to solve it either :/

amannn · 2020-09-11T07:23:42Z

@patrickhulce Oh right, that calibration guide is really helpful! I definitely understand that this is a hard problem.

Thank you for your help!

An extra check I wanted to add on top of #11779 It was described back in #10910 > the TestedAsMobileDevice logic correctly handles real-mobile-device, but doesn't handle external UA emulation. We could detect there's external UA emulation being applied and throw if we don't see configuration that matches. Or with the proposal implemented, we may just log a warning to let them know we see i We can only check this sort of mismatch after we've gathered the host UA, so it can't be done earlier.

paulirish added needs-discussion 7.0 breaking labels Jun 4, 2020

devtools-bot added the needs-priority label Jun 4, 2020

paulirish mentioned this issue Jun 4, 2020

handle formFactor=none paulirish/lh-scorecalc#16

Closed

paulirish added P2 and removed needs-discussion needs-priority labels Jun 23, 2020

paulirish self-assigned this Jun 23, 2020

abarre mentioned this issue Jun 25, 2020

UAModifier not taken into account in lighthouse catchpoint/WebPageTest.agent#358

Open

brendankenny mentioned this issue Jun 26, 2020

report: add full-page-screenshot to experimental config #10716

Merged

connorjclark mentioned this issue Jul 17, 2020

Ensure running lighthouse with external emulation should work with full-page-screenshot #11122

Open

patrickhulce mentioned this issue Aug 27, 2020

Desktop performance score lower in CLI than when testing in the browser GoogleChrome/lighthouse-ci#430

Closed

This was referenced Sep 15, 2020

Allow custom user-agent via --extra-headers #8756

Closed

☂️ 👣 Breaking changes for v7 #11207

Closed

patrickhulce mentioned this issue Dec 3, 2020

core(fr): add base fraggle rock snapshot runner #11748

Merged

paulirish mentioned this issue Dec 7, 2020

core(emulation): refactor emulation settings & CLI flags #11779

Merged

patrickhulce mentioned this issue Dec 15, 2020

core(config): only allow lighthouse:default extension #11835

Merged

devtools-bot closed this as completed in #11779 Dec 16, 2020

rposbo mentioned this issue Sep 27, 2021

Lighthouse emulation for desktop sites is currently broken catchpoint/WebPageTest#1554

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconfigure emulation flags #10910

Reconfigure emulation flags #10910

paulirish commented Jun 4, 2020 •

edited

Loading

paulirish commented Jun 4, 2020 •

edited

Loading

connorjclark commented Jun 4, 2020 •

edited

Loading

patrickhulce commented Jun 5, 2020

brendankenny commented Jun 15, 2020 •

edited

Loading

paulirish commented Jun 15, 2020 •

edited

Loading

benschwarz commented Jun 15, 2020

paulirish commented Jun 16, 2020

amannn commented Sep 10, 2020

patrickhulce commented Sep 10, 2020

amannn commented Sep 11, 2020

Reconfigure emulation flags #10910

Reconfigure emulation flags #10910

Comments

paulirish commented Jun 4, 2020 • edited Loading

Goals

Starting Proposal

User stories

Extra thoughts and questions

paulirish commented Jun 4, 2020 • edited Loading

connorjclark commented Jun 4, 2020 • edited Loading

patrickhulce commented Jun 5, 2020

brendankenny commented Jun 15, 2020 • edited Loading

paulirish commented Jun 15, 2020 • edited Loading

benschwarz commented Jun 15, 2020

paulirish commented Jun 16, 2020

amannn commented Sep 10, 2020

patrickhulce commented Sep 10, 2020

amannn commented Sep 11, 2020

paulirish commented Jun 4, 2020 •

edited

Loading

paulirish commented Jun 4, 2020 •

edited

Loading

connorjclark commented Jun 4, 2020 •

edited

Loading

brendankenny commented Jun 15, 2020 •

edited

Loading

paulirish commented Jun 15, 2020 •

edited

Loading