Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm][firefox] crit: OpenQA.Selenium.WebDriverException: Failed to decode response from marionette #101617

Open
1 of 3 tasks
lewing opened this issue Apr 26, 2024 · 12 comments · Fixed by dotnet/xharness#1221
Open
1 of 3 tasks
Assignees
Labels
arch-wasm WebAssembly architecture area-Infrastructure-mono disabled-test The test is disabled in source code against the issue os-browser Browser variant of arch-wasm
Milestone

Comments

@lewing
Copy link
Member

lewing commented Apr 26, 2024

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

Requires increasing the shared memory. This edit of description is done to make PR's pipeline pass Enforce GitHub issue link presence step.

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=656594
Build error leg or test failing: WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
Pull request: #101538

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "crit: OpenQA.Selenium.WebDriverException: Failed to decode response from marionette",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=656594
Error message validated: [crit: OpenQA.Selenium.WebDriverException: Failed to decode response from marionette]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/26/2024 5:46:49 PM UTC

Report

Build Definition Test Pull Request
663429 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101640
663389 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101808
663336 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101762
663287 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101781
663268 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101800
663262 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100951
663222 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101797
663200 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101624
662930 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101761
663164 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101640
663076 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100951
663068 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101717
662942 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101684
662938 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100951
662949 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101767
662830 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
662542 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101761
662574 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100619
662347 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101624
662312 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #99490
662296 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101753
662281 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
662225 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
662158 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101761
662154 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101277
662112 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101451
662100 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101753
662046 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101732
661923 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
662033 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101767
661965 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101766
661299 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101745
661873 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101762
661888 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101716
661869 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101624
661816 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101717
661799 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101443
661662 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100266
661671 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
661154 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101715
661633 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
661543 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101743
661229 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101735
661476 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101755
661485 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
661442 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101743
661457 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101732
661428 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101753
661405 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100030
661384 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101752
661374 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101751
661304 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101674
661247 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101740
661214 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101277
661111 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
660990 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101631
660981 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101624
660955 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101451
660967 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101212
661010 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101693
661006 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101693
660940 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101725
661019 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101318
660849 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
660861 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101674
660836 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100619
660793 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101684
660756 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101712
660679 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101717
660693 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101660
660723 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101716
660615 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101715
660246 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101705
660588 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100823
660568 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #99818
660529 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101653
660471 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101693
660460 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101712
660440 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100266
660399 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
660391 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101671
660374 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
660018 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101318
660346 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101162
660271 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101626
660253 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
660135 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
660111 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101626
660077 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101212
660071 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101702
659911 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101624
659901 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
659840 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101626
659826 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101196
659861 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101693
659851 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution
659792 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #100266
659789 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101690
659721 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101212
659694 dotnet/runtime WasmTestOnFirefox-ST-System.Runtime.Tests.WorkItemExecution #101151
Displaying 100 of 181 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 181
@lewing lewing added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab labels Apr 26, 2024
@lewing lewing added this to the 9.0.0 milestone Apr 26, 2024
@lewing lewing added arch-wasm WebAssembly architecture os-browser Browser variant of arch-wasm area-Infrastructure-mono and removed area-Infrastructure-libraries labels Apr 26, 2024
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Copy link
Contributor

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Apr 26, 2024
@lewing lewing added in-pr There is an active PR which will close this issue when it is merged blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' disabled-test The test is disabled in source code against the issue and removed blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' in-pr There is an active PR which will close this issue when it is merged labels May 2, 2024
@lewing
Copy link
Member Author

lewing commented May 2, 2024

test is now disabled, failures should stop rolling in soon

24-Hour Hit Count 7-Day Hit Count 1-Month Count
24 175 175

@pavelsavara
Copy link
Member

This has some hints https://stackoverflow.com/questions/49734915/failed-to-decode-response-from-marionette-message-in-python-firefox-headless-s

  • docker memory need be at least 2gb
  • or there are firefox zombies from previous run

@pavelsavara pavelsavara removed the Known Build Error Use this to report build issues in the .NET Helix tab label May 15, 2024
@pavelsavara pavelsavara modified the milestones: 9.0.0, Future May 15, 2024
@akoeplinger
Copy link
Member

akoeplinger commented May 16, 2024

Yeah looks like we'd need a way to specify either --shm-size or mount the host's /dev/shm as a volume into the container.

Here's where we run docker in helix: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/resources/helix-scripts/helix/dockerhelper.py&version=GBmain&line=162&lineEnd=177&lineStartColumn=1&lineEndColumn=1&lineStyle=plain&_a=contents

@ilonatommy
Copy link
Member

ilonatommy commented Jun 11, 2024

All LibraryTests_HybridGlobalization on firefox fail because of it.

Yeah looks like we'd need a way to specify either --shm-size.

The link you pasted gives the answer, docker's client run function has shm_size parameter. I can make this change if it would be an accepted solution.
https://docker-py.readthedocs.io/en/stable/containers.html?highlight=shm_size#docker.models.containers.ContainerCollection.run

edit: by default it's 64MB.

@akoeplinger
Copy link
Member

akoeplinger commented Jun 12, 2024

Yeah I think adding the parameter with e.g. 2g is the best option we have. You'll need to send a PR to the dotnet-helix-machines repo I linked above and wait for dnceng to get it rolled out.

@ilonatommy
Copy link
Member

@ilonatommy
Copy link
Member

It is not caught because runtime-wasm is excluded from Build Analysis step and the failures happen on runtime-wasm (Build browser-wasm windows Release LibraryTests_HybridGlobalization) e.g. log

Console log: 'WasmTestOnFirefox-ST-Common.Tests' from job 1443fe94-33bc-46e6-bbf7-575bc33b2e86 (windows.amd64.server2022.open.rt) using docker image mcr.microsoft.com/dotnet-buildtools/prereqs:windowsservercore-ltsc2022-helix-webassembly on a002B1J
running %HELIX_CORRELATION_PAYLOAD%\scripts\1a81aacd35e2468bb85a763d39aec6a5\execute.cmd in C:\h\w\ABFC097D\w\B776098D\e max 1800 seconds
....
JavaScript error: resource://gre/modules/WindowsLaunchOnLogin.sys.mjs, line 28: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIWindowsRegKey.open]
JavaScript error: resource://gre/modules/WindowsLaunchOnLogin.sys.mjs, line 28: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIWindowsRegKey.open]
[18:52:33] crit: OpenQA.Selenium.WebDriverException: Failed to decode response from marionette
                    at OpenQA.Selenium.Remote.RemoteWebDriver.UnpackAndThrowOnError(Response errorResponse)
                    at OpenQA.Selenium.Remote.RemoteWebDriver.Execute(String driverCommandToExecute, Dictionary`2 parameters)
                    at OpenQA.Selenium.Remote.RemoteWebDriver.StartSession(ICapabilities desiredCapabilities)
                    at OpenQA.Selenium.Remote.RemoteWebDriver..ctor(ICommandExecutor commandExecutor, ICapabilities desiredCapabilities)
                    at OpenQA.Selenium.Firefox.FirefoxDriver..ctor(FirefoxDriverService service, FirefoxOptions options, TimeSpan commandTimeout)
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestBrowserCommand.<>c__DisplayClass13_0.<GetFirefoxDriver>b__1(FirefoxDriverService driverService) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmTestBrowserCommand.cs:line 146
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestBrowserCommand.CreateWebDriver[TDriverService](Func`1 getDriverService, Func`2 getDriver) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmTestBrowserCommand.cs:line 308
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestBrowserCommand.CreateWebDriver[TDriverService](Func`1 getDriverService, Func`2 getDriver) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmTestBrowserCommand.cs:line 313
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestBrowserCommand.GetFirefoxDriver(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmTestBrowserCommand.cs:line 144
                    at Microsoft.DotNet.XHarness.CLI.Commands.Wasm.WasmTestBrowserCommand.InvokeInternal(ILogger logger) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/WASM/Browser/WasmTestBrowserCommand.cs:line 72
                    at Microsoft.DotNet.XHarness.CLI.Commands.XHarnessCommand`1.Invoke(IEnumerable`1 arguments) in /_/src/Microsoft.DotNet.XHarness.CLI/Commands/XHarnessCommand.cs:line 145
XHarness exit code: 71 (GENERAL_FAILURE)

@ilonatommy
Copy link
Member

ilonatommy commented Jun 24, 2024

Waiting for the results of tests after the changes get propagated: #104017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-wasm WebAssembly architecture area-Infrastructure-mono disabled-test The test is disabled in source code against the issue os-browser Browser variant of arch-wasm
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants