Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout set to 10 minutes, but timing out after 3 minutes #9314

Closed
githubuser100007 opened this issue Mar 21, 2021 · 26 comments
Closed

Timeout set to 10 minutes, but timing out after 3 minutes #9314

githubuser100007 opened this issue Mar 21, 2021 · 26 comments

Comments

@githubuser100007
Copy link

githubuser100007 commented Mar 21, 2021

🐛 Bug Report

I am setting the timeout to 10 minutes because I am using https://github.com/scrapinghub/crawlera-headless-proxy

I am seeing it timeout after 3 minutes (still downloading all of the JS/CSS files)

To Reproduce

Set page load timeout to 10 minutes

Expected behavior

Expecting it to timeout after 10 minutes, not 3.

Test script or set of commands reproducing this issue

driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));

try {
driver.get(uRL.toExternalForm());
} catch (TimeoutException te2) {
log.warn("Chrome Driver thread timed out: " + uRL.toExternalForm());
}

Environment

OS: Windows
Browser: Chrome
Browser version: Version 89.0.4389.90 (Official Build) (64-bit)
Browser Driver version: ChromeDriver 89.0.4389.23
Language Bindings version: Java 8
Selenium Grid version (if applicable): Beta 2

@ghost ghost added the needs-triaging label Mar 21, 2021
@githubuser100007
Copy link
Author

Updated to reflect OS was Windows.

@githubuser100007
Copy link
Author

githubuser100007 commented Mar 24, 2021

This is what is being printed from Selenium Beta-2 (note the pageLoad and script timouts disregard my setting, and what is weird is that the outcome is timing out at 180000 ms, not the 600000 that was set, or the 300000 that is shown in these logs)

Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [472f80a2d4250c338c8fb9b8ade2c9be, getPageSource {}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 88.0.4324.96 (68dba2d8a0b14..., userDataDir: /tmp/.com.google.Chrome.374gW6}, goog:chromeOptions: {debuggerAddress:
localhost:39145}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, **timeouts: {implicit: 0, pageL
oad: 300000, script: 30000},** unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 472f80a2d4250c338c8fb9b8ade2c9be

@githubuser100007
Copy link
Author

githubuser100007 commented Mar 25, 2021

Here is more stack dump, again happens at 3 minutes instead of the 10 minute timeout I set:

org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.0.0-beta-2', revision: 'Unknown'
System info: host: 'DEV01', ip: '172.30.0.127', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_231'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [12c302ddd6915400e72f0497133acb7c, get {url=https://www.walmart.com/}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 89.0.4389.23 (61b08ee2c5002..., userDataDir: C:\Users\user\AppData\Loc...}, goog:chromeOptions: {debuggerAddress: localhost:55683}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(manual, http=tst05.de..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 12c302ddd6915400e72f0497133acb7c
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:71)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)
	at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
	at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
	at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:103)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:153)
	at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:128)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:620)
	at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:334)
	at com.util.SeleniumBrowser.browse(SeleniumBrowser.java:169)
Caused by: java.util.concurrent.TimeoutException
	at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
	at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
	at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)
	... 13 more

@diemol
Copy link
Member

diemol commented Mar 26, 2021

Could you please provide a complete code example to understand better what is happening? Code snippets are not enough in this case. Would be great to see how the driver is created and the url being loaded.

@githubuser100007
Copy link
Author

githubuser100007 commented Mar 26, 2021

Note that I use the webVitalsSnippet script injection on a driver without proxy and it works fine. So I don't think that has anything to do with it, but I included it here anyways, since that is what is happening.

    private RemoteWebDriver createDriver() throws Exception {
    	
		ChromiumDriver driver = null;

    	ChromeOptions options = new ChromeOptions();
		options.setBinary("/usr/bin/chrome"); 

		//remove unexepectedalertexceptions
		options.addArguments("--silent", "--headless", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");  
		options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);

		// no downloads
		Map<String, Object> chromePrefs = new HashMap<String, Object>();
		chromePrefs.put("download_restrictions", 3);
		options.setExperimentalOption("prefs", chromePrefs);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
        options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
        options.setCapability( "goog:loggingPrefs", logPrefs );
        options.setPageLoadStrategy(PageLoadStrategy.EAGER);

		DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withSilent(true);
		ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build(); 
		chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

		//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
		Proxy proxy = new Proxy(); 
        proxy.setHttpProxy("127.0.0.1:3128"); 
        proxy.setSslProxy("127.0.0.1:3128"); 
        options.setCapability(CapabilityType.PROXY, proxy); 

		driver = new ChromeDriver(chromeDriverService, options);

		// Inject Core Web Vitals scripts
		Map<String, Object> parameters = new HashMap<String, Object>();
		parameters.put("source",webVitalsSnippet);
		driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

		// Stop any unwanted downloading of *.pdf *.crdownload et al. files
		parameters = new HashMap<String, Object>();
		parameters.put("behavior","deny");
		driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

    	driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
    	driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(5000));

		return driver;

    }
	public static void main(String[] args) throws Exception {

		RemoteWebDriver driver = createDriver();

		long timer = System.currentTimeMillis();
		try {
			driver.get("https://www.walmart.com");
		} catch (TimeoutException te2) {
			log.warn("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
		} finally {
			driver.quit();
		}
		
	}

webVitalsSnippet:

var vitalsCLS = 'unset';
function logCLS({value}) {
  	vitalsCLS = value.toString();
  	console.log('CLS: ' + vitalsCLS);
}

var vitalsFID = 'unset';
function logFID({value}) {
  	vitalsFID = value.toString();
  	console.log('FID: ' + vitalsFID);
}

var vitalsLCP = 'unset';
function logLCP({value}) {
  	vitalsLCP = value.toString();
  	console.log('LCP: ' + vitalsLCP);
}

window.onload = function() {
	
	var script = document.createElement('script');
	script.src = 'https://unpkg.com/web-vitals';
  	script.onload = function() {
  	
    // When loading `web-vitals` using a classic script, all the public
    // methods can be found on the `webVitals` global namespace.
	webVitals.getCLS(logCLS, true); 
    webVitals.getFID(logFID, true); 
    webVitals.getLCP(logLCP, true); 
  
  }

  document.body.appendChild(script);

} 

@pujagani
Copy link
Contributor

pujagani commented Mar 30, 2021

Thank you for sharing the script and the stack trace. In my attempt to recreate the issue, I have been running the same example that is shared with a few tweaks (commenting out the proxy bit since it was mentioned in the follow-up comment that it is not a concern here).

public static void main(String[] args) throws Exception {

    SetTimeout setTimeout = new SetTimeout();

    RemoteWebDriver driver = setTimeout.createDriver();

    long timer = System.currentTimeMillis();
    try {
        driver.get("https://www.walmart.com/");
        System.out.println(driver.getTitle());
    } catch (TimeoutException te2) {
        System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
    } finally {
        driver.quit();
    }

}

private RemoteWebDriver createDriver() throws Exception {

    ChromiumDriver driver = null;

    ChromeOptions options = new ChromeOptions();
   // options.setBinary("/usr/bin/chrome");

    //remove unexepectedalertexceptions
    options.addArguments("--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
    options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);

    // no downloads
    Map<String, Object> chromePrefs = new HashMap<String, Object>();
    chromePrefs.put("download_restrictions", 3);
    options.setExperimentalOption("prefs", chromePrefs);

    LoggingPreferences logPrefs = new LoggingPreferences();
    logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
    options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
    options.setCapability( "goog:loggingPrefs", logPrefs );
    options.setPageLoadStrategy(PageLoadStrategy.EAGER);

    DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withLogLevel(ChromeDriverLogLevel.INFO);
    ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build();
    chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

    //running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
//        Proxy proxy = new Proxy();
//        proxy.setHttpProxy("127.0.0.1:3128");
//        proxy.setSslProxy("127.0.0.1:3128");
//        options.setCapability(CapabilityType.PROXY, proxy);

    driver = new ChromeDriver(chromeDriverService, options);

     //Inject Core Web Vitals scripts
//        Map<String, Object> parameters = new HashMap<String, Object>();
//        parameters.put("source", "var vitalsCLS = 'unset';\n" +
//                "function logCLS({value}) {\n" +
//                "  \tvitalsCLS = value.toString();\n" +
//                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsFID = 'unset';\n" +
//                "function logFID({value}) {\n" +
//                "  \tvitalsFID = value.toString();\n" +
//                "  \tconsole.log('FID: ' + vitalsFID);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsLCP = 'unset';\n" +
//                "function logLCP({value}) {\n" +
//                "  \tvitalsLCP = value.toString();\n" +
//                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
//                "}\n" +
//                "\n" +
//                "window.onload = function() {\n" +
//                "\t\n" +
//                "\tvar script = document.createElement('script');\n" +
//                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
//                "  \tscript.onload = function() {\n" +
//                "  \t\n" +
//                "    // When loading `web-vitals` using a classic script, all the public\n" +
//                "    // methods can be found on the `webVitals` global namespace.\n" +
//                "\twebVitals.getCLS(logCLS, true); \n" +
//                "    webVitals.getFID(logFID, true); \n" +
//                "    webVitals.getLCP(logLCP, true); \n" +
//                "  \n" +
//                "  }\n" +
//                "\n" +
//                "  document.body.appendChild(script);\n" +
//                "\n" +
//                "} ");
//        driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

     //Stop any unwanted downloading of *.pdf *.crdownload et al. files
//        parameters = new HashMap<String, Object>();
//        parameters.put("behavior","deny");
//        driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

    WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
    System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
    WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
    System.out.println(scriptTimeout.getScriptTimeout().toMillis());

    return driver;

} 

Logs:

Mar 30, 2021 7:39:27 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:39:28 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
600000
600000
Verify your identity

When the page load timeout is set to 2 milliseconds as shown below :

WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(2));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());

Logs:

Mar 30, 2021 7:46:25 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:46:25 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
2
600000
Chrome Driver thread timed out: 36

I have run the same two code samples and checked the debug logs and the timeout values are set as expected with the Chrome driver.

@githubuser100007
Copy link
Author

githubuser100007 commented Mar 30, 2021

An early timeout is not the problem. It's when you set it to 10 minutes that is the problem.

I'm wondering if the chromedriver doesn't support changing its timeouts, so anything earlier than it's default of 3 minutes is okay but anything after 3 minutes the chrome driver times out no matter what at 3 minutes.

@pujagani
Copy link
Contributor

pujagani commented Mar 31, 2021

As Diego rightly pointed out, the issue seems to be due to the default values of HTTP client config. The read timeout defaults to 3 mins. Hence as you mentioned, the issue springs up when the timeout is set higher than 3 mins.
The issue is not with the Chrome driver since Chrome driver debug logs show the timeout being set correctly and when the value is set less than 3 mins the value is considered.
The solution for this is to configure the client timeout when creating the driver using the RemoteWebDriverBuilder.

private RemoteWebDriver createDriver() throws Exception {

  RemoteWebDriver driver = null;

  ChromeOptions options = new ChromeOptions();
  // options.setBinary("/usr/bin/chrome");

  //remove unexepectedalertexceptions
  options.addArguments("--unexpectedAlertBehaviour=dismiss","--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");

  // no downloads
  Map<String, Object> chromePrefs = new HashMap<String, Object>();
  chromePrefs.put("download_restrictions", 3);
  options.setExperimentalOption("prefs", chromePrefs);

  LoggingPreferences logPrefs = new LoggingPreferences();
  logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
  //options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
  options.setCapability("goog:loggingPrefs", logPrefs );
  options.setPageLoadStrategy(PageLoadStrategy.EAGER);

  DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
  ChromeDriverService chromeDriverService = (ChromeDriverService) serviceBuilder.build();
  chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

  //running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
  Proxy proxy = new Proxy();
  proxy.setHttpProxy("127.0.0.1:3128");
  proxy.setSslProxy("127.0.0.1:3128");
  //options.setProxy(proxy);

  //driver = new ChromeDriver(chromeDriverService, options);

  driver = (RemoteWebDriver) RemoteWebDriver
          .builder()
          .config(ClientConfig.defaultConfig()
                  .connectionTimeout(Duration.ofMinutes(10))
                  .readTimeout(Duration.ofMinutes(10)))
          .withDriverService(chromeDriverService)
          .addAlternative(options)
          .setCapability("proxy", proxy)
          .build();

   //Inject Core Web Vitals scripts
//        Map<String, Object> parameters = new HashMap<String, Object>();
//        parameters.put("source", "var vitalsCLS = 'unset';\n" +
//                "function logCLS({value}) {\n" +
//                "  \tvitalsCLS = value.toString();\n" +
//                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsFID = 'unset';\n" +
//                "function logFID({value}) {\n" +
//                "  \tvitalsFID = value.toString();\n" +
//                "  \tconsole.log('FID: ' + vitalsFID);\n" +
//                "}\n" +
//                "\n" +
//                "var vitalsLCP = 'unset';\n" +
//                "function logLCP({value}) {\n" +
//                "  \tvitalsLCP = value.toString();\n" +
//                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
//                "}\n" +
//                "\n" +
//                "window.onload = function() {\n" +
//                "\t\n" +
//                "\tvar script = document.createElement('script');\n" +
//                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
//                "  \tscript.onload = function() {\n" +
//                "  \t\n" +
//                "    // When loading `web-vitals` using a classic script, all the public\n" +
//                "    // methods can be found on the `webVitals` global namespace.\n" +
//                "\twebVitals.getCLS(logCLS, true); \n" +
//                "    webVitals.getFID(logFID, true); \n" +
//                "    webVitals.getLCP(logLCP, true); \n" +
//                "  \n" +
//                "  }\n" +
//                "\n" +
//                "  document.body.appendChild(script);\n" +
//                "\n" +
//                "} ");
//        driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

   //Stop any unwanted downloading of *.pdf *.crdownload et al. files
//        parameters = new HashMap<String, Object>();
//        parameters.put("behavior","deny");
//        driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

  WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
  System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
  WebDriver.Timeouts scriptTimeout  = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
  System.out.println(scriptTimeout.getScriptTimeout().toMillis());

  return driver;

}

Can you please try as shown above and see if changing the client config values helps solve the issue you are facing? Thank you! Please provide feedback on the same.

@githubuser100007
Copy link
Author

I'm getting the following error when building the driver the new way:

java.lang.IllegalArgumentException: Illegal key values seen in w3c capabilities: [loggingPrefs, unexpectedAlertBehaviour]
at org.openqa.selenium.remote.NewSessionPayload.lambda$validate$5(NewSessionPayload.java:203)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:372)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.openqa.selenium.remote.NewSessionPayload.validate(NewSessionPayload.java:207)
at org.openqa.selenium.remote.NewSessionPayload.(NewSessionPayload.java:152)
at org.openqa.selenium.remote.NewSessionPayload.create(NewSessionPayload.java:99)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.getPayload(RemoteWebDriverBuilder.java:465)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.getRemoteDriver(RemoteWebDriverBuilder.java:362)
at org.openqa.selenium.remote.RemoteWebDriverBuilder.build(RemoteWebDriverBuilder.java:339)

@diemol
Copy link
Member

diemol commented Apr 1, 2021

The W3C valid capabilities are shown here https://www.w3.org/TR/webdriver1/#capabilities

loggingPrefs and unexpectedAlertBehaviour are not, hence the error message.

@githubuser100007
Copy link
Author

Removed those capabilities and now getting an error trying to execute two of the critical settings on the driver for us:

		// Inject Core Web Vitals scripts
		Map<String, Object> parameters = new HashMap<String, Object>();
		parameters.put("source",webVitalsSnippet);
		((ChromiumDriver)driver).executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);

		// Stop any unwanted downloading of *.pdf *.crdownload et al. files
		parameters = new HashMap<String, Object>();
		parameters.put("behavior","deny");
		((ChromiumDriver)driver).executeCdpCommand("Browser.setDownloadBehavior", parameters);

Error:

java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebDriver cannot be cast to org.openqa.selenium.chromium.ChromiumDriver

@diemol
Copy link
Member

diemol commented Apr 1, 2021

Please use as a base the example provided above by @pujagani, which works. Otherwise this issue will become a debugging session and for that it would be preferrable for you to join our Slack/IRC channel https://www.selenium.dev/support/

@githubuser100007
Copy link
Author

Is there no other way to issue CDP requests? This is a prerequisite for us.

@diemol
Copy link
Member

diemol commented Apr 1, 2021

I understand the issue is about the 3 minutes timeout, which is triggered since it is the default one. We provided an example above that shows how to increase the timeout. Am I missing something?

@githubuser100007
Copy link
Author

Removing functionality to increase the timeout does not equal a workaround. Is this related to: #7914

@pujagani
Copy link
Contributor

pujagani commented Apr 5, 2021

Thank you for providing the feedback. I received the same error while working with it, but to ensure the options are set as desired, "options.addArguments("--unexpectedAlertBehaviour=dismiss" and "options.setCapability("goog:loggingPrefs", logPrefs); was added in the previous example.
ChromeDriver 75.0.3770.8 onwards the http://chromedriver.chromium.org/downloads states that it has "Renamed capability loggingPrefs to goog:loggingPrefs, as required by W3C standard". So both are covered.

Now the next concern is valid regarding the CDP commands. I am sharing an example below which a workaround to set a client config as well as run the CDP commands. Since currently, I was not able to figure a direct way to do both in a simpler manner. (Will discuss this with the Selenium contributors for future implementations)


public class SetTimeout {

    public static void main(String[] args) throws Exception {

        SetTimeout setTimeout = new SetTimeout();

        RemoteWebDriver driver = setTimeout.createDriver();

        long timer = System.currentTimeMillis();
        try {
            driver.get("https://www.walmart.com/");
            System.out.println(driver.getTitle());
        } catch (TimeoutException te2) {
            System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer)); // prints approximately 3 minutes, no matter what
        } finally {
            driver.quit();
        }

    }

    private RemoteWebDriver createDriver() throws Exception {

        RemoteWebDriver driver = null;

        ChromeOptions options = new ChromeOptions();

        //remove unexepectedalertexceptions
        options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");

        // no downloads
        Map<String, Object> chromePrefs = new HashMap<String, Object>();
        chromePrefs.put("download_restrictions", 3);
        options.setExperimentalOption("prefs", chromePrefs);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
        options.setCapability("goog:loggingPrefs", logPrefs);
        options.setPageLoadStrategy(PageLoadStrategy.EAGER);


        DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
        ChromeDriverService chromeDriverService = (ChromeDriverService) serviceBuilder.build();
        chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
        chromeDriverService.start();

        //running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
        //  Proxy proxy = new Proxy();
        //  proxy.setHttpProxy("127.0.0.1:3128");
        //  proxy.setSslProxy("127.0.0.1:3128");
        //options.setProxy(proxy);

        ClientConfig config = ClientConfig.defaultConfig().connectionTimeout(Duration.ofMinutes(10))
                .readTimeout(Duration.ofMinutes(10)).baseUrl(chromeDriverService.getUrl());
        Tracer tracer = OpenTelemetryTracer.getInstance();
        CommandExecutor executor = new HttpCommandExecutor(
                buildChromiumCommandMappings("goog"),
                config,
                new TracedHttpClient.Factory(tracer, HttpClient.Factory.createDefault()));

        driver = new RemoteWebDriver(executor, options);

        //Inject Core Web Vitals scripts
        Map<String, Object> parameters = new HashMap<String, Object>();
        parameters.put("source", "var vitalsCLS = 'unset';\n" +
                "function logCLS({value}) {\n" +
                "  \tvitalsCLS = value.toString();\n" +
                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
                "}\n" +
                "\n" +
                "var vitalsFID = 'unset';\n" +
                "function logFID({value}) {\n" +
                "  \tvitalsFID = value.toString();\n" +
                "  \tconsole.log('FID: ' + vitalsFID);\n" +
                "}\n" +
                "\n" +
                "var vitalsLCP = 'unset';\n" +
                "function logLCP({value}) {\n" +
                "  \tvitalsLCP = value.toString();\n" +
                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
                "}\n" +
                "\n" +
                "window.onload = function() {\n" +
                "\t\n" +
                "\tvar script = document.createElement('script');\n" +
                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
                "  \tscript.onload = function() {\n" +
                "  \t\n" +
                "    // When loading `web-vitals` using a classic script, all the public\n" +
                "    // methods can be found on the `webVitals` global namespace.\n" +
                "\twebVitals.getCLS(logCLS, true); \n" +
                "    webVitals.getFID(logFID, true); \n" +
                "    webVitals.getLCP(logLCP, true); \n" +
                "  \n" +
                "  }\n" +
                "\n" +
                "  document.body.appendChild(script);\n" +
                "\n" +
                "} ");

        Command addScriptCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Page.addScriptToEvaluateOnNewDocument", "params", parameters));
        driver.getCommandExecutor().execute(addScriptCommand);

        //Stop any unwanted downloading of *.pdf *.crdownload et al. files
        parameters = new HashMap<String, Object>();
        parameters.put("behavior", "deny");

        Command setDownloadBehaviorCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Browser.setDownloadBehavior", "params", parameters));
        driver.getCommandExecutor().execute(setDownloadBehaviorCommand);
        // driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);

        WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
        System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
        WebDriver.Timeouts scriptTimeout = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
        System.out.println(scriptTimeout.getScriptTimeout().toMillis());

        return driver;

    }

    private Map<String, CommandInfo> buildChromiumCommandMappings(String vendorKeyword) {
        String sessionPrefix = "/session/:sessionId/";
        String chromiumPrefix = sessionPrefix + "chromium";
        String vendorPrefix = sessionPrefix + vendorKeyword;

        HashMap<String, CommandInfo> mappings = new HashMap<>();

        mappings.put(ChromiumDriverCommand.LAUNCH_APP,
                new CommandInfo(chromiumPrefix + "/launch_app", HttpMethod.POST));

        String networkConditions = chromiumPrefix + "/network_conditions";
        mappings.put(ChromiumDriverCommand.GET_NETWORK_CONDITIONS,
                new CommandInfo(networkConditions, HttpMethod.GET));
        mappings.put(ChromiumDriverCommand.SET_NETWORK_CONDITIONS,
                new CommandInfo(networkConditions, HttpMethod.POST));
        mappings.put(ChromiumDriverCommand.DELETE_NETWORK_CONDITIONS,
                new CommandInfo(networkConditions, HttpMethod.DELETE));

        mappings.put(ChromiumDriverCommand.EXECUTE_CDP_COMMAND,
                new CommandInfo(vendorPrefix + "/cdp/execute", HttpMethod.POST));

        // Cast / Media Router APIs
        String cast = vendorPrefix + "/cast";
        mappings.put(ChromiumDriverCommand.GET_CAST_SINKS,
                new CommandInfo(cast + "/get_sinks", HttpMethod.GET));
        mappings.put(ChromiumDriverCommand.SET_CAST_SINK_TO_USE,
                new CommandInfo(cast + "/set_sink_to_use", HttpMethod.POST));
        mappings.put(ChromiumDriverCommand.START_CAST_TAB_MIRRORING,
                new CommandInfo(cast + "/start_tab_mirroring", HttpMethod.POST));
        mappings.put(ChromiumDriverCommand.GET_CAST_ISSUE_MESSAGE,
                new CommandInfo(cast + "/get_issue_message", HttpMethod.GET));
        mappings.put(ChromiumDriverCommand.STOP_CASTING,
                new CommandInfo(cast + "/stop_casting", HttpMethod.POST));

        mappings.put(ChromiumDriverCommand.SET_PERMISSION,
                new CommandInfo(sessionPrefix + "/permissions", HttpMethod.POST));

        return unmodifiableMap(mappings);
    }

    static final class ChromiumDriverCommand {
        private ChromiumDriverCommand() {
        }

        static final String LAUNCH_APP = "launchApp";
        static final String GET_NETWORK_CONDITIONS = "getNetworkConditions";
        static final String SET_NETWORK_CONDITIONS = "setNetworkConditions";
        static final String DELETE_NETWORK_CONDITIONS = "deleteNetworkConditions";
        static final String EXECUTE_CDP_COMMAND = "executeCdpCommand";

        // Cast Media Router APIs
        static final String GET_CAST_SINKS = "getCastSinks";
        static final String SET_CAST_SINK_TO_USE = "selectCastSink";
        static final String START_CAST_TAB_MIRRORING = "startCastTabMirroring";
        static final String GET_CAST_ISSUE_MESSAGE = "getCastIssueMessage";
        static final String STOP_CASTING = "stopCasting";

        static final String SET_PERMISSION = "setPermission";
    }

}

I hope this helps fix the issue.

@githubuser100007
Copy link
Author

I get an error running this code. See below for logs.

10 May 2021 14:23:58,474 ERROR [Thread-1] (WebdriverProxyFactory.java:54) - Unable to create driver.
org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
Build info: version: '4.0.0-beta-2', revision: 'Unknown'
System info: host: 'DEV01', ip: '172.30.0.127', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_231'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [null, newSession {desiredCapabilities=Capabilities {browserName: chrome, goog:loggingPrefs: org.openqa.selenium.logging..., pageLoadStrategy: eager, proxy: Proxy(manual, http=tst05.de...}}]
Capabilities {}
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:638)
	at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:252)
	at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:174)
Caused by: java.io.UncheckedIOException: java.net.ConnectException: Connection refused: no further information: localhost/0:0:0:0:0:0:0:1:10560

@pujagani
Copy link
Contributor

Thank you for providing an update. However, with the same example after multiple runs, I am not able to reproduce this error.
Can you provide some more details, please? Please also provide the code snippet used to reproduce the issue.

@githubuser100007
Copy link
Author

githubuser100007 commented May 11, 2021

Here is the code that I am running. Please let me know if I've somehow missed something.

    	RemoteWebDriver driver = null;

    	ChromeOptions options = new ChromeOptions();
		options.setBinary("/usr/bin/chrome"); 

		//remove unexepectedalertexceptions
		options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--headless", "--blink-settings=imagesEnabled=false", "--disable-blink-features", 
				"--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");  

		// no downloads
		Map<String, Object> chromePrefs = new HashMap<String, Object>();
		chromePrefs.put("download_restrictions", 3);
		chromePrefs.put("download.default_directory", "/dev/null");
		
		options.setExperimentalOption("prefs", chromePrefs);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
        options.setCapability( "goog:loggingPrefs", logPrefs );
        options.setPageLoadStrategy(PageLoadStrategy.EAGER);

		DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withSilent(true);
		ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build(); 
		chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));

		ClientConfig config = ClientConfig.defaultConfig().connectionTimeout(Duration.ofMinutes(10))
                .readTimeout(Duration.ofMinutes(10)).baseUrl(chromeDriverService.getUrl());
        Tracer tracer = OpenTelemetryTracer.getInstance();
        CommandExecutor executor = new HttpCommandExecutor(
                buildChromiumCommandMappings("goog"),
                config,
                new TracedHttpClient.Factory(tracer, HttpClient.Factory.createDefault()));

        driver = new RemoteWebDriver(executor, options);
        
      //Inject Core Web Vitals scripts
        Map<String, Object> parameters = new HashMap<String, Object>();
        parameters.put("source", "var vitalsCLS = 'unset';\n" +
                "function logCLS({value}) {\n" +
                "  \tvitalsCLS = value.toString();\n" +
                "  \tconsole.log('CLS: ' + vitalsCLS);\n" +
                "}\n" +
                "\n" +
                "var vitalsFID = 'unset';\n" +
                "function logFID({value}) {\n" +
                "  \tvitalsFID = value.toString();\n" +
                "  \tconsole.log('FID: ' + vitalsFID);\n" +
                "}\n" +
                "\n" +
                "var vitalsLCP = 'unset';\n" +
                "function logLCP({value}) {\n" +
                "  \tvitalsLCP = value.toString();\n" +
                "  \tconsole.log('LCP: ' + vitalsLCP);\n" +
                "}\n" +
                "\n" +
                "window.onload = function() {\n" +
                "\t\n" +
                "\tvar script = document.createElement('script');\n" +
                "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
                "  \tscript.onload = function() {\n" +
                "  \t\n" +
                "    // When loading `web-vitals` using a classic script, all the public\n" +
                "    // methods can be found on the `webVitals` global namespace.\n" +
                "\twebVitals.getCLS(logCLS, true); \n" +
                "    webVitals.getFID(logFID, true); \n" +
                "    webVitals.getLCP(logLCP, true); \n" +
                "  \n" +
                "  }\n" +
                "\n" +
                "  document.body.appendChild(script);\n" +
                "\n" +
                "} ");

        Command addScriptCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Page.addScriptToEvaluateOnNewDocument", "params", parameters));
        driver.getCommandExecutor().execute(addScriptCommand);

        //Stop any unwanted downloading of *.pdf *.crdownload et al. files
        parameters = new HashMap<String, Object>();
        parameters.put("behavior", "deny");

        Command setDownloadBehaviorCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Browser.setDownloadBehavior", "params", parameters));
        driver.getCommandExecutor().execute(setDownloadBehaviorCommand);

    	driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
    	driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(5000));

The error occurs here:

driver = new RemoteWebDriver(executor, options);

@pujagani
Copy link
Contributor

Thank you for sharing the code snippet.
You have missed adding chromeDriverService.start(); as shared earlier in the code example as a fix for this issue.
Since the service is not started and RemoteWebDriver is trying to use its URL, the java.net.ConnectException: Connection refused is thrown.
Please add that after chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null")); and give it a try.

@githubuser100007
Copy link
Author

That works! Thanks for the workaround.

@githubuser100007
Copy link
Author

Reopening in case you want to fix the original issue.

@githubuser100007
Copy link
Author

Got a bunch of Too many open files errors, do I have to close the service separately from the driver?

@diemol
Copy link
Member

diemol commented May 11, 2021

Please make sure to be using the most recent releases (beta-3 is the most recent).

I believe you have been able to move forward and I'd like to avoid having this issue turned into a debugging session. If there is a concrete issue with code showing how to reproduce it, please open a new issue.

Please join our Slack/IRC channel for more questions, we'd be happy to reply there. https://www.selenium.dev/support/

@diemol diemol closed this as completed May 11, 2021
@githubuser100007
Copy link
Author

githubuser100007 commented May 11, 2021

I understand. However, after running several large scale tests after the initial unit test, it appears that the linux machine runs out of sockets. So I am trying to confirm that the chromedriverservice.start() call doesn't need to be closed when the driver is closed (this is used in the context of a connection pool), which was not a requirement before in our original code.

Does driver.close handle the chromedriverservice.close() call?

@pujagani
Copy link
Contributor

Thank you for the feedback and the details. Probably this will help, quit the RemoteWebDriver then close the ChromeDriver service. I stepped through the code to confirm that RemoteWebDriver quit will not close the ChromeDriver service. Thanks for bringing this to our notice.

public class SetTimeout {

    ChromeDriverService chromeDriverService;

    public static void main(String[] args) throws Exception {

        SetTimeout setTimeout = new SetTimeout();

        RemoteWebDriver driver = setTimeout.createDriver();

        long timer = System.currentTimeMillis();
        try {
            driver.get("https://www.walmart.com/");
            System.out.println(driver.getTitle());
        } catch (TimeoutException te2) {
            System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer)); // prints approximately 3 minutes, no matter what
        } finally {
            driver.quit();
            setTimeout.getChromeDriverService().stop();
        }

    }

    public ChromeDriverService getChromeDriverService() {
        return chromeDriverService;
    }


    private RemoteWebDriver createDriver() throws Exception {

        RemoteWebDriver driver = null;

        ChromeOptions options = new ChromeOptions();

        //remove unexepectedalertexceptions
        options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");

        // no downloads
        Map<String, Object> chromePrefs = new HashMap<String, Object>();
        chromePrefs.put("download_restrictions", 3);
        options.setExperimentalOption("prefs", chromePrefs);

        LoggingPreferences logPrefs = new LoggingPreferences();
        logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
        options.setCapability("goog:loggingPrefs", logPrefs);
        options.setPageLoadStrategy(PageLoadStrategy.EAGER);


        DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
        chromeDriverService = (ChromeDriverService) serviceBuilder.build();
        chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
        chromeDriverService.start();

// Rest of the code is same as shared earlier

@github-actions github-actions bot locked and limited conversation to collaborators Sep 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants