Merge pull request aerokube#21 in ALAT/mesos-selenoid from tests to m…

…aster * commit '44d644ac1d6bebfeeec23d496083629ad91d8d13': (45 commits) fix tests Added ability to override DNS servers list (fixes aerokube#294) Added ability to override session timeout (fixes aerokube#262) Fixed one-liner installation command (fixes aerokube#431) Added Firefox 60 image information Passing all browser environment variables to video container (related to aerokube/images#33) Updated Chromedriver version for Chrome 65 and 66 Added missing port entry for fileserver port (related to aerokube#424) Added Chrome 66 image information fix map example in the labels docs Reverted back Dockerfile Updated browser image information for latest Firefox versions Added iq option logo Fixed typo in extension capabilities docs (fixes aerokube#415) Added Opera 52.0 image information and update for Chrome 65.0 Added note about video recording Using map for Docker labels everywhere Bump Docker client version (fixes aerokube#404) Added tests for /vnc and /log API (related to aerokube#404) Added link to YouTube channel ...
alfa-laboratory · May 23, 2018 · 6c3039a · 6c3039a
2 parents b459322 + 44d644a
commit 6c3039a
Show file tree

Hide file tree

Showing 36 changed files with 614 additions and 245 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,3 +3,4 @@
 selenoid
 vendor/*/
 coverage.*
+.DS_Store
diff --git a/Dockerfile b/Dockerfile
@@ -4,6 +4,4 @@ RUN apk add -U tzdata && rm -Rf /var/cache/apk/*
 COPY selenoid /usr/bin
 
 EXPOSE 4444
-ENTRYPOINT ["/usr/bin/selenoid"]
-
-CMD ["-listen", ":4444", "-conf", "/etc/selenoid/browsers.json", "-video-output-dir", "/opt/selenoid/video/"]
+ENTRYPOINT ["/usr/bin/selenoid", "-listen", ":4444", "-conf", "/etc/selenoid/browsers.json", "-video-output-dir", "/opt/selenoid/video/"]
diff --git a/Gopkg.lock b/Gopkg.lock
diff --git a/Gopkg.toml b/Gopkg.toml
@@ -25,9 +25,5 @@
   name = "github.com/aandryashin/matchers"
 
 [[constraint]]
-  name = "github.com/docker/docker"
-  revision = "92309e34e42aec3a0e041403bdd3c4fc0dc20269"
-
-[[override]]
-  name = "github.com/docker/distribution"
-  revision = "277ed486c948042cab91ad367c379524f3b25e18"
+  branch = "master"
+  name = "github.com/aerokube/util"
diff --git a/README.md b/README.md
@@ -14,12 +14,8 @@ Selenoid is a powerful implementation of [Selenium](http://github.com/SeleniumHQ
 ### One-command Installation
 Start browser automation in minutes by copy-pasting just **one command**:
 ```
-$ docker run --rm                                   \
-    -v /var/run/docker.sock:/var/run/docker.sock    \
-    -v ${HOME}:/root                                \
-    -e OVERRIDE_HOME=${HOME}                        \
-    aerokube/cm:latest-release selenoid start       \
-    --vnc --tmpfs 128
+$ curl -s https://aerokube.com/cm/bash | bash \
+    && ./cm selenoid start --vnc --tmpfs 128
 ```
 **That's it!** You can now use Selenoid instead of Selenium server. Specify the following Selenium URL in tests:
 ```
@@ -36,10 +32,14 @@ New images are added right after official releases. You can create your custom i
 New **[rich user interface]((https://github.com/aerokube/selenoid-ui))** showing browser screen and Selenium session logs:
 ![Selenoid UI](docs/img/selenoid-ui.png)
 
+### Video Recording
+* Any browser session can be saved to [H.264](https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC) video ([example](https://www.youtube.com/watch?v=maB298oO5cI))
+* An API to list, download and delete recorded video files
+
 ### Lightweight and Lightning Fast
 Suitable for personal usage and in big clusters:
 * Consumes **10 times** less memory than Java-based Selenium server under the same load
-* **Small 7 Mb binary** with no external dependencies (no need to install Java)
+* **Small 6 Mb binary** with no external dependencies (no need to install Java)
 * **Browser consumption API** working out of the box
 * Ability to send browser logs to **centralized log storage** (e.g. to the [ELK-stack](https://logz.io/learn/complete-guide-elk-stack/))
 * Fully **isolated** and **reproducible** environment
@@ -50,7 +50,13 @@ Maintained by a growing community:
 * Telegram [support channel](https://t.me/aerokube)
 * Support by [email](mailto:support@aerokube.com)
 * StackOverflow [tag](https://stackoverflow.com/questions/tagged/selenoid)
+* YouTube [channel](https://www.youtube.com/channel/UC9HvE3FNfTvftzpvXi9c69g)
 
 ## Complete Guide & Build Instructions
 
 Complete reference guide (including building instructions) can be found at: http://aerokube.com/selenoid/latest/
+
+## Known Users
+
+[![JetBrains](docs/img/logo/jetbrains.png)](http://jetbrains.com/) [![Yandex](docs/img/logo/yandex.png)](https://yandex.com/company/) [![Sberbank Technology](docs/img/logo/sbertech.png)](http://sber-tech.com/) [![ThoughtWorks](docs/img/logo/thoughtworks.png)](https://thoughtworks.com/) [![SuperJob](docs/img/logo/superjob.png)](http://superjob.ru/) [![PropellerAds](docs/img/logo/propellerads.png)](http://propellerads.com/) [![AlfaBank](docs/img/logo/alfabank.png)](https://alfabank.com/) [![3CX](docs/img/logo/3cx.png)](https://www.3cx.com/) [![IQ Option](docs/img/logo/iq_option.png)](https://iqoption.com/)
+
diff --git a/config/config.go b/config/config.go
@@ -59,6 +59,7 @@ type Browser struct {
 	Hosts   []string          `json:"hosts,omitempty"`
 	ShmSize int64             `json:"shmSize,omitempty"`
 	Labels  map[string]string `json:"labels,omitempty"`
+	Sysctl  map[string]string `json:"sysctl,omitempty"`
 }
 
 // Versions configuration

diff --git a/docs/browser-image-information.adoc b/docs/browser-image-information.adoc
@@ -64,7 +64,7 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 |===
 | Image | VNC Image | Selenoid Version | Geckodriver Version | Firefox Version | Client Version
 
-| selenoid/firefox:48.0 | selenoid/vnc:firefox_48.0 | 1.3.9 | 0.13.0 | 48.0.2 (page load timeout, native events and proxies don't work) .11+<.^|
+| selenoid/firefox:48.0 | selenoid/vnc:firefox_48.0 | 1.3.9 | 0.13.0 | 48.0.2 (page load timeout, native events and proxies don't work) .13+<.^|
 **Java, selenium-webdriver.js**: 3.4.0 and above
 **Python**: 3.5.0 and above 
 | selenoid/firefox:49.0 | selenoid/vnc:firefox_49.0 | 1.3.9 | 0.13.0 | 49.0.2 (page load timeout, native events and switching between windows don't work) 
@@ -76,7 +76,9 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 | selenoid/firefox:55.0 | selenoid/vnc:firefox_55.0 | 1.3.9 | 0.18.0 | 55.0.1 (switching windows may not work) 
 | selenoid/firefox:56.0 | selenoid/vnc:firefox_56.0 | 1.3.9 | 0.19.1 | 56.0.1 
 | selenoid/firefox:57.0 | selenoid/vnc:firefox_57.0 | 1.3.9 | 0.19.1 | 57.0 
-| selenoid/firefox:58.0 | selenoid/vnc:firefox_58.0 | 1.3.9 | 0.19.1 | 58.0
+| selenoid/firefox:58.0 | selenoid/vnc:firefox_58.0 | 1.6.0 | 0.20.1 | 58.0
+| selenoid/firefox:59.0 | selenoid/vnc:firefox_59.0 | 1.6.0 | 0.20.1 | 59.0.1
+| selenoid/firefox:60.0 | selenoid/vnc:firefox_60.0 | 1.6.0 | 0.20.1 | 60.0
 |===
 
 
@@ -103,6 +105,8 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 | selenoid/chrome:62.0 | selenoid/vnc:chrome_62.0 | 2.33 | 62.0.3202.62
 | selenoid/chrome:63.0 | selenoid/vnc:chrome_63.0 | 2.33 | 63.0.3239.84
 | selenoid/chrome:64.0 | selenoid/vnc:chrome_64.0 | 2.35 | 64.0.3282.119
+| selenoid/chrome:65.0 | selenoid/vnc:chrome_65.0 | 2.38 | 65.0.3325.181
+| selenoid/chrome:66.0 | selenoid/vnc:chrome_66.0 | 2.38 | 66.0.3359.117
 |===
 
 [NOTE]
@@ -152,6 +156,7 @@ We do not consider these images really stable. Many of base operations like work
 | selenoid/opera:49.0 | selenoid/vnc:opera_49.0 | 2.32 | 49.0.2725.39
 | selenoid/opera:50.0 | selenoid/vnc:opera_50.0 | 2.32 | 50.0.2762.45
 | selenoid/opera:51.0 | selenoid/vnc:opera_51.0 | 2.33 | 51.0.2830.26
+| selenoid/opera:52.0 | selenoid/vnc:opera_52.0 | 2.35 | 52.0.2871.37
 |===
 
 [NOTE]

diff --git a/docs/browsers-configuration-file.adoc b/docs/browsers-configuration-file.adoc
@@ -120,6 +120,7 @@ Selenoid proxies connections to either Selenium server or standalone driver bina
     "env" : ["TZ=Europe/Moscow", "ONE_MORE_VARIABLE=itsValue"],
     "hosts" : ["one.example.com:192.168.0.1", "two.example.com:192.168.0.2"],
     "labels" : {"component": "frontend", "project": "my-project"},
+    "sysctl" : {"net.ipv4.tcp_timestamps": "2", "kern.maxprocperuid": "1000"},
     "shmSize" : 268435456,
 },
 ----
@@ -145,6 +146,8 @@ We recommend to use our https://github.com/aerokube/cm[configuration tool] to av
 
 * *labels* (_optional_) - This field allows to add custom labels to running container. Specified as an object of `"key": "value"` pairs.
 
+* *sysctl* (_optional_) - This field allows to adjust kernel parameters of running container. Specified as an object of `"key": "value"` pairs.
+
 * *shmSize* (_optional_) - Use it to override shared memory size for browser container.
 
 === Syncing Browser Images from Existing File

diff --git a/docs/cli-flags.adoc b/docs/cli-flags.adoc
@@ -25,6 +25,8 @@ The following flags are supported by `selenoid` command:
     Network address to accept connections (default ":4444")
 -log-conf string
     Container logging configuration file (default "config/container-logs.json")
+-max-timeout duration
+    Maximum valid session idle timeout in time.Duration format (default 1h0m0s)
 -mem value
     Containers memory limit e.g. 128m or 1g
 -retry-count int

diff --git a/docs/faq.adoc b/docs/faq.adoc
@@ -0,0 +1,102 @@
+== Frequently Asked Questions
+
+=== Logs and Dirs
+
+**Where are Selenoid logs?**
+
+Selenoid outputs its logs to stdout. Selenoid launched as a binary should output logs to the screen. To see Selenoid logs launched as Docker container type:
+
+    $ docker logs selenoid
+
+To follow the logs add one more flag:
+
+    $ docker logs -f selenoid
+
+**Where are recorded videos stored?**
+
+Default location when installed with `cm` is `~/.aerokube/selenoid/video` or `C:\Users\<user>\.aerokube\selenoid\video`.
+
+=== Limits and Timeouts
+
+**How can I limit overall browsers consumption?**
+
+You have to use `-limit` flag to specify total number of parallel sessions. Default value is 5. See <<Resources Consumption>> section on how to determine total number of parallel sessions.
+
+**Can I limit per-version browser consumption?**
+
+No, this is not supported. We consider the only reasonable limitation should be the overall browsers consumption. This is important to not overload the hardware. 
+
+**How can I adjust Selenoid timeouts?**
+
+The main timeout flag is `-timeout`, specified as `60s` or `2m` or `1h`. It means maximum amount of time between subsequent HTTP requests to Selenium API. When there are no requests during this time period - session is automatically closed. Selenoid also has more subtle timeouts like:
+
+* `-service-startup-timeout` - container or driver process startup timeout
+* `-session-attempt-timeout` - new session HTTP request timeout, applied when container or driver has started
+* `-session-delete-timeout` - container or process removal timeout, applied after `driver.quit()` call
+
+=== Resources Consumption
+
+**How many resources browser containers consume?**
+
+This depends on your tests. We recommend to start with 1 CPU and 1 Gb of memory per container as a rough estimate and then increase `-limit` checking that your tests work stably.
+
+**Do VNC and non-VNC browser images memory and CPU consumption differ?**
+
+The only difference between these images - is a running VNC server (`x11vnc`) consuming approximately 20 Megabytes of RAM in idle state which is negligible compared to browser memory consumption.
+
+=== Features not Working
+
+**Selenoid does not start: open config/browsers.json: no such file or directory**
+
+This usually happens when Selenoid is started in Docker container with custom command-line arguments, e.g.:
+
+    $ docker run <some-args> aerokube/selenoid:some-version -limit 10
+
+In that case you have to specify path to configuration file explicitly (`cm` tool does this automatically):
+
+    $ docker run <some-args> aerokube/selenoid:some-version -limit 10 -conf /etc/selenoid/browsers.json 
+
+**Getting error: create container: Error response from daemon: client version 1.36 is too new**
+
+You have to run Selenoid binary \ container with `DOCKER_API_VERSION` variable specifying your Docker API version. `cm` tool does this automatically for you. To determine API version type:
+
+    $ docker version | grep API
+
+Then run Selenoid like the following:
+
+    $ DOCKER_API_VERSION=1.32 ./selenoid <rest-of-args> # As a binary
+    $ docker run -e DOCKER_API_VERSION=1.32 <rest-of-args> aerokube/selenoid:some-version # As Docker container
+
+**Video feature not working**
+
+When running Selenoid as Docker container video feature can be not working (because of misconfiguration). If your video files are named like `selenoid607667f7e1c7923779e35506b040300d.mp4` and you are seeing the following log message...
+```
+2018/03/20 21:06:37 [9] [VIDEO_ERROR] [Failed to rename /video/selenoid607667f7e1c7923779e35506b040300d.mp4 to /video/8019c4bc-9bec-4a8b-aa40-68d1db0cffd2.mp4: rename /video/selenoid607667f7e1c7923779e35506b040300d.mp4 /video/8019c4bc-9bec-4a8b-aa40-68d1db0cffd2.mp4: no such file or directory]
+```
+\... then check that:
+
+. You are passing an `OVERRIDE_VIDEO_OUTPUT_DIR` environment variable pointing to a directory on the `host machine` where video files are actually stored
+. When passing custom arguments to Selenoid container (such as `-limit` or `-timeout`) you also have to pass `-video-output-dir /opt/selenoid/video` and mount host machine video dir to `/opt/selenoid/video`
+
+**Can't open Selenoid video with Firefox**
+
+This is because we are using H264 codec which is not supported in Firefox for licensing reasons. Should work like a charm in Google Chrome or VLC player.
+
+**Can't get VNC feature to work: Disconnected**
+
+Please check the following:
+
+. You have `enableVNC = true` capability in your tests
+. You are using browser images with `vnc` in their name, e.g. `selenoid/vnc:firefox:58.0`.
+
+**Seeing black screen with a cross in VNC window**
+
+You are using `driver.close()` instead of `driver.quit()` and just closed the last browser tab instead of removing the session.
+
+**Can't maximize browser window**
+
+This is because of missing window manager in browser images. Should be fixed soon.
+
+**Can Selenoid pull browser images automatically?**
+
+No, we did not implement this feature intentionally. We consider that all such cluster maintenance tasks can influence performance and stability when done automatically.
diff --git a/docs/file-download.adoc b/docs/file-download.adoc
@@ -0,0 +1,13 @@
+== Accessing Files Downloaded with Browser
+
+[NOTE]
+====
+. This feature only works when browsers are run in containers.
+. Files are accessible only when browser session is running.
+====
+
+Your tests may need to download files with browsers. To analyze these files a common requirement is then to somehow extract downloaded files from browser containers. A possible solution can be dealing with container volumes. But Selenoid provides a `/download` API and dramatically simplifies downloading files. Having a running session `f2bcd32b-d932-4cdc-a639-687ab8e4f840` you can access all downloaded files using an URL:
+```
+http://selenoid-host.example.com:4444/download/f2bcd32b-d932-4cdc-a639-687ab8e4f840/myfile.txt
+```
+In order for this feature to work an HTTP file server should be listening inside browser container on port `8080`. Download directory inside container to be used in tests is usually `~/Downloads`.
diff --git a/docs/img/logo/3cx.png b/docs/img/logo/3cx.png
diff --git a/docs/img/logo/alfabank.png b/docs/img/logo/alfabank.png
diff --git a/docs/img/logo/iq_option.png b/docs/img/logo/iq_option.png
diff --git a/docs/img/logo/jetbrains.png b/docs/img/logo/jetbrains.png
diff --git a/docs/img/logo/propellerads.png b/docs/img/logo/propellerads.png
diff --git a/docs/img/logo/sbertech.png b/docs/img/logo/sbertech.png
diff --git a/docs/img/logo/superjob.png b/docs/img/logo/superjob.png
diff --git a/docs/img/logo/thoughtworks.png b/docs/img/logo/thoughtworks.png
diff --git a/docs/img/logo/yandex.png b/docs/img/logo/yandex.png
diff --git a/docs/index.adoc b/docs/index.adoc
@@ -17,6 +17,7 @@ It is using Docker to launch browsers. Please refer to https://github.com/aeroku
 
 == Getting Started
 include::quick-start-guide.adoc[leveloffset=+1]
+include::faq.adoc[leveloffset=+1]
 include::windows.adoc[leveloffset=+1]
 include::browser-images.adoc[leveloffset=+1]
 
@@ -37,6 +38,7 @@ include::special-capabilities.adoc[leveloffset=+1]
 include::selenoid-without-docker.adoc[leveloffset=+1]
 include::usage-statistics.adoc[leveloffset=+1]
 include::file-upload.adoc[leveloffset=+1]
+include::file-download.adoc[leveloffset=+1]
 
 include::contributing.adoc[]
 

diff --git a/docs/log-files.adoc b/docs/log-files.adoc
@@ -51,6 +51,7 @@ The following statuses are available:
 | CREATING_CONTAINER | Docker container with browser is creating 
 | DEFAULT_VERSION | Selenoid is using default browser version 
 | DELETED_VIDEO_FILE | Video file was deleted by user 
+| DOWNLOADING_FILE | User requested to download file from browser container 
 | ENVIRONMENT_NOT_AVAILABLE | Browser with desired name and version does not exist
 | FAILED_TO_REMOVE_CONTAINER | Failed to remove Docker container 
 | FAILED_TO_TERMINATE_PROCESS | An error occurred while terminating driver process