merge with selenoid-master fork

alfa-laboratory · May 21, 2018 · 44d644a · 44d644a
2 parents 264469a + 95d8824
commit 44d644a
Show file tree

Hide file tree

Showing 34 changed files with 609 additions and 240 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,3 +3,4 @@
 selenoid
 vendor/*/
 coverage.*
+.DS_Store
diff --git a/Dockerfile b/Dockerfile
@@ -4,6 +4,4 @@ RUN apk add -U tzdata && rm -Rf /var/cache/apk/*
 COPY selenoid /usr/bin
 
 EXPOSE 4444
-ENTRYPOINT ["/usr/bin/selenoid"]
-
-CMD ["-listen", ":4444", "-conf", "/etc/selenoid/browsers.json", "-video-output-dir", "/opt/selenoid/video/"]
+ENTRYPOINT ["/usr/bin/selenoid", "-listen", ":4444", "-conf", "/etc/selenoid/browsers.json", "-video-output-dir", "/opt/selenoid/video/"]
diff --git a/Gopkg.lock b/Gopkg.lock
diff --git a/Gopkg.toml b/Gopkg.toml
@@ -25,9 +25,5 @@
   name = "github.com/aandryashin/matchers"
 
 [[constraint]]
-  name = "github.com/docker/docker"
-  revision = "92309e34e42aec3a0e041403bdd3c4fc0dc20269"
-
-[[override]]
-  name = "github.com/docker/distribution"
-  revision = "277ed486c948042cab91ad367c379524f3b25e18"
+  branch = "master"
+  name = "github.com/aerokube/util"
diff --git a/README.md b/README.md
@@ -14,12 +14,8 @@ Selenoid is a powerful implementation of [Selenium](http://github.com/SeleniumHQ
 ### One-command Installation
 Start browser automation in minutes by copy-pasting just **one command**:
 ```
-$ docker run --rm                                   \
-    -v /var/run/docker.sock:/var/run/docker.sock    \
-    -v ${HOME}:/root                                \
-    -e OVERRIDE_HOME=${HOME}                        \
-    aerokube/cm:latest-release selenoid start       \
-    --vnc --tmpfs 128
+$ curl -s https://aerokube.com/cm/bash | bash \
+    && ./cm selenoid start --vnc --tmpfs 128
 ```
 **That's it!** You can now use Selenoid instead of Selenium server. Specify the following Selenium URL in tests:
 ```
@@ -36,10 +32,14 @@ New images are added right after official releases. You can create your custom i
 New **[rich user interface]((https://github.com/aerokube/selenoid-ui))** showing browser screen and Selenium session logs:
 ![Selenoid UI](docs/img/selenoid-ui.png)
 
+### Video Recording
+* Any browser session can be saved to [H.264](https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC) video ([example](https://www.youtube.com/watch?v=maB298oO5cI))
+* An API to list, download and delete recorded video files
+
 ### Lightweight and Lightning Fast
 Suitable for personal usage and in big clusters:
 * Consumes **10 times** less memory than Java-based Selenium server under the same load
-* **Small 7 Mb binary** with no external dependencies (no need to install Java)
+* **Small 6 Mb binary** with no external dependencies (no need to install Java)
 * **Browser consumption API** working out of the box
 * Ability to send browser logs to **centralized log storage** (e.g. to the [ELK-stack](https://logz.io/learn/complete-guide-elk-stack/))
 * Fully **isolated** and **reproducible** environment
@@ -50,7 +50,13 @@ Maintained by a growing community:
 * Telegram [support channel](https://t.me/aerokube)
 * Support by [email](mailto:support@aerokube.com)
 * StackOverflow [tag](https://stackoverflow.com/questions/tagged/selenoid)
+* YouTube [channel](https://www.youtube.com/channel/UC9HvE3FNfTvftzpvXi9c69g)
 
 ## Complete Guide & Build Instructions
 
 Complete reference guide (including building instructions) can be found at: http://aerokube.com/selenoid/latest/
+
+## Known Users
+
+[![JetBrains](docs/img/logo/jetbrains.png)](http://jetbrains.com/) [![Yandex](docs/img/logo/yandex.png)](https://yandex.com/company/) [![Sberbank Technology](docs/img/logo/sbertech.png)](http://sber-tech.com/) [![ThoughtWorks](docs/img/logo/thoughtworks.png)](https://thoughtworks.com/) [![SuperJob](docs/img/logo/superjob.png)](http://superjob.ru/) [![PropellerAds](docs/img/logo/propellerads.png)](http://propellerads.com/) [![AlfaBank](docs/img/logo/alfabank.png)](https://alfabank.com/) [![3CX](docs/img/logo/3cx.png)](https://www.3cx.com/) [![IQ Option](docs/img/logo/iq_option.png)](https://iqoption.com/)
+
diff --git a/config/config.go b/config/config.go
@@ -59,6 +59,7 @@ type Browser struct {
 	Hosts   []string          `json:"hosts,omitempty"`
 	ShmSize int64             `json:"shmSize,omitempty"`
 	Labels  map[string]string `json:"labels,omitempty"`
+	Sysctl  map[string]string `json:"sysctl,omitempty"`
 }
 
 // Versions configuration

diff --git a/docs/browser-image-information.adoc b/docs/browser-image-information.adoc
@@ -64,7 +64,7 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 |===
 | Image | VNC Image | Selenoid Version | Geckodriver Version | Firefox Version | Client Version
 
-| selenoid/firefox:48.0 | selenoid/vnc:firefox_48.0 | 1.3.9 | 0.13.0 | 48.0.2 (page load timeout, native events and proxies don't work) .11+<.^|
+| selenoid/firefox:48.0 | selenoid/vnc:firefox_48.0 | 1.3.9 | 0.13.0 | 48.0.2 (page load timeout, native events and proxies don't work) .13+<.^|
 **Java, selenium-webdriver.js**: 3.4.0 and above
 **Python**: 3.5.0 and above 
 | selenoid/firefox:49.0 | selenoid/vnc:firefox_49.0 | 1.3.9 | 0.13.0 | 49.0.2 (page load timeout, native events and switching between windows don't work) 
@@ -76,7 +76,9 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 | selenoid/firefox:55.0 | selenoid/vnc:firefox_55.0 | 1.3.9 | 0.18.0 | 55.0.1 (switching windows may not work) 
 | selenoid/firefox:56.0 | selenoid/vnc:firefox_56.0 | 1.3.9 | 0.19.1 | 56.0.1 
 | selenoid/firefox:57.0 | selenoid/vnc:firefox_57.0 | 1.3.9 | 0.19.1 | 57.0 
-| selenoid/firefox:58.0 | selenoid/vnc:firefox_58.0 | 1.3.9 | 0.19.1 | 58.0
+| selenoid/firefox:58.0 | selenoid/vnc:firefox_58.0 | 1.6.0 | 0.20.1 | 58.0
+| selenoid/firefox:59.0 | selenoid/vnc:firefox_59.0 | 1.6.0 | 0.20.1 | 59.0.1
+| selenoid/firefox:60.0 | selenoid/vnc:firefox_60.0 | 1.6.0 | 0.20.1 | 60.0
 |===
 
 
@@ -103,6 +105,8 @@ WARNING: Firefox 53.0+ images require Selenium client 3.4.0 or newer.
 | selenoid/chrome:62.0 | selenoid/vnc:chrome_62.0 | 2.33 | 62.0.3202.62
 | selenoid/chrome:63.0 | selenoid/vnc:chrome_63.0 | 2.33 | 63.0.3239.84
 | selenoid/chrome:64.0 | selenoid/vnc:chrome_64.0 | 2.35 | 64.0.3282.119
+| selenoid/chrome:65.0 | selenoid/vnc:chrome_65.0 | 2.38 | 65.0.3325.181
+| selenoid/chrome:66.0 | selenoid/vnc:chrome_66.0 | 2.38 | 66.0.3359.117
 |===
 
 [NOTE]
@@ -152,6 +156,7 @@ We do not consider these images really stable. Many of base operations like work
 | selenoid/opera:49.0 | selenoid/vnc:opera_49.0 | 2.32 | 49.0.2725.39
 | selenoid/opera:50.0 | selenoid/vnc:opera_50.0 | 2.32 | 50.0.2762.45
 | selenoid/opera:51.0 | selenoid/vnc:opera_51.0 | 2.33 | 51.0.2830.26
+| selenoid/opera:52.0 | selenoid/vnc:opera_52.0 | 2.35 | 52.0.2871.37
 |===
 
 [NOTE]

diff --git a/docs/browsers-configuration-file.adoc b/docs/browsers-configuration-file.adoc
@@ -120,6 +120,7 @@ Selenoid proxies connections to either Selenium server or standalone driver bina
     "env" : ["TZ=Europe/Moscow", "ONE_MORE_VARIABLE=itsValue"],
     "hosts" : ["one.example.com:192.168.0.1", "two.example.com:192.168.0.2"],
     "labels" : {"component": "frontend", "project": "my-project"},
+    "sysctl" : {"net.ipv4.tcp_timestamps": "2", "kern.maxprocperuid": "1000"},
     "shmSize" : 268435456,
 },
 ----
@@ -145,6 +146,8 @@ We recommend to use our https://github.com/aerokube/cm[configuration tool] to av
 
 * *labels* (_optional_) - This field allows to add custom labels to running container. Specified as an object of `"key": "value"` pairs.
 
+* *sysctl* (_optional_) - This field allows to adjust kernel parameters of running container. Specified as an object of `"key": "value"` pairs.
+
 * *shmSize* (_optional_) - Use it to override shared memory size for browser container.
 
 === Syncing Browser Images from Existing File

diff --git a/docs/cli-flags.adoc b/docs/cli-flags.adoc
@@ -25,6 +25,8 @@ The following flags are supported by `selenoid` command:
     Network address to accept connections (default ":4444")
 -log-conf string
     Container logging configuration file (default "config/container-logs.json")
+-max-timeout duration
+    Maximum valid session idle timeout in time.Duration format (default 1h0m0s)
 -mem value
     Containers memory limit e.g. 128m or 1g
 -retry-count int

diff --git a/docs/faq.adoc b/docs/faq.adoc
@@ -0,0 +1,102 @@
+== Frequently Asked Questions
+
+=== Logs and Dirs
+
+**Where are Selenoid logs?**
+
+Selenoid outputs its logs to stdout. Selenoid launched as a binary should output logs to the screen. To see Selenoid logs launched as Docker container type:
+
+    $ docker logs selenoid
+
+To follow the logs add one more flag:
+
+    $ docker logs -f selenoid
+
+**Where are recorded videos stored?**
+
+Default location when installed with `cm` is `~/.aerokube/selenoid/video` or `C:\Users\<user>\.aerokube\selenoid\video`.
+
+=== Limits and Timeouts
+
+**How can I limit overall browsers consumption?**
+
+You have to use `-limit` flag to specify total number of parallel sessions. Default value is 5. See <<Resources Consumption>> section on how to determine total number of parallel sessions.
+
+**Can I limit per-version browser consumption?**
+
+No, this is not supported. We consider the only reasonable limitation should be the overall browsers consumption. This is important to not overload the hardware. 
+
+**How can I adjust Selenoid timeouts?**
+
+The main timeout flag is `-timeout`, specified as `60s` or `2m` or `1h`. It means maximum amount of time between subsequent HTTP requests to Selenium API. When there are no requests during this time period - session is automatically closed. Selenoid also has more subtle timeouts like:
+
+* `-service-startup-timeout` - container or driver process startup timeout
+* `-session-attempt-timeout` - new session HTTP request timeout, applied when container or driver has started
+* `-session-delete-timeout` - container or process removal timeout, applied after `driver.quit()` call
+
+=== Resources Consumption
+
+**How many resources browser containers consume?**
+
+This depends on your tests. We recommend to start with 1 CPU and 1 Gb of memory per container as a rough estimate and then increase `-limit` checking that your tests work stably.
+
+**Do VNC and non-VNC browser images memory and CPU consumption differ?**
+
+The only difference between these images - is a running VNC server (`x11vnc`) consuming approximately 20 Megabytes of RAM in idle state which is negligible compared to browser memory consumption.
+
+=== Features not Working
+
+**Selenoid does not start: open config/browsers.json: no such file or directory**
+
+This usually happens when Selenoid is started in Docker container with custom command-line arguments, e.g.:
+
+    $ docker run <some-args> aerokube/selenoid:some-version -limit 10
+
+In that case you have to specify path to configuration file explicitly (`cm` tool does this automatically):
+
+    $ docker run <some-args> aerokube/selenoid:some-version -limit 10 -conf /etc/selenoid/browsers.json 
+
+**Getting error: create container: Error response from daemon: client version 1.36 is too new**
+
+You have to run Selenoid binary \ container with `DOCKER_API_VERSION` variable specifying your Docker API version. `cm` tool does this automatically for you. To determine API version type:
+
+    $ docker version | grep API
+
+Then run Selenoid like the following:
+
+    $ DOCKER_API_VERSION=1.32 ./selenoid <rest-of-args> # As a binary
+    $ docker run -e DOCKER_API_VERSION=1.32 <rest-of-args> aerokube/selenoid:some-version # As Docker container
+
+**Video feature not working**
+
+When running Selenoid as Docker container video feature can be not working (because of misconfiguration). If your video files are named like `selenoid607667f7e1c7923779e35506b040300d.mp4` and you are seeing the following log message...
+```
+2018/03/20 21:06:37 [9] [VIDEO_ERROR] [Failed to rename /video/selenoid607667f7e1c7923779e35506b040300d.mp4 to /video/8019c4bc-9bec-4a8b-aa40-68d1db0cffd2.mp4: rename /video/selenoid607667f7e1c7923779e35506b040300d.mp4 /video/8019c4bc-9bec-4a8b-aa40-68d1db0cffd2.mp4: no such file or directory]
+```
+\... then check that:
+
+. You are passing an `OVERRIDE_VIDEO_OUTPUT_DIR` environment variable pointing to a directory on the `host machine` where video files are actually stored
+. When passing custom arguments to Selenoid container (such as `-limit` or `-timeout`) you also have to pass `-video-output-dir /opt/selenoid/video` and mount host machine video dir to `/opt/selenoid/video`
+
+**Can't open Selenoid video with Firefox**
+
+This is because we are using H264 codec which is not supported in Firefox for licensing reasons. Should work like a charm in Google Chrome or VLC player.
+
+**Can't get VNC feature to work: Disconnected**
+
+Please check the following:
+
+. You have `enableVNC = true` capability in your tests
+. You are using browser images with `vnc` in their name, e.g. `selenoid/vnc:firefox:58.0`.
+
+**Seeing black screen with a cross in VNC window**
+
+You are using `driver.close()` instead of `driver.quit()` and just closed the last browser tab instead of removing the session.
+
+**Can't maximize browser window**
+
+This is because of missing window manager in browser images. Should be fixed soon.
+
+**Can Selenoid pull browser images automatically?**
+
+No, we did not implement this feature intentionally. We consider that all such cluster maintenance tasks can influence performance and stability when done automatically.
diff --git a/docs/file-download.adoc b/docs/file-download.adoc
@@ -0,0 +1,13 @@
+== Accessing Files Downloaded with Browser
+
+[NOTE]
+====
+. This feature only works when browsers are run in containers.
+. Files are accessible only when browser session is running.
+====
+
+Your tests may need to download files with browsers. To analyze these files a common requirement is then to somehow extract downloaded files from browser containers. A possible solution can be dealing with container volumes. But Selenoid provides a `/download` API and dramatically simplifies downloading files. Having a running session `f2bcd32b-d932-4cdc-a639-687ab8e4f840` you can access all downloaded files using an URL:
+```
+http://selenoid-host.example.com:4444/download/f2bcd32b-d932-4cdc-a639-687ab8e4f840/myfile.txt
+```
+In order for this feature to work an HTTP file server should be listening inside browser container on port `8080`. Download directory inside container to be used in tests is usually `~/Downloads`.
diff --git a/docs/img/logo/3cx.png b/docs/img/logo/3cx.png
diff --git a/docs/img/logo/alfabank.png b/docs/img/logo/alfabank.png
diff --git a/docs/img/logo/iq_option.png b/docs/img/logo/iq_option.png
diff --git a/docs/img/logo/jetbrains.png b/docs/img/logo/jetbrains.png
diff --git a/docs/img/logo/propellerads.png b/docs/img/logo/propellerads.png
diff --git a/docs/img/logo/sbertech.png b/docs/img/logo/sbertech.png
diff --git a/docs/img/logo/superjob.png b/docs/img/logo/superjob.png
diff --git a/docs/img/logo/thoughtworks.png b/docs/img/logo/thoughtworks.png
diff --git a/docs/img/logo/yandex.png b/docs/img/logo/yandex.png
diff --git a/docs/index.adoc b/docs/index.adoc
@@ -17,6 +17,7 @@ It is using Docker to launch browsers. Please refer to https://github.com/aeroku
 
 == Getting Started
 include::quick-start-guide.adoc[leveloffset=+1]
+include::faq.adoc[leveloffset=+1]
 include::windows.adoc[leveloffset=+1]
 include::browser-images.adoc[leveloffset=+1]
 
@@ -37,6 +38,7 @@ include::special-capabilities.adoc[leveloffset=+1]
 include::selenoid-without-docker.adoc[leveloffset=+1]
 include::usage-statistics.adoc[leveloffset=+1]
 include::file-upload.adoc[leveloffset=+1]
+include::file-download.adoc[leveloffset=+1]
 
 include::contributing.adoc[]
 

diff --git a/docs/log-files.adoc b/docs/log-files.adoc
@@ -51,6 +51,7 @@ The following statuses are available:
 | CREATING_CONTAINER | Docker container with browser is creating 
 | DEFAULT_VERSION | Selenoid is using default browser version 
 | DELETED_VIDEO_FILE | Video file was deleted by user 
+| DOWNLOADING_FILE | User requested to download file from browser container 
 | ENVIRONMENT_NOT_AVAILABLE | Browser with desired name and version does not exist
 | FAILED_TO_REMOVE_CONTAINER | Failed to remove Docker container 
 | FAILED_TO_TERMINATE_PROCESS | An error occurred while terminating driver process