learn-by-example-reinforcement-learning-gym-setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "c060571f57aa8f9209c2155301b927239936899d"
   },
   "source": [
    "# Setup Gym with Atari games on Kaggle\n",
    "Welcome to my notebook on Kaggle. I did record my notes so it might help others in their journey to understand Neural Networks by examples (in this case Reinforcement Learning with Gym for Atari games from OpenAI). \n",
    "\n",
    "This notebook is to show how to set-up gym including the gym[atari] environments in Kaggle so you can get to work and try to beat all those Atari 2600 games like Pong, Pacman, Spaceinvaders, etc\n",
    "\n",
    "If you are interested in Reinforcement Learning you might be interested in some of my other notebooks:  \n",
    "https://www.kaggle.com/charel/learn-by-example-reinforcement-learning-with-gym  \n",
    "If you are new to Neural Networks you might want to have a look at my notebooks below:    \n",
    "https://www.kaggle.com/charel/learn-neural-networks-by-example-mnist-digits  \n",
    "https://www.kaggle.com/charel/learn-by-example-rnn-lstm-gru-time-series  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b5efd17086477a6de3fe833d2f72cb65d8ec2ad4"
   },
   "source": [
    "# Gym  \n",
    "In 2014 Google DeepMind published a paper titled \"Playing Atari with Deep Reinforcement Learning\" that can play Atari 2600 games at expert human levels. This was the first breakthrough in applying deep neural networks for reinforcement learning.\n",
    "![alt text](https://cdn-images-1.medium.com/max/800/1*3ZgGbUpEyAZb9POWijRq4Q.png)\n",
    "\n",
    "\n",
    "Gym is released by Open AI in 2016 (http://gym.openai.com/docs/). It is a toolkit for developing and comparing reinforcement learning algorithms.\n",
    "<img src=\"https://i.imgur.com/ria9HOm.jpg%20\" width=800>\n",
    "\n",
    "Source: [OpenAI](https://openai.com/)\n",
    "\n",
    "In 2018 Gym-retro was released as its successor: https://blog.openai.com/gym-retro/\n",
    "\n",
    "There are many many games made available. and you need to install the gym[atari] environment to make the Atari games available. In Kaggle you have to enable the Internet-beta feature available to make it internet connected (see to the left of the screen in the settings) and to be able to install the required packages\n",
    "![alt text](https://cdn-images-1.medium.com/max/800/1*vUMIoHkl-PuIjbTqbtn8dA.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: gym in /opt/conda/lib/python3.6/site-packages (0.12.1)\r\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (from gym) (1.1.0)\r\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from gym) (1.12.0)\r\n",
      "Requirement already satisfied: requests>=2.0 in /opt/conda/lib/python3.6/site-packages (from gym) (2.21.0)\r\n",
      "Requirement already satisfied: pyglet>=1.2.0 in /opt/conda/lib/python3.6/site-packages (from gym) (1.3.2)\r\n",
      "Requirement already satisfied: numpy>=1.10.4 in /opt/conda/lib/python3.6/site-packages (from gym) (1.16.3)\r\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (2.6)\r\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (3.0.4)\r\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (1.22)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym) (2019.3.9)\r\n",
      "Requirement already satisfied: future in /opt/conda/lib/python3.6/site-packages (from pyglet>=1.2.0->gym) (0.17.1)\r\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n",
      "Requirement already satisfied: gym[box2d] in /opt/conda/lib/python3.6/site-packages (0.12.1)\r\n",
      "Requirement already satisfied: scipy in /opt/conda/lib/python3.6/site-packages (from gym[box2d]) (1.1.0)\r\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from gym[box2d]) (1.12.0)\r\n",
      "Requirement already satisfied: pyglet>=1.2.0 in /opt/conda/lib/python3.6/site-packages (from gym[box2d]) (1.3.2)\r\n",
      "Requirement already satisfied: requests>=2.0 in /opt/conda/lib/python3.6/site-packages (from gym[box2d]) (2.21.0)\r\n",
      "Requirement already satisfied: numpy>=1.10.4 in /opt/conda/lib/python3.6/site-packages (from gym[box2d]) (1.16.3)\r\n",
      "Collecting box2d-py>=2.3.5; extra == \"box2d\" (from gym[box2d])\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/06/bd/6cdc3fd994b0649dcf5d9bad85bd9e26172308bbe9a421bfc6fdbf5081a6/box2d_py-2.3.8-cp36-cp36m-manylinux1_x86_64.whl (448kB)\r\n",
      "\u001b[K    100% |████████████████████████████████| 450kB 21.0MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: future in /opt/conda/lib/python3.6/site-packages (from pyglet>=1.2.0->gym[box2d]) (0.17.1)\r\n",
      "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[box2d]) (1.22)\r\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[box2d]) (3.0.4)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[box2d]) (2019.3.9)\r\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests>=2.0->gym[box2d]) (2.6)\r\n",
      "Installing collected packages: box2d-py\r\n",
      "Successfully installed box2d-py-2.3.8\r\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n",
      "Collecting atari_py\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/57/5d/dd62a272613b6c88d67e363239aa936ad9ee4f5b11855180c094c2e3b887/atari_py-0.1.15-cp36-cp36m-manylinux1_x86_64.whl (2.6MB)\r\n",
      "\u001b[K    100% |████████████████████████████████| 2.6MB 10.7MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from atari_py) (1.12.0)\r\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.6/site-packages (from atari_py) (1.16.3)\r\n",
      "Installing collected packages: atari-py\r\n",
      "Successfully installed atari-py-0.1.15\r\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!pip install gym \n",
    "!pip install 'gym[box2d]'\n",
    "!pip install atari_py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "13b64a70e319a88a887e3010c0e518a9d37da9ce"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "from gym import wrappers\n",
    "from gym import envs\n",
    "import numpy as np \n",
    "#import datetime\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#import time\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6712bd57cc9dfc42b2364a576f309581d30a6338"
   },
   "source": [
    "## Show game screens\n",
    "Let's show some game screens of these iconic games"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7887e3f2a84e3e4917c8d66fd734c9bbdf90998e"
   },
   "source": [
    "## Breakout\n",
    "\n",
    "![alt text](http://www.atarimania.com/2600/boxes/hi_res/breakout_color_cart_4.jpg)\n",
    "\n",
    "\n",
    "Source: [Atari mania](http://www.atarimania.com/game-atari-2600-vcs-breakout_18135.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "b2df72562ea9a4f497766fdf4f9793341b898dcd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space: Box(210, 160, 3)\n",
      "Action space: Discrete(4)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADm1JREFUeJzt3X/sVfV9x/Hna1j9g3YBqyNGcKCjXXDZqCWObGq6uVokTdH9YTFLpZsZmmjSRpcFa7KZJU22rmDSbLPBSIqL9UdHrWaxVsaammXDCpYiqChYjHyDMHURh00t8N4f5/Ndj1++l+/93ve5vedeX4/k5p77Ob8+J35ffs45nPu+igjMrHe/MugOmA07h8gsySEyS3KIzJIcIrMkh8gsqW8hkrRM0h5JeyWt6dd+zAZN/fh3IkkzgBeBTwIHgKeBayPiucZ3ZjZg/RqJLgb2RsTLEfEu8ACwok/7Mhuo0/q03XOBV2ufDwC/22lhSX5swtro9Yg4e6qF+hWiKUlaDawe1P7NuvBKNwv1K0RjwLza57ml7f9FxHpgPXgksuHWr2uip4GFkhZIOh1YCTzap32ZDVRfRqKIOCbpZuB7wAxgQ0Ts7se+zAatL7e4p92JFp7OrVu3btrr3HLLLaltTFy/qW1ktaEPE03sU5/2uT0ilky1kJ9YMEsa2N25YdOPUWIQo10TfhkjzTDxSGSW5JHIpm2q0e/9NlJ5JDJL8khkU5pqZBnEdVmbeCQyS/JI1KUm/m/blm0Mwz6HiUcisySHyCzJj/2YdebHfsx+GVpxY2Hu3Lnvu3+gs/br9m/SI5FZkkNkluQQmSU5RGZJPYdI0jxJ35f0nKTdkr5Q2u+QNCZpR3ktb667Zu2TuTt3DLg1Ip6R9CFgu6TNZd6dEfHVfPfM2q/nEEXEQeBgmX5b0vNURRvN3lcauSaSNB/4GPBUabpZ0k5JGyTNbmIfZm2VDpGkDwKbgC9GxBHgLuACYDHVSLW2w3qrJW2TtO3o0aPZbpgNTCpEkj5AFaD7IuLbABFxKCKOR8QJ4G6q4vYniYj1EbEkIpbMnDkz0w2zgcrcnRNwD/B8RKyrtZ9TW+xqYFfv3TNrv8zdud8HPgc8K2lHafsScK2kxUAA+4EbUj00a7nM3bn/ADTJrMd6747Z8PETC2ZJrfgqxFT8NQnrh6ZqR3gkMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzpPT3iSTtB94GjgPHImKJpDOBB4H5VF8RvyYi/ie7L7M2amok+oOIWFz7VbE1wJaIWAhsKZ/NRlK/TudWABvL9Ebgqj7tx2zgmghRAE9I2i5pdWmbU8oMA7wGzGlgP2at1ESNhUsiYkzSrwGbJb1QnxkRMdkPG5fArQaYPduVhm14pUeiiBgr74eBh6kqnh4aL+JY3g9Psp4roNpIyJYRnll+VgVJM4ErqCqePgqsKoutAh7J7MeszbKnc3OAh6uKwpwGfDMiHpf0NPCQpOuBV4Brkvsxa61UiCLiZeB3Jml/A7g8s22zYeEnFsyShqIC6tZlywbdBRtB/9nQdjwSmSU5RGZJDpFZkkNkluQQmSUNxd25E79xZNBdMOvII5FZkkNkluQQmSU5RGZJDpFZkkNkljQUt7jf/NV3Bt0Fs448EpklOURmST2fzkn6KFWV03HnA38FzAL+HPjv0v6liHis5x6atVzPIYqIPcBiAEkzgDGqaj9/CtwZEV9tpIdmLdfU6dzlwL6IeKWh7ZkNjabuzq0E7q99vlnSdcA24NZsMfs3f/PdzOpmk3u9mc2kRyJJpwOfAb5Vmu4CLqA61TsIrO2w3mpJ2yRtO3r0aLYbZgPTxOnclcAzEXEIICIORcTxiDgB3E1VEfUkroBqo6KJEF1L7VRuvHxwcTVVRVSzkZW6Jiqlgz8J3FBr/oqkxVS/FrF/wjyzkZOtgHoU+PCEts+lemQ2ZIbi2blvnjhv0F2wEXRFQ9vxYz9mSQ6RWZJDZJbkEJklOURmSUNxd+7dB+4YdBdsFF3RzI+reCQyS3KIzJIcIrMkh8gsySEyS3KIzJKG4hb3vz++dNBdsBH06SvWNbIdj0RmSQ6RWZJDZJbUVYgkbZB0WNKuWtuZkjZLeqm8zy7tkvQ1SXsl7ZR0Ub86b9YG3Y5E3wCWTWhbA2yJiIXAlvIZquo/C8trNVUJLbOR1VWIIuJJ4M0JzSuAjWV6I3BVrf3eqGwFZk2oAGQ2UjLXRHMi4mCZfg2YU6bPBV6tLXegtL2HizfaqGjkxkJEBFWJrOms4+KNNhIyITo0fppW3g+X9jFgXm25uaXNbCRlQvQosKpMrwIeqbVfV+7SLQXeqp32mY2crh77kXQ/8AngLEkHgL8G/hZ4SNL1wCvANWXxx4DlwF7gHarfKzIbWV2FKCKu7TDr8kmWDeCmTKfMhomfWDBLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLmjJEHaqf/r2kF0qF04clzSrt8yX9VNKO8vp6Pztv1gbdjETf4OTqp5uB34qI3wZeBG6rzdsXEYvL68ZmumnWXlOGaLLqpxHxREQcKx+3UpXFMntfauKa6M+A79Y+L5D0I0k/kHRpp5VcAdVGReqX8iTdDhwD7itNB4HzIuINSR8HviPpwog4MnHdiFgPrAeYN2/etKqnmrVJzyORpM8Dnwb+pJTJIiJ+FhFvlOntwD7gIw3006y1egqRpGXAXwKfiYh3au1nS5pRps+n+nmVl5voqFlbTXk616H66W3AGcBmSQBby524y4C/kfRz4ARwY0RM/EkWs5EyZYg6VD+9p8Oym4BN2U6ZDRM/sWCW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW1GsF1DskjdUqnS6vzbtN0l5JeyR9ql8dN2uLXiugAtxZq3T6GICkRcBK4MKyzj+NFy4xG1U9VUA9hRXAA6V01k+AvcDFif6ZtV7mmujmUtB+g6TZpe1c4NXaMgdK20lcAdVGRa8hugu4AFhMVfV07XQ3EBHrI2JJRCyZOXNmj90wG7yeQhQRhyLieEScAO7mF6dsY8C82qJzS5vZyOq1Auo5tY9XA+N37h4FVko6Q9ICqgqoP8x10azdeq2A+glJi4EA9gM3AETEbkkPAc9RFbq/KSKO96frZu3QaAXUsvyXgS9nOmU2TPzEglmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkm9Fm98sFa4cb+kHaV9vqSf1uZ9vZ+dN2uDKb/ZSlW88R+Ae8cbIuKz49OS1gJv1ZbfFxGLm+qgWdt18/XwJyXNn2yeJAHXAH/YbLfMhkf2muhS4FBEvFRrWyDpR5J+IOnS5PbNWq+b07lTuRa4v/b5IHBeRLwh6ePAdyRdGBFHJq4oaTWwGmD27NkTZ5sNjZ5HIkmnAX8MPDjeVmpwv1GmtwP7gI9Mtr4roNqoyJzO/RHwQkQcGG+QdPb4r0BIOp+qeOPLuS6atVs3t7jvB/4L+KikA5KuL7NW8t5TOYDLgJ3llve/ADdGRLe/KGE2lHot3khEfH6Stk3Apny3zIaHn1gwS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS8o+xd2It2ac4F9n/e+gu2GT2LpsWWr9pY8/3lBPmvd7TzzRyHY8EpklOURmSQ6RWVIrromsvdp8TdMWHonMkjwS2ftWU6OsIqKRDaU6IQ2+E2Yn2x4RS6ZaqJuvh8+T9H1Jz0naLekLpf1MSZslvVTeZ5d2SfqapL2Sdkq6KH8sZu3VzTXRMeDWiFgELAVukrQIWANsiYiFwJbyGeBKqgIlC6lKYt3VeK/NWmTKEEXEwYh4pky/DTwPnAusADaWxTYCV5XpFcC9UdkKzJJ0TuM9N2uJad2dK+WEPwY8BcyJiINl1mvAnDJ9LvBqbbUDpc1sJHV9d07SB6kq+XwxIo5UZbgrERHTvTlQr4BqNsy6GokkfYAqQPdFxLdL86Hx07Tyfri0jwHzaqvPLW3vUa+A2mvnzdqgm7tzAu4Bno+IdbVZjwKryvQq4JFa+3XlLt1S4K3aaZ/Z6ImIU76AS4AAdgI7yms58GGqu3IvAf8GnFmWF/CPVHW4nwWWdLGP8MuvFr62TfW3GxH+x1azU2jmH1vN7NQcIrMkh8gsySEyS3KIzJLa8n2i14Gj5X1UnMXoHM8oHQt0fzy/3s3GWnGLG0DStlF6emGUjmeUjgWaPx6fzpklOURmSW0K0fpBd6Bho3Q8o3Qs0PDxtOaayGxYtWkkMhtKAw+RpGWS9pTCJmumXqN9JO2X9KykHZK2lbZJC7m0kaQNkg5L2lVrG9pCNB2O5w5JY+W/0Q5Jy2vzbivHs0fSp6a9w24e9e7XC5hB9ZWJ84HTgR8DiwbZpx6PYz9w1oS2rwBryvQa4O8G3c9T9P8y4CJg11T9p/oazHepvvKyFHhq0P3v8njuAP5ikmUXlb+7M4AF5e9xxnT2N+iR6GJgb0S8HBHvAg9QFToZBZ0KubRORDwJvDmheWgL0XQ4nk5WAA9ExM8i4ifAXqq/y64NOkSjUtQkgCckbS+1I6BzIZdhMYqFaG4up6AbaqfX6eMZdIhGxSURcRFVzb2bJF1WnxnVecPQ3gYd9v4XdwEXAIuBg8DapjY86BB1VdSk7SJirLwfBh6mOh3oVMhlWKQK0bRNRByKiOMRcQK4m1+csqWPZ9AhehpYKGmBpNOBlVSFToaGpJmSPjQ+DVwB7KJzIZdhMVKFaCZct11N9d8IquNZKekMSQuoKvf+cFobb8GdlOXAi1R3RW4fdH966P/5VHd3fgzsHj8GOhRyaeMLuJ/qFOfnVNcE13fqPz0UomnJ8fxz6e/OEpxzasvfXo5nD3DldPfnJxbMkgZ9Omc29BwisySHyCzJITJLcojMkhwisySHyCzJITJL+j+3QFvlMGmcOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make(\"BreakoutNoFrameskip-v4\")\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.grid(False)\n",
    "print(\"Observation space:\", env.observation_space)\n",
    "print(\"Action space:\", env.action_space)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "4bb9fa071274df1f85290791f0a53bbe482e6a37"
   },
   "source": [
    "## Pacman\n",
    "\n",
    "![alt text](http://www.atarimania.com/2600/boxes/hi_res/ms_pac_man_silver_1987_cart_7.jpg)\n",
    "\n",
    "\n",
    "Source: [Atari mania](http://www.atarimania.com/game-atari-2600-vcs-ms-pac-man_7391.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "811d96cca823be22ae9e440b729b033d5b40d74f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space: Box(210, 160, 3)\n",
      "Action space: Discrete(9)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEjxJREFUeJzt3X/sXXV9x/HnyyrGIAtF2DeO4gCDJkig1k5JJg2bopUwCzMx7R+KPyKaiJmZy1Y02YjGhDnR1WRjK7EZGEZ1QydxldmRKZoMBCrWIlYK1tCmtirO30Ep7/1xzm1Pb7+X++Nzzj2fc76vR3Jz7/3cc+75nHO/7/v5nM/3c95XEYGZze4ZbVfArOscRGaJHERmiRxEZokcRGaJHERmiRoLIklrJe2WtEfSxqa2Y9Y2NfF/IknLgO8ClwD7gHuBDRHx7do3ZtayplqilwN7IuLRiPgNsBVY19C2zFr1zIbe93TgscrzfcArRi0sydMmLEc/iojTxi3UVBCNJekq4CqA0086iXve9ra2qmK2qBWbNn1/kuWaCqL9wBmV5yvKsiMiYjOwGeCChYVjWqIVtz2/oWrNbt8bDhxXlmM9czR87HI9bot9xpNo6pzoXuAcSWdJOgFYD9ze0LbMWtVISxQRT0q6GvgvYBmwJSIebGJbZm1r7JwoIrYB25p6f7NceMaCWaLWRuemMcmJ6bhlUl+vo57Tvl5HPeexzRyP3Sx/I7NyS2SWqJFpP9O6YGEhtm3YcOR5jkOgHuKeXVeHuFds2nR/RKwet55bIrNEDiKzRA4is0QOIrNEDiKzRJ34P9Ekpp082MYI0awTHHPThWM3zzp2Ioi6MkRqS5O7c2aJHERmiRxEZok6cU40D/OYRNlXS/3YuSUyS+SWqFTHN1+Xvj3rtNSP3cxBJOkM4GZgAQhgc0RsknQt8A7gh+Wi7y+vcp1Zlw+w9V9KS/Qk8L6I2CHpJOB+SdvL1z4eER9Nr55Z/mYOoog4ABwoH/9c0kMUSRvNlpRaBhYknQm8FLinLLpa0k5JWyQtr2MbZrlKHliQ9FzgNuC9EfEzSTcAH6I4T/oQcD1wXHrT4QyoqZo+b1rqJ88p+n7skloiSc+iCKBbIuKzABFxMCIOR8RTwI0Uye2PExGbI2J1RKx+3nOek1INs1bNHESSBHwSeCgiPlYpr35lXAHsmr16ZvlL6c79IfAm4FuSHijL3g9skLSSoju3F3hnUg3NMpcyOvc1QIu85KyntqR0YsZCDskb55HksK/JG3P4/CZdZhaeO2eWyMkbzUpO3mjWEgeRWSIHkVkiB5FZoiyHuMflGGvj8uNZcsbNYxtNaLrebV0+3tTxdUtklshBZJbIQWSWyEFklshBZJYoy9G5WdQ9+tbE1KOuJijswrFp89i6JTJL1JuWKPWbZx7fXF1peYZ14di0eWzdEpklqiPbz17g58Bh4MmIWC3pFODTwJkUl4i/MSJ+krotsxzV1RL9UUSsrFx7sRG4MyLOAe4sn5v1UlPduXXATeXjm4DLG9qOWevqGFgI4EuSAvjniNgMLJRphgF+QJH0fqSdP3lW9ifdXTi5bktX6t1UPesIoldGxH5Jvwtsl/Sd6osREWWAHaOaAZVlJ9dQjaXt7vO+fMzzC3dd3Eo9lqLk7lxE7C/vDwGfo8h4enCQxLG8P7TIekcyoPKME1OrsaQNB9CoMmtGahrhE8ufVUHSicBrKDKe3g5cWS52JfD5lO3YaHef92Uu3HXxkZan+tiBNB+pLdEC8DVJ3wS+DvxnRNwBXAdcIulh4NXlc2tINVgcOPOXdE4UEY8CFyxS/mPgVSnvXTWPxHw5JjmcZJl9u489/xk+F8pxv9r4/OrYxii9mfazlLklalcWyRt1wopg4T2NbqOrM6jHGRc0fRmla+Xz27fRyRvN5sFBZJbIQdRx1aHtxe6teQ6iHnAgtSuL0bnzl/+WbVMk1qvjpHIeiRKdvDEvUyeZ3DTZcm6JzBI5iMwSOYjMEjmIzBI5iMwSZTE6V4cuTOuZpY5N78ckI1Z9PbZ1cUtklqg3LVGO347DulDHxXSh3k7eaNZhDiKzRDN35yS9mCLL6cDZwF8DJwPvAH5Ylr8/IrbNXEOzzM0cRBGxG1gJIGkZsJ8i289bgY9HxEdrqaFZ5uoaWHgV8EhEfF/S1CuPS95YxwTHLpwcd1VXj21d9a7rnGg9cGvl+dWSdkraIml5Tdswy1JyEEk6AXg98G9l0Q3ACym6egeA60esd5Wk+yTdx1O/TK2GWWvqaIleB+yIiIMAEXEwIg5HxFPAjRQZUY/jDKjWF3UE0QYqXblB+uDSFRQZUc16K2lgoUwdfAnwzkrxRyStpPi1iL1DrzWm6QSD80hymKsckjfmfGxTM6D+EnjeUNmbkmpk1jGdSN7Y1W/waU3S2o3yhTdvPeb5ZTevb3ybXTP135GTNy4dwwE0qsya4SDquC+8eSuX3bz+SMtTfexAmg8HUQ9Ug8WBM38Ooh6onv9Mei5k9cniorxpkzfOoo0Eg/M6QW+7JerqsR1XbydvNJsTB5FZIgeRWSIHUcdVh7YXu7fmZTGwUIcuzGpoqo5NB9JSPraTcEtklshBZJaoN925HLsYw2apYw4/oNXXY1sXt0RmiRxEZokcRGaJJgqiMvXVIUm7KmWnSNou6eHyfnlZLkmfkLSnTJu1qqnKm+VgoitbJa0BfgHcHBHnlWUfAR6PiOskbQSWR8RfSboUeA9wKfAKYFNEvOJp33/Mla116ML/Omy0Vj6/Oq9sjYi7gMeHitcBN5WPbwIur5TfHIW7gZOHMgBZA3Zed9eR2+C5zUfKOdFCRAy+Hn4ALJSPTwceqyy3ryw7hpM31mc4YHZedxfnb1zjQJqTWgYWougTTpXxxMkbm3H+xjXA0UCy5qUE0cFBN628P1SW7wfOqCy3oiyzhpy/cc0xwWPzlTJj4XbgSuC68v7zlfKrJW2lGFj4aaXbN5M6kv+lbmMeyRtTt7FYIOW4X218fnVsY5SJgkjSrcDFwKmS9gF/QxE8n5H0duD7wBvLxbdRjMztAX5F8XtFZr3VieSNdVgKQ9zDXbk+nRPlPMTdmwmoS5nPg9rlaT891adWKHcOoh4YDhgH0Hy5O9cTDpz2ZBFE0yZvbGNQoI5fT6jj5Dj1Ir06ttnGftdh2mPn5I1mc+IgMkvkIDJL5CAyS5TFwEIdcjj5rbsOs9Zj3tvM8dg5eaNZh/SmJUr95qnjmyuHOrSxzRzew3nnzDrMQWSWyEFklshBZJbIQWSWaOzonKQtwGXAoUrixr8D/gT4DfAI8NaI+D9JZwIPAbvL1e+OiHdNW6kmRlr6ciVrX/ZjWvMYAZx1cu8kLdG/AGuHyrYD50XE+cB3gWsqrz0SESvL29QBZNY1Y4NoseynEfGliHiyfHo3RVossyWpjnOitwFfrDw/S9I3JH1F0kWjVqpmQP3xr39dQzXM2pE0Y0HSB4AngVvKogPACyLix5JeBvyHpJdExM+G142IzcBmgAsWFtpPOWQ2o5mDSNJbKAYcXlWmESYingCeKB/fL+kR4EXAfSmVrCMxXx0JBuuo5zTrL/Ye026jjuSN48zj2OWQIHKUmbpzktYCfwm8PiJ+VSk/TdKy8vHZwDnAo3VU1CxXkwxxL5b99Brg2cB2SXB0KHsN8EFJvwWeAt4VEcM/yTK1Sb4xxi2T+vok5jGJctptdGG/J3mPOvajqX8PjA2iiNiwSPEnRyx7G3BbaqXMusQzFswSOYjMEjmIzBL15srWqRPzdfQq0jaSN+YwXy/nz9ctkVkiB5FZIgeRWSIHkVmiTgwszCMxX47zv2atxzTm8cPHs9Qj1894MW6JzBJ1oiWax3BlV+Z/1W0e8xLrqkeu23BLZJbIQWSWyEFklshBZJbIQWSWqBOjc7lqYzJoG1L3s+/GtkSStkg6JGlXpexaSfslPVDeLq28do2kPZJ2S3ptHZXc94YDx9zMcjJrBlSAj1cynW4DkHQusB54SbnOPw4Sl5j11UwZUJ/GOmBrRDwREd8D9gAvT6ifWfZSBhaulrSz7O4tL8tOBx6rLLOvLDuOM6BaX8w6sHAD8CEgyvvrKdIJTyy3DKht/vr0qDrMox5tbHOSenRl0AVmbIki4mBEHI6Ip4AbOdpl2w+cUVl0RVlm1lsztUSSnh8Rg6+OK4DByN3twL9K+hjwexQZUL+eXMs5yOGbL9cJqPOQSz1mMWsG1IslraTozu0F3gkQEQ9K+gzwbYpE9++OiMOplezyAbb+qzUDarn8h4EPp1TKrEs87ccskYPILFFv5s41fd6Uy3nZUh18yKEOo7glMkvkIDJL5CAyS+QgMkvUiYGFHH74eB5JDnP94eMcfpS4dz98bGZHKaL1CdRcsLAQ2zYcnRiR83Cm9ddxLdWmTfdHxOpx67klMkvkIDJL5CAyS+QgMkuU5RC302JZl7glMks0a/LGT1cSN+6V9EBZfqakX1de+6cmK2+Wg7H/J5K0BvgFcHNEnLfI69cDP42ID0o6E/jCYsuN2Ub7/6wyO95E/yea5PLwu8rgOI4kAW8E/nja2qXavv0PALjkknuPPB48n+Y9Uta3ZtyxahUAa3fsaLkmk0k9J7oIOBgRD1fKzpL0DUlfkXRR4vsvavDHPxwAg9emeY9Z17dm3LFqFWt37GDtjh3csWrVkYDKWWoQbQBurTw/ALwgIl4K/DlF+qzfWWzFagbUaTc6+OOvtkazvses61szhlufQTDlbOYhbknPBP4UeNmgLCKeAJ4oH98v6RHgRcBxgVLNgJp6TpQaDA6mvA0CKdfuXcr/iV4NfCci9g0KJJ0GPB4RhyWdTZG88dHEOo6V+sfv4MlPNWhyb4kmGeK+Ffhf4MWS9kl6e/nSeo7tygGsAXaWQ97/DrwrIib9RQkzYHQA5RpMsyZvJCLeskjZbcBt6dWajrtz/ZNz921Yr2YsVAcb2ljf0g23PNVAyjWoOhtEgyHu1Pew/A2GvHPV2SAaGA6EaQMjdX2rVxdanmFZXB7uaT+WKV8ebjYPDiKzRA4is0RZXtlq7fvq3x+dO3zRe7/aYk3y55bIjjMIoEHwVAPKjucgsmMMB5ADaTwHkVkiB5FZIgeRHWO4+zbcvbPjecaCLcqjc8CEMxYcRGajedqP2Tw4iMwSTXJ5+BmS/kfStyU9KOnPyvJTJG2X9HB5v7wsl6RPSNojaaekPK/pNavJJC3Rk8D7IuJc4ELg3ZLOBTYCd0bEOcCd5XOA11EkKDkHuAq4ofZam2VkbBBFxIGI2FE+/jnwEHA6sA64qVzsJuDy8vE6ipTDERF3AydL8u9HWm9NdU5UphN+KXAPsBARg99A+QGwUD4+HXisstq+ssyslyaexS3puRSZfN4bET8r0nAXIiKmHaaWdBVFd8+s0yZqiSQ9iyKAbomIz5bFBwfdtPL+UFm+HzijsvqKsuwYEbE5IlZPMg5vlrNJRucEfBJ4KCI+VnnpduDK8vGVwOcr5W8uR+kupPjZFf/0nfVXRDztDXglEMBO4IHydinwPIpRuYeB/wZOKZcX8A/AI8C3gNUTbCN88y3D233j/nYjwtN+zJ6Gp/2YzYODyCyRg8gskYPILJGDyCxRLnnnfgT8srzvi1Ppz/70aV9g8v35/UneLIshbgBJ9/Vp9kKf9qdP+wL174+7c2aJHERmiXIKos1tV6BmfdqfPu0L1Lw/2ZwTmXVVTi2RWSe1HkSS1kraXSY22Th+jfxI2ivpW5IekHRfWbZoIpccSdoi6ZCkXZWyziaiGbE/10raX35GD0i6tPLaNeX+7Jb02qk3OMlU76ZuwDKKSybOBk4Avgmc22adZtyPvcCpQ2UfATaWjzcCf9t2PZ+m/muAVcCucfWnuAzmixSXvFwI3NN2/Sfcn2uBv1hk2XPLv7tnA2eVf4/Lptle2y3Ry4E9EfFoRPwG2EqR6KQPRiVyyU5E3AU8PlTc2UQ0I/ZnlHXA1oh4IiK+B+yh+LucWNtB1JekJgF8SdL9Ze4IGJ3IpSv6mIjm6rILuqXSvU7en7aDqC9eGRGrKHLuvVvSmuqLUfQbOjsM2vX6l24AXgisBA4A19f1xm0H0URJTXIXEfvL+0PA5yi6A6MSuXRFUiKa3ETEwYg4HBFPATdytMuWvD9tB9G9wDmSzpJ0ArCeItFJZ0g6UdJJg8fAa4BdjE7k0hW9SkQzdN52BcVnBMX+rJf0bElnUWTu/fpUb57BSMqlwHcpRkU+0HZ9Zqj/2RSjO98EHhzsAyMSueR4A26l6OL8luKc4O2j6s8MiWgy2Z9PlfXdWQbO8yvLf6Dcn93A66bdnmcsmCVquztn1nkOIrNEDiKzRA4is0QOIrNEDiKzRA4is0QOIrNE/w/RoSIsTs1XHAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make(\"MsPacmanNoFrameskip-v4\")\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.grid(False)\n",
    "print(\"Observation space:\", env.observation_space)\n",
    "print(\"Action space:\", env.action_space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b5d42a48c5f1e1704ecbc332b8a78628899c5456"
   },
   "source": [
    "## Spaceinvaders\n",
    "\n",
    "![alt text](http://www.atarimania.com/2600/boxes/hi_res/space_invaders_silver_1986_cart_2.jpg)\n",
    "\n",
    "\n",
    "Source: [Atari mania](http://www.atarimania.com/game-atari-2600-vcs-space-invaders_8102.html/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "60af82b5260b334e2962b76990c8b37a25d313f9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space: Box(210, 160, 3)\n",
      "Action space: Discrete(6)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAEetJREFUeJzt3X+sHNV5xvHvU/PjD5IKG6hlgVPbyKEC2joGAVICoiX8bCVDWlGjNpAU9YKKpURKpRhTtVZbXdE0BClqS2QUK1ClBlRCoJQ2uFYSVCkmXBPH2IDBJqbYNXYgFdA0JQHe/rFzyXq5e3d2zszO7NznI43u3rMzZ9+jnXfP7NmZM4oIzKy4X6g7ALNx5yQyS+QkMkvkJDJL5CQyS+QkMktUWRJJukzSbkl7JK2t6nXM6qYqfieSNA94DrgY2A88AVwTEU+X/mJmNauqJzoH2BMRL0TET4F7gFUVvZZZrY6qqN6TgZe6/t8PnNtvZUk+bcKa6JWIOGnQSlUl0UCSJoCJul7fLIcX86xUVRIdABZ3/X9KVvauiNgAbAD3RDbeqvpO9ASwXNJSSccAq4GHKnots1pV0hNFxFuS1gDfAOYBGyNiVxWvZVa3Soa4hw7Ch3PWTNsi4uxBK/mMBbNETiKzRE4is0ROIrNEtf3YOpuVf7Vy6G2e/NMnK4gkzbDtqKINd0+eN/Q2167bWnocqYZtxyjb0NrRudQduC2JXIbUHXiMEznX6Fwjk6h3B86zgzdxBx62HaPoifLs4A3ZgY8wbDtKasP4JlEZUnfgPDv4KJKgCVJ34Dw7eEVJkMq/E5mNQiN7Ih/OlceHc0l8ONfNh3PF+XBudo1MIvdE5XFPlGR8k6gM7onK455odo1Morb8RuMfW8tT04+t45tEZfCPreXxj62za20SmZXAvxOZjULhJJK0WNI3JT0taZekT2Xl6yUdkLQ9W64oL1yz5kk5i/st4DMR8aSk9wPbJG3Onrs9Ij6fHp5Z8xVOoog4CBzMHr8h6Rk6kzaazSmlfCeStAT4EPB4VrRG0g5JGyXNL+M1zJoqOYkkvQ+4H/h0RLwO3AGcCqyg01Pd1me7CUlTkqZSYzCrU9IQt6SjgYeBb0TEF2Z4fgnwcEScOaAeD3FbE1U7xC1JwJeBZ7oTSNKirtWuAnYWfQ2zcZAyOvdh4OPAU5K2Z2XrgGskrQAC2AfckBShWcP5jAWz/nzGwiCTk4sHr1RxHU2JoQntGFsRUftC59CvsmVycnGusmG2L6OOYbavqh2jjmHMlqlc+2/dCVR1Ek2/wd1vdJEdb3JycSl1pO684xzDGC65kqiRkzdWYd26l9495Fi37qUBa8+8PZBcR8r2ZdTRhBjaZs4MLMx0zD7MDtDvmD+1jmF3wiraMeoYxogHFqb1fmr29ip5ty+jjqLbl1HHbNuPsh1tMyd6otReZLYdJLWOUfaGTYhhzLgnMhuFOTWw0G3Yw4+ZPmlT6yhyCFR2O+qIoW3cE5klav13opkGBWYqy7N9GXUU2b6MOgZtX0YdLfxe5Nl+zBJ5YMFsFJxEZomcRGaJnERmiZxEZomSf2yVtA94A3gbeCsizpa0ALgXWELnEvGrI+K/U1/LrInK6ol+IyJWdA0HrgW2RMRyYEv2v1krVXXazyrgwuzxXcC3gM9W9Fq5DPvD4qDty6ijyI+TZbejjhjapoyeKIBHJW2TNJGVLcymGQZ4GVhYwusUNlMCFLkEoew6hj3nrIp2jDqGNiojiT4SESuBy4GbJF3Q/WR0Tol4zxkJo54BtfvTsugVoWXUkbJ9GXU0IYa2KfW0H0nrgf8B/gi4MCIOZpM5fisiTptlu5GcOzeTYc+dq6KOYc+dqyKGMupoYUJVf9qPpOOy26og6TjgEjoznj4EXJetdh3wYMrrpOq+CrP3isxhti+jjqLbl1FHE2Joo9SBhYXAA50ZhTkK+MeI+DdJTwD3SboeeBG4OvF1knmikubE0DZJSRQRLwC/PkP5q8BFKXWbjYs5c2Xr5OTiI3qTokPDqXWkbF9GHU2IoW182o9ZIieRWSJf2WrWn69sNRsFJ5FZIieRWSInkVkiJ5FZIieRWSInkVkiJ5FZotYn0WzXwOS5xmbQOql1FLnRWBUxjKodbdT6JDKr2pxJou5PyiKfmr2f1kXrSNm+jDqaEEPbzJlLIaCcNzy1DsfQQhFR+0JnIpPKlsnJxe/+7X487PZl1FF0+zLbUWcMY7ZM5dl/C/dEkk6jM8vptGXAnwHH05mo5IdZ+bqIeKTo65RhpovGis6xUGYdRec3GOcY2qhwEkXEbmAFgKR5wAHgAeCTwO0R8flSIjRruLIGFi4C9kbEiyXVZzY+SvpOsxFYkz1eT2cS+x1Z+fy6vxPBe7/XFNm+jDpSti+rHXXHMEZLru9EyVe2SjoG+C/gjIg4JGkh8EoWxF8CiyLiD2fYbgKYnnb4rKQgZtFvaqe8Uz7Ntl5qHcNMO1VVO4rcPDmlHWNmNDc+lrQKuCkiLpnhuSXAwxFx5oA60oIwq8bILg+/Btg0/U82bfC0q+jMiGrWWkk/tmZTB18M3NBV/DlJK+gczu3rec6sdTzbj1l/nu3HbBScRGaJnERmiZxEZonmzKUQvvFxc2JomznRE/nGx82JoY1an0QzndbSe9e7vNuXUUeR7cuoY9D2o2pHG82J34n6vcF5D0Nm20FS6xjmUKiqdowyhjHj34mm+aK85sTQRnMiiaZ13/G6rjocQ/vMqSQyq4KTyCxV3TP9jHK2n0Flw2xfRh1FZ9oZ5xjGbMl1Zat7IrNUdfdCVfdEs31K5vkEHbROah15P8WrbEfv/BFVtmPMltHMsVAGX09kDeXficxGIVcSSdoo6bCknV1lCyRtlvR89nd+Vi5JX5S0R9IOSSurCt6sCfL2RF8BLuspWwtsiYjlwJbsf4DLgeXZMgHckR6mWXPlSqKIeAz4UU/xKuCu7PFdwJVd5XdHx1bg+J4ZgMxaJeU70cKIOJg9fhlYmD0+Geg+H2R/VnYESROSpiRNJcRgVrtSLsqLiBh2hC0iNgAbwKNzNt5SeqJD04dp2d/DWfkBoPt8+VOyMrNWSumJHgKuA27N/j7YVb5G0j3AucBrXYd9tfHl4c2JoW3yDnFvAr4DnCZpv6Tr6STPxZKeBz6a/Q/wCPACsAe4E/jj0qMeki8Pb04MbdT6Mxb6fWoOezeFMusYdvsy6hi0fRl1tLBH8hkLZqMwZ6bMmlbGoUdqHY6hXVp/OAedN3vdupeGOnSZafvpxyl1FN2+jDqaEMOY8eFcrybMLeAY2mdOJZFZFZxEZonmzMBC7zDssMOyM30HKFpHytBwWe1I+Y5XRjvaZE4MLJgV5IEFs1FwEpklchKZJXISmSVyEpklchKZJXISmSVyEpklmlNJlPfepFXW4RjaZ2AS9Zn99G8kPZvNcPqApOOz8iWSfiJpe7Z8qcrg8/Ll4c2JoY3y9ERf4b2zn24GzoyIXwOeA27uem5vRKzIlhvLCbO42d7gIncPL7uOIncPryKGUbWjjQaegBoRj0la0lP2aNe/W4HfLTes8nW/yUXe8N5tUusoutOV2Y66YmidnPcPWgLs7PPcPwN/0LXej4HvAd8Gzp+lzglgKlsqvc9MFXeYK6OOYe/pU0U7Rh3DmC257k+UdCmEpFuAt4CvZkUHgQ9ExKuSzgK+LumMiHi9d1vPgGptUXh0TtIngN8Gfj+mb3cX8WZEvJo93gbsBT5YQpyl6p4zoa46HEOLFDmcozPQ8DRwUs96JwHzssfL6EwfvKBJt5us4/AlNYYq2lFHDGO4lHM4l81+eiFwoqT9wJ/TGY07FtgsCWBrNhJ3AfAXkn4GvAPcGBG9t2Qxa5cqbmQ87EKFnybdn5Z1ffqmxlB2O+qKYQwX3/jYLJEvDzcbBSeRWSInkVkiJ5FZIieRWSInkVkiJ5FZIieRWSInkVkiJ5FZIieRWSInkVkiJ5FZIieRWSInkVkiJ5FZoqIzoK6XdKBrptMrup67WdIeSbslXVpV4GZNUXQGVIDbu2Y6fQRA0unAauCMbJu/lzSvrGDNmmhgEkXEY0DeyUZWAfdkU2f9ANgDnJMQn1njpXwnWpNNaL9R0vys7GSgexKy/VnZe0iakDQlaSohBrPaFU2iO4BTgRV0Zj29bdgKImJDRJydZyIIsyYrlEQRcSgi3o6Id4A7+fkh2wGge4bzU7Iys9YqlESSFnX9exUwPXL3ELBa0rGSlgLLge+mhWjWbEVnQL1Q0go6E9ztA24AiIhdku6jM8XwW8BNEfF2NaGbNYMnbzTrz5M3mo1C0v2J5rp/+Z1fOeL/37r/WcdQUwx1ck9klshJVFDvp2+/MsfQfk4is0ROIrNETiKzRE4is0ROIrNETiKzRE4is0ROogJm+x1kVL+ROIbmcBKZJXISmSVyEpklchKZJSo6eeO9XRM37pO0PStfIuknXc99qcrgzZogz/VEXwH+Frh7uiAifm/6saTbgNe61t8bESvKCtCs6QYmUUQ8JmnJTM9JEnA18JvlhjUeei8+q2NY1zHUL/U70fnAoYh4vqtsqaTvSfq2pPMT6zdrvogYuABLgJ0zlN8BfKbr/2OBE7LHZ9GZDfUX+9Q5AUxlS3jx0sBlKk9+FO6JJB0FfAy4d7osm4P71ezxNmAv8MGZtvcMqNYWKYdzHwWejYj90wWSTpq+C4SkZXQmb3whLUSzZsszxL0J+A5wmqT9kq7PnloNbOpZ/QJgRzbk/U/AjRGR944SZmPJkzea9efJG81GwUlklshJZJbISWSWyElklsgT2rfc3ZPnDb3Nteu2VhBJe7knMkvknqjlBvUqRXoqO5J7IrNE7olazj1N9dwTmSVyT9Ry/k5UPfdEZoncE7Wce5rquScySzRW1xNdec0vVR2K2bu+vulwruuJxuJwblTJ859nnALAB3btH7CmleVjv7oMgK89Nb6zCOS5PHyxpG9KelrSLkmfysoXSNos6fns7/ysXJK+KGmPpB2SVlbdCLM65emJ3qIzLdaTkt4PbJO0GfgEsCUibpW0FlgLfBa4nM4EJcuBc+lMq3XubC9w/IKjuPDSBcVbYVajgT1RRByMiCezx28AzwAnA6uAu7LV7gKuzB6vAu6Ojq3A8ZIWlR65WUMMNTqXTSf8IeBxYGFEHMyeehlYmD0+mc6kjdP2Z2VmrZQ7iSS9D7gf+HREvN79XHSG+IYa5pM0IWlK0tSb//fOMJuaNUqu0TlJR9NJoK9GxNey4kOSFkXEwexw7XBWfgBY3LX5KVnZESJiA7ABYP4JR9c/zo5H5eowzqNy0/KMzgn4MvBMRHyh66mHgOuyx9cBD3aVX5uN0p0HvNZ12GfWOnl6og8DHweemr6ZF7AOuBW4L5sR9UU6t1gBeAS4AtgD/C/wyVIjNmuYPPcn+g9AfZ6+aIb1A7gpMS6zseFz58wSOYnMEjmJzBI5icwSOYnMEjXleqIfAj8GXqk7lhKdSHva06a2QP72/HJEnDRopUYkEYCkqTbdv7VN7WlTW6D89vhwziyRk8gsUZOSaEPdAZSsTe1pU1ug5PY05juR2bhqUk9kNpZqTyJJl0nanU1ssrbueIqQtE/SU5K2S5rKymacyKWJJG2UdFjSzq6ysZ2Ipk971ks6kL1H2yVd0fXczVl7dku6dOgXjIjaFmAesBdYBhwDfB84vc6YCrZjH3BiT9nngLXZ47XAX9cd5yzxXwCsBHYOip/OZS7/SufM/vOAx+uOP2d71gN/MsO6p2f73bHA0mx/nDfM69XdE50D7ImIFyLip8A9dCY6aYN+E7k0TkQ8Bvyop3hsJ6Lp055+VgH3RMSbEfEDOtfBnTPM69WdRG2Z1CSARyVtkzSRlfWbyGVctHEimjXZIejGrsPr5PbUnURt8ZGIWElnzr2bJF3Q/WR0jhvGdhh03OPP3AGcCqwADgK3lVVx3UmUa1KTpouIA9nfw8ADdA4HDk0f5vRM5DIu+sU/lu9ZRByKiLcj4h3gTn5+yJbcnrqT6AlguaSlko4BVtOZ6GRsSDoumxkWSccBlwA76T+Ry7ho1UQ0Pd/brqLzHkGnPaslHStpKZ2Ze787VOUNGEm5AniOzqjILXXHUyD+ZXRGd74P7JpuA3ACsAV4Hvh3YEHdsc7Shk10DnF+Ruc7wfX94qczKvd32fv1FHB23fHnbM8/ZPHuyBJnUdf6t2Tt2Q1cPuzr+YwFs0R1H86ZjT0nkVkiJ5FZIieRWSInkVkiJ5FZIieRWSInkVmi/weiZzrQdx/PMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "env = gym.make(\"SpaceInvadersNoFrameskip-v4\")\n",
    "plt.imshow(env.render('rgb_array'))\n",
    "plt.grid(False)\n",
    "print(\"Observation space:\", env.observation_space)\n",
    "print(\"Action space:\", env.action_space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "94434fa3e41536e2a901722d164f0467e75ff113"
   },
   "source": [
    "## Capture a video\n",
    "Below the code to set-up a virtual monitor and capture a movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "640339e36fc751c6ae2922a73395bf29c8a71abc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Get:1 http://security.debian.org/debian-security stretch/updates InRelease [94.3 kB]\r\n",
      "Ign:2 http://deb.debian.org/debian stretch InRelease\r\n",
      "Get:3 http://deb.debian.org/debian stretch-updates InRelease [91.0 kB]\r\n",
      "Err:1 http://security.debian.org/debian-security stretch/updates InRelease\r\n",
      "  Couldn't create temporary file /tmp/apt.conf.IDcOz6 for passing config to apt-key\r\n",
      "Err:3 http://deb.debian.org/debian stretch-updates InRelease\r\n",
      "  Couldn't create temporary file /tmp/apt.conf.CdJAv7 for passing config to apt-key\r\n",
      "Get:4 http://deb.debian.org/debian stretch Release [118 kB]\r\n",
      "Get:5 http://deb.debian.org/debian stretch Release.gpg [2434 B]\r\n",
      "Err:5 http://deb.debian.org/debian stretch Release.gpg\r\n",
      "  Couldn't create temporary file /tmp/apt.conf.OOH9V9 for passing config to apt-key\r\n",
      "Ign:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease\r\n",
      "Ign:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease\r\n",
      "Get:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]\r\n",
      "Get:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release [564 B]\r\n",
      "Get:10 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]\r\n",
      "Err:10 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg\r\n",
      "  Couldn't create temporary file /tmp/apt.conf.3tyHPr for passing config to apt-key\r\n",
      "Get:11 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release.gpg [801 B]\r\n",
      "Err:11 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release.gpg\r\n",
      "  Couldn't create temporary file /tmp/apt.conf.iG6wTu for passing config to apt-key\r\n",
      "Fetched 308 kB in 1s (211 kB/s)\r\n",
      "\r\n",
      "W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://security.debian.org/debian-security stretch/updates InRelease: Couldn't create temporary file /tmp/apt.conf.IDcOz6 for passing config to apt-key\r\n",
      "W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://deb.debian.org/debian stretch-updates InRelease: Couldn't create temporary file /tmp/apt.conf.CdJAv7 for passing config to apt-key\r\n",
      "W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://deb.debian.org/debian stretch Release: Couldn't create temporary file /tmp/apt.conf.OOH9V9 for passing config to apt-key\r\n",
      "W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: Couldn't create temporary file /tmp/apt.conf.3tyHPr for passing config to apt-key\r\n",
      "W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release: Couldn't create temporary file /tmp/apt.conf.iG6wTu for passing config to apt-key\r\n",
      "W: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/InRelease  Couldn't create temporary file /tmp/apt.conf.CdJAv7 for passing config to apt-key\r\n",
      "W: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/InRelease  Couldn't create temporary file /tmp/apt.conf.IDcOz6 for passing config to apt-key\r\n",
      "W: Failed to fetch http://deb.debian.org/debian/dists/stretch/Release.gpg  Couldn't create temporary file /tmp/apt.conf.OOH9V9 for passing config to apt-key\r\n",
      "W: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release.gpg  Couldn't create temporary file /tmp/apt.conf.3tyHPr for passing config to apt-key\r\n",
      "W: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Release.gpg  Couldn't create temporary file /tmp/apt.conf.iG6wTu for passing config to apt-key\r\n",
      "W: Some index files failed to download. They have been ignored, or old ones used instead.\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "The following additional packages will be installed:\r\n",
      "  freeglut3 libglu1-mesa\r\n",
      "Suggested packages:\r\n",
      "  python-tk python-numpy libgle3\r\n",
      "The following NEW packages will be installed:\r\n",
      "  freeglut3 libglu1-mesa python-opengl\r\n",
      "0 upgraded, 3 newly installed, 0 to remove and 48 not upgraded.\r\n",
      "Need to get 791 kB of archives.\r\n",
      "After this operation, 6260 kB of additional disk space will be used.\r\n",
      "Get:1 http://deb.debian.org/debian stretch/main amd64 freeglut3 amd64 2.8.1-3 [123 kB]\r\n",
      "Get:2 http://deb.debian.org/debian stretch/main amd64 libglu1-mesa amd64 9.0.0-2.1 [168 kB]\r\n",
      "Get:3 http://deb.debian.org/debian stretch/main amd64 python-opengl all 3.1.0+dfsg-1 [501 kB]\r\n",
      "Fetched 791 kB in 0s (8306 kB/s)\r\n",
      "debconf: delaying package configuration, since apt-utils is not installed\r\n",
      "Selecting previously unselected package freeglut3:amd64.\r\n",
      "(Reading database ... 37078 files and directories currently installed.)\r\n",
      "Preparing to unpack .../freeglut3_2.8.1-3_amd64.deb ...\r\n",
      "Unpacking freeglut3:amd64 (2.8.1-3) ...\r\n",
      "Selecting previously unselected package libglu1-mesa:amd64.\r\n",
      "Preparing to unpack .../libglu1-mesa_9.0.0-2.1_amd64.deb ...\r\n",
      "Unpacking libglu1-mesa:amd64 (9.0.0-2.1) ...\r\n",
      "Selecting previously unselected package python-opengl.\r\n",
      "Preparing to unpack .../python-opengl_3.1.0+dfsg-1_all.deb ...\r\n",
      "Unpacking python-opengl (3.1.0+dfsg-1) ...\r\n",
      "Setting up freeglut3:amd64 (2.8.1-3) ...\r\n",
      "Processing triggers for libc-bin (2.24-11+deb9u3) ...\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libcuda.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104 is empty, not checked.\r\n",
      "Setting up libglu1-mesa:amd64 (9.0.0-2.1) ...\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libcuda.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104 is empty, not checked.\r\n",
      "Setting up python-opengl (3.1.0+dfsg-1) ...\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "xvfb is already the newest version (2:1.19.2-1+deb9u5).\r\n",
      "0 upgraded, 0 newly installed, 0 to remove and 48 not upgraded.\r\n",
      "Collecting pyvirtualdisplay\r\n",
      "  Downloading https://files.pythonhosted.org/packages/68/6b/4bc5678b5219edba7190ae45a7e3d02b03dc7d20ffae84047c7494b355c3/PyVirtualDisplay-0.2.3-py2.py3-none-any.whl\r\n",
      "Collecting EasyProcess (from pyvirtualdisplay)\r\n",
      "  Downloading https://files.pythonhosted.org/packages/fa/29/40040d1d64a224a5e44df9572794a66494618ffe5c77199214aeceedb8a7/EasyProcess-0.2.7-py2.py3-none-any.whl\r\n",
      "Installing collected packages: EasyProcess, pyvirtualdisplay\r\n",
      "Successfully installed EasyProcess-0.2.7 pyvirtualdisplay-0.2.3\r\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n",
      "Collecting piglet\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/86/f6/ef278239ebe525466ea51a7dd9d6d3211d197ac4b4abc76e17cdd419f69c/piglet-0.4.4.tar.gz (52kB)\r\n",
      "\u001b[K    100% |████████████████████████████████| 61kB 5.0MB/s \r\n",
      "\u001b[?25hCollecting Parsley (from piglet)\r\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/2b/d6/4fed8d65e28a970e1c5cb33ce9c7e22e3de745e1b2ae37af051ef16aea3b/Parsley-1.3-py2.py3-none-any.whl (88kB)\r\n",
      "\u001b[K    100% |████████████████████████████████| 92kB 13.2MB/s \r\n",
      "\u001b[?25hRequirement already satisfied: attrs in /opt/conda/lib/python3.6/site-packages (from piglet) (18.1.0)\r\n",
      "Collecting astunparse (from piglet)\r\n",
      "  Downloading https://files.pythonhosted.org/packages/2e/37/5dd0dd89b87bb5f0f32a7e775458412c52d78f230ab8d0c65df6aabc4479/astunparse-1.6.2-py2.py3-none-any.whl\r\n",
      "Requirement already satisfied: markupsafe in /opt/conda/lib/python3.6/site-packages (from piglet) (1.0)\r\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in /opt/conda/lib/python3.6/site-packages (from astunparse->piglet) (0.31.1)\r\n",
      "Requirement already satisfied: six<2.0,>=1.6.1 in /opt/conda/lib/python3.6/site-packages (from astunparse->piglet) (1.12.0)\r\n",
      "Building wheels for collected packages: piglet\r\n",
      "  Building wheel for piglet (setup.py) ... \u001b[?25l-\b \bdone\r\n",
      "\u001b[?25h  Stored in directory: /tmp/.cache/pip/wheels/c7/59/a5/5bd1a35a4a4596714c4c7925a1751e7b1580b6ced363fd7969\r\n",
      "Successfully built piglet\r\n",
      "Installing collected packages: Parsley, astunparse, piglet\r\n",
      "Successfully installed Parsley-1.3 astunparse-1.6.2 piglet-0.4.4\r\n",
      "\u001b[33mYou are using pip version 19.0.3, however version 19.1.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n",
      "\r\n",
      "\r\n",
      "\r\n",
      "The following additional packages will be installed:\r\n",
      "  i965-va-driver libaacs0 libass5 libavc1394-0 libavcodec57 libavdevice57\r\n",
      "  libavfilter6 libavformat57 libavresample3 libavutil55 libbdplus0 libbluray1\r\n",
      "  libbs2b0 libcaca0 libcdio-cdda1 libcdio-paranoia1 libcdio13 libchromaprint1\r\n",
      "  libcrystalhd3 libdc1394-22 libebur128-1 libflite1 libfribidi0 libgme0\r\n",
      "  libiec61883-0 libjack-jackd2-0 libmpg123-0 libnuma1 libopenal-data\r\n",
      "  libopenal1 libopencv-core2.4v5 libopencv-imgproc2.4v5 libopenmpt0 libopus0\r\n",
      "  libpgm-5.2-0 libpostproc54 libraw1394-11 librubberband2 libsamplerate0\r\n",
      "  libsdl2-2.0-0 libshine3 libslang2 libsnappy1v5 libsndio6.1 libsodium18\r\n",
      "  libsoxr0 libspeex1 libssh-gcrypt-4 libswresample2 libswscale4 libtbb2\r\n",
      "  libtheora0 libusb-1.0-0 libva-drm1 libva-x11-1 libva1 libvdpau-va-gl1\r\n",
      "  libvdpau1 libvpx4 libx264-148 libx265-95 libxss1 libxvidcore4 libzmq5\r\n",
      "  libzvbi-common libzvbi0 mesa-va-drivers mesa-vdpau-drivers va-driver-all\r\n",
      "  vdpau-driver-all\r\n",
      "Suggested packages:\r\n",
      "  ffmpeg-doc libbluray-bdj firmware-crystalhd alsa-base jackd2 libportaudio2\r\n",
      "  opus-tools libraw1394-doc sndiod speex libfglrx-amdxvba1 nvidia-vdpau-driver\r\n",
      "  nvidia-legacy-340xx-vdpau-driver nvidia-legacy-304xx-vdpau-driver\r\n",
      "The following NEW packages will be installed:\r\n",
      "  ffmpeg i965-va-driver libaacs0 libass5 libavc1394-0 libavcodec57\r\n",
      "  libavdevice57 libavfilter6 libavformat57 libavresample3 libavutil55\r\n",
      "  libbdplus0 libbluray1 libbs2b0 libcaca0 libcdio-cdda1 libcdio-paranoia1\r\n",
      "  libcdio13 libchromaprint1 libcrystalhd3 libdc1394-22 libebur128-1 libflite1\r\n",
      "  libfribidi0 libgme0 libiec61883-0 libjack-jackd2-0 libmpg123-0 libnuma1\r\n",
      "  libopenal-data libopenal1 libopencv-core2.4v5 libopencv-imgproc2.4v5\r\n",
      "  libopenmpt0 libopus0 libpgm-5.2-0 libpostproc54 libraw1394-11 librubberband2\r\n",
      "  libsamplerate0 libsdl2-2.0-0 libshine3 libslang2 libsnappy1v5 libsndio6.1\r\n",
      "  libsodium18 libsoxr0 libspeex1 libssh-gcrypt-4 libswresample2 libswscale4\r\n",
      "  libtbb2 libtheora0 libusb-1.0-0 libva-drm1 libva-x11-1 libva1\r\n",
      "  libvdpau-va-gl1 libvdpau1 libvpx4 libx264-148 libx265-95 libxss1\r\n",
      "  libxvidcore4 libzmq5 libzvbi-common libzvbi0 mesa-va-drivers\r\n",
      "  mesa-vdpau-drivers va-driver-all vdpau-driver-all\r\n",
      "0 upgraded, 71 newly installed, 0 to remove and 48 not upgraded.\r\n",
      "Need to get 35.5 MB of archives.\r\n",
      "After this operation, 124 MB of additional disk space will be used.\r\n",
      "Get:1 http://deb.debian.org/debian stretch/main amd64 libxss1 amd64 1:1.2.2-1 [17.5 kB]\r\n",
      "Get:2 http://deb.debian.org/debian stretch/main amd64 libslang2 amd64 2.3.1-5 [503 kB]\r\n",
      "Get:3 http://deb.debian.org/debian stretch/main amd64 libva1 amd64 1.7.3-2 [52.1 kB]\r\n",
      "Get:4 http://deb.debian.org/debian stretch/main amd64 libva-drm1 amd64 1.7.3-2 [15.0 kB]\r\n",
      "Get:5 http://deb.debian.org/debian stretch/main amd64 libva-x11-1 amd64 1.7.3-2 [19.5 kB]\r\n",
      "Get:6 http://deb.debian.org/debian stretch/main amd64 libvdpau1 amd64 1.1.1-6 [41.2 kB]\r\n",
      "Get:7 http://deb.debian.org/debian stretch/main amd64 libavutil55 amd64 7:3.2.12-1~deb9u1 [219 kB]\r\n",
      "Get:8 http://deb.debian.org/debian stretch/main amd64 libcrystalhd3 amd64 1:0.0~git20110715.fdd2f19-12 [45.8 kB]\r\n",
      "Get:9 http://deb.debian.org/debian stretch/main amd64 libopus0 amd64 1.2~alpha2-1 [171 kB]\r\n",
      "Get:10 http://deb.debian.org/debian stretch/main amd64 libshine3 amd64 3.1.0-5 [26.2 kB]\r\n",
      "Get:11 http://deb.debian.org/debian stretch/main amd64 libsnappy1v5 amd64 1.1.3-3 [51.0 kB]\r\n",
      "Get:12 http://deb.debian.org/debian stretch/main amd64 libspeex1 amd64 1.2~rc1.2-1+b2 [54.8 kB]\r\n",
      "Get:13 http://deb.debian.org/debian stretch/main amd64 libsoxr0 amd64 0.1.2-2 [64.7 kB]\r\n",
      "Get:14 http://deb.debian.org/debian stretch/main amd64 libswresample2 amd64 7:3.2.12-1~deb9u1 [98.2 kB]\r\n",
      "Get:15 http://deb.debian.org/debian stretch/main amd64 libtheora0 amd64 1.1.1+dfsg.1-14+b1 [177 kB]\r\n",
      "Get:16 http://deb.debian.org/debian stretch/main amd64 libvpx4 amd64 1.6.1-3+deb9u1 [776 kB]\r\n",
      "Get:17 http://deb.debian.org/debian stretch/main amd64 libx264-148 amd64 2:0.148.2748+git97eaef2-1 [611 kB]\r\n",
      "Get:18 http://deb.debian.org/debian stretch/main amd64 libnuma1 amd64 2.0.11-2.1 [33.3 kB]\r\n",
      "Get:19 http://deb.debian.org/debian stretch/main amd64 libx265-95 amd64 2.1-2+b2 [986 kB]\r\n",
      "Get:20 http://deb.debian.org/debian stretch/main amd64 libxvidcore4 amd64 2:1.3.4-1+b2 [242 kB]\r\n",
      "Get:21 http://deb.debian.org/debian stretch/main amd64 libzvbi-common all 0.2.35-13 [63.7 kB]\r\n",
      "Get:22 http://deb.debian.org/debian stretch/main amd64 libzvbi0 amd64 0.2.35-13 [268 kB]\r\n",
      "Get:23 http://deb.debian.org/debian stretch/main amd64 libavcodec57 amd64 7:3.2.12-1~deb9u1 [4450 kB]\r\n",
      "Get:24 http://deb.debian.org/debian stretch/main amd64 libraw1394-11 amd64 2.1.2-1+b1 [47.2 kB]\r\n",
      "Get:25 http://deb.debian.org/debian stretch/main amd64 libavc1394-0 amd64 0.5.4-4+b1 [20.0 kB]\r\n",
      "Get:26 http://deb.debian.org/debian stretch/main amd64 libfribidi0 amd64 0.19.7-1+b1 [45.7 kB]\r\n",
      "Get:27 http://deb.debian.org/debian stretch/main amd64 libass5 amd64 1:0.13.4-2 [90.4 kB]\r\n",
      "Get:28 http://deb.debian.org/debian stretch/main amd64 libbluray1 amd64 1:0.9.3-3 [137 kB]\r\n",
      "Get:29 http://deb.debian.org/debian stretch/main amd64 libchromaprint1 amd64 1.4.2-1 [39.9 kB]\r\n",
      "Get:30 http://deb.debian.org/debian stretch/main amd64 libgme0 amd64 0.6.0-4 [126 kB]\r\n",
      "Get:31 http://deb.debian.org/debian stretch/main amd64 libmpg123-0 amd64 1.23.8-1+b1 [137 kB]\r\n",
      "Get:32 http://deb.debian.org/debian stretch/main amd64 libopenmpt0 amd64 0.2.7386~beta20.3-3+deb9u3 [483 kB]\r\n",
      "Get:33 http://deb.debian.org/debian stretch/main amd64 libssh-gcrypt-4 amd64 0.7.3-2+deb9u2 [170 kB]\r\n",
      "Get:34 http://deb.debian.org/debian stretch/main amd64 libavformat57 amd64 7:3.2.12-1~deb9u1 [940 kB]\r\n",
      "Get:35 http://deb.debian.org/debian stretch/main amd64 libavresample3 amd64 7:3.2.12-1~deb9u1 [95.7 kB]\r\n",
      "Get:36 http://deb.debian.org/debian stretch/main amd64 libbs2b0 amd64 3.1.0+dfsg-2.2 [12.2 kB]\r\n",
      "Get:37 http://deb.debian.org/debian stretch/main amd64 libebur128-1 amd64 1.2.2-2 [14.7 kB]\r\n",
      "Get:38 http://deb.debian.org/debian stretch/main amd64 libflite1 amd64 2.0.0-release-3+b1 [12.9 MB]\r\n",
      "Get:39 http://deb.debian.org/debian stretch/main amd64 libtbb2 amd64 4.3~20150611-2 [136 kB]\r\n",
      "Get:40 http://deb.debian.org/debian stretch/main amd64 libopencv-core2.4v5 amd64 2.4.9.1+dfsg1-2 [653 kB]\r\n",
      "Get:41 http://deb.debian.org/debian stretch/main amd64 libopencv-imgproc2.4v5 amd64 2.4.9.1+dfsg1-2 [618 kB]\r\n",
      "Get:42 http://deb.debian.org/debian stretch/main amd64 libpostproc54 amd64 7:3.2.12-1~deb9u1 [95.6 kB]\r\n",
      "Get:43 http://deb.debian.org/debian stretch/main amd64 libsamplerate0 amd64 0.1.8-8+b2 [949 kB]\r\n",
      "Get:44 http://deb.debian.org/debian stretch/main amd64 librubberband2 amd64 1.8.1-7 [86.0 kB]\r\n",
      "Get:45 http://deb.debian.org/debian stretch/main amd64 libswscale4 amd64 7:3.2.12-1~deb9u1 [194 kB]\r\n",
      "Get:46 http://deb.debian.org/debian stretch/main amd64 libpgm-5.2-0 amd64 5.2.122~dfsg-2 [157 kB]\r\n",
      "Get:47 http://deb.debian.org/debian stretch/main amd64 libsodium18 amd64 1.0.11-2 [161 kB]\r\n",
      "Get:48 http://deb.debian.org/debian stretch/main amd64 libzmq5 amd64 4.2.1-4+deb9u1 [201 kB]\r\n",
      "Get:49 http://deb.debian.org/debian stretch/main amd64 libavfilter6 amd64 7:3.2.12-1~deb9u1 [811 kB]\r\n",
      "Get:50 http://deb.debian.org/debian stretch/main amd64 libcaca0 amd64 0.99.beta19-2+b2 [347 kB]\r\n",
      "Get:51 http://deb.debian.org/debian stretch/main amd64 libcdio13 amd64 0.83-4.3+b1 [168 kB]\r\n",
      "Get:52 http://deb.debian.org/debian stretch/main amd64 libcdio-cdda1 amd64 0.83-4.3+b1 [132 kB]\r\n",
      "Get:53 http://deb.debian.org/debian stretch/main amd64 libcdio-paranoia1 amd64 0.83-4.3+b1 [132 kB]\r\n",
      "Get:54 http://deb.debian.org/debian stretch/main amd64 libusb-1.0-0 amd64 2:1.0.21-1 [53.9 kB]\r\n",
      "Get:55 http://deb.debian.org/debian stretch/main amd64 libdc1394-22 amd64 2.2.5-1 [106 kB]\r\n",
      "Get:56 http://deb.debian.org/debian stretch/main amd64 libiec61883-0 amd64 1.2.0-2 [30.2 kB]\r\n",
      "Get:57 http://deb.debian.org/debian stretch/main amd64 libjack-jackd2-0 amd64 1.9.10+20150825git1ed50c92~dfsg-5 [292 kB]\r\n",
      "Get:58 http://deb.debian.org/debian stretch/main amd64 libopenal-data all 1:1.17.2-4 [107 kB]\r\n",
      "Get:59 http://deb.debian.org/debian stretch/main amd64 libsndio6.1 amd64 1.1.0-3 [23.5 kB]\r\n",
      "Get:60 http://deb.debian.org/debian stretch/main amd64 libopenal1 amd64 1:1.17.2-4+b2 [212 kB]\r\n",
      "Get:61 http://deb.debian.org/debian stretch/main amd64 libsdl2-2.0-0 amd64 2.0.5+dfsg1-2 [358 kB]\r\n",
      "Get:62 http://deb.debian.org/debian stretch/main amd64 libavdevice57 amd64 7:3.2.12-1~deb9u1 [116 kB]\r\n",
      "Get:63 http://deb.debian.org/debian stretch/main amd64 ffmpeg amd64 7:3.2.12-1~deb9u1 [1532 kB]\r\n",
      "Get:64 http://deb.debian.org/debian stretch/main amd64 i965-va-driver amd64 1.7.3-1 [353 kB]\r\n",
      "Get:65 http://deb.debian.org/debian stretch/main amd64 libaacs0 amd64 0.8.1-2 [48.6 kB]\r\n",
      "Get:66 http://deb.debian.org/debian stretch/main amd64 libbdplus0 amd64 0.1.2-2 [46.9 kB]\r\n",
      "Get:67 http://deb.debian.org/debian stretch/main amd64 libvdpau-va-gl1 amd64 0.4.2-1 [71.7 kB]\r\n",
      "Get:68 http://deb.debian.org/debian stretch/main amd64 mesa-va-drivers amd64 13.0.6-1+b2 [1460 kB]\r\n",
      "Get:69 http://deb.debian.org/debian stretch/main amd64 mesa-vdpau-drivers amd64 13.0.6-1+b2 [1600 kB]\r\n",
      "Get:70 http://deb.debian.org/debian stretch/main amd64 va-driver-all amd64 1.7.3-2 [11.1 kB]\r\n",
      "Get:71 http://deb.debian.org/debian stretch/main amd64 vdpau-driver-all amd64 1.1.1-6 [20.4 kB]\r\n",
      "Fetched 35.5 MB in 0s (69.3 MB/s)\r\n",
      "debconf: delaying package configuration, since apt-utils is not installed\r\n",
      "Selecting previously unselected package libxss1:amd64.\r\n",
      "(Reading database ... 39445 files and directories currently installed.)\r\n",
      "Preparing to unpack .../00-libxss1_1%3a1.2.2-1_amd64.deb ...\r\n",
      "Unpacking libxss1:amd64 (1:1.2.2-1) ...\r\n",
      "Selecting previously unselected package libslang2:amd64.\r\n",
      "Preparing to unpack .../01-libslang2_2.3.1-5_amd64.deb ...\r\n",
      "Unpacking libslang2:amd64 (2.3.1-5) ...\r\n",
      "Selecting previously unselected package libva1:amd64.\r\n",
      "Preparing to unpack .../02-libva1_1.7.3-2_amd64.deb ...\r\n",
      "Unpacking libva1:amd64 (1.7.3-2) ...\r\n",
      "Selecting previously unselected package libva-drm1:amd64.\r\n",
      "Preparing to unpack .../03-libva-drm1_1.7.3-2_amd64.deb ...\r\n",
      "Unpacking libva-drm1:amd64 (1.7.3-2) ...\r\n",
      "Selecting previously unselected package libva-x11-1:amd64.\r\n",
      "Preparing to unpack .../04-libva-x11-1_1.7.3-2_amd64.deb ...\r\n",
      "Unpacking libva-x11-1:amd64 (1.7.3-2) ...\r\n",
      "Selecting previously unselected package libvdpau1:amd64.\r\n",
      "Preparing to unpack .../05-libvdpau1_1.1.1-6_amd64.deb ...\r\n",
      "Unpacking libvdpau1:amd64 (1.1.1-6) ...\r\n",
      "Selecting previously unselected package libavutil55:amd64.\r\n",
      "Preparing to unpack .../06-libavutil55_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavutil55:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libcrystalhd3:amd64.\r\n",
      "Preparing to unpack .../07-libcrystalhd3_1%3a0.0~git20110715.fdd2f19-12_amd64.deb ...\r\n",
      "Unpacking libcrystalhd3:amd64 (1:0.0~git20110715.fdd2f19-12) ...\r\n",
      "Selecting previously unselected package libopus0:amd64.\r\n",
      "Preparing to unpack .../08-libopus0_1.2~alpha2-1_amd64.deb ...\r\n",
      "Unpacking libopus0:amd64 (1.2~alpha2-1) ...\r\n",
      "Selecting previously unselected package libshine3:amd64.\r\n",
      "Preparing to unpack .../09-libshine3_3.1.0-5_amd64.deb ...\r\n",
      "Unpacking libshine3:amd64 (3.1.0-5) ...\r\n",
      "Selecting previously unselected package libsnappy1v5:amd64.\r\n",
      "Preparing to unpack .../10-libsnappy1v5_1.1.3-3_amd64.deb ...\r\n",
      "Unpacking libsnappy1v5:amd64 (1.1.3-3) ...\r\n",
      "Selecting previously unselected package libspeex1:amd64.\r\n",
      "Preparing to unpack .../11-libspeex1_1.2~rc1.2-1+b2_amd64.deb ...\r\n",
      "Unpacking libspeex1:amd64 (1.2~rc1.2-1+b2) ...\r\n",
      "Selecting previously unselected package libsoxr0:amd64.\r\n",
      "Preparing to unpack .../12-libsoxr0_0.1.2-2_amd64.deb ...\r\n",
      "Unpacking libsoxr0:amd64 (0.1.2-2) ...\r\n",
      "Selecting previously unselected package libswresample2:amd64.\r\n",
      "Preparing to unpack .../13-libswresample2_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libswresample2:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libtheora0:amd64.\r\n",
      "Preparing to unpack .../14-libtheora0_1.1.1+dfsg.1-14+b1_amd64.deb ...\r\n",
      "Unpacking libtheora0:amd64 (1.1.1+dfsg.1-14+b1) ...\r\n",
      "Selecting previously unselected package libvpx4:amd64.\r\n",
      "Preparing to unpack .../15-libvpx4_1.6.1-3+deb9u1_amd64.deb ...\r\n",
      "Unpacking libvpx4:amd64 (1.6.1-3+deb9u1) ...\r\n",
      "Selecting previously unselected package libx264-148:amd64.\r\n",
      "Preparing to unpack .../16-libx264-148_2%3a0.148.2748+git97eaef2-1_amd64.deb ...\r\n",
      "Unpacking libx264-148:amd64 (2:0.148.2748+git97eaef2-1) ...\r\n",
      "Selecting previously unselected package libnuma1:amd64.\r\n",
      "Preparing to unpack .../17-libnuma1_2.0.11-2.1_amd64.deb ...\r\n",
      "Unpacking libnuma1:amd64 (2.0.11-2.1) ...\r\n",
      "Selecting previously unselected package libx265-95:amd64.\r\n",
      "Preparing to unpack .../18-libx265-95_2.1-2+b2_amd64.deb ...\r\n",
      "Unpacking libx265-95:amd64 (2.1-2+b2) ...\r\n",
      "Selecting previously unselected package libxvidcore4:amd64.\r\n",
      "Preparing to unpack .../19-libxvidcore4_2%3a1.3.4-1+b2_amd64.deb ...\r\n",
      "Unpacking libxvidcore4:amd64 (2:1.3.4-1+b2) ...\r\n",
      "Selecting previously unselected package libzvbi-common.\r\n",
      "Preparing to unpack .../20-libzvbi-common_0.2.35-13_all.deb ...\r\n",
      "Unpacking libzvbi-common (0.2.35-13) ...\r\n",
      "Selecting previously unselected package libzvbi0:amd64.\r\n",
      "Preparing to unpack .../21-libzvbi0_0.2.35-13_amd64.deb ...\r\n",
      "Unpacking libzvbi0:amd64 (0.2.35-13) ...\r\n",
      "Selecting previously unselected package libavcodec57:amd64.\r\n",
      "Preparing to unpack .../22-libavcodec57_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavcodec57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libraw1394-11:amd64.\r\n",
      "Preparing to unpack .../23-libraw1394-11_2.1.2-1+b1_amd64.deb ...\r\n",
      "Unpacking libraw1394-11:amd64 (2.1.2-1+b1) ...\r\n",
      "Selecting previously unselected package libavc1394-0:amd64.\r\n",
      "Preparing to unpack .../24-libavc1394-0_0.5.4-4+b1_amd64.deb ...\r\n",
      "Unpacking libavc1394-0:amd64 (0.5.4-4+b1) ...\r\n",
      "Selecting previously unselected package libfribidi0:amd64.\r\n",
      "Preparing to unpack .../25-libfribidi0_0.19.7-1+b1_amd64.deb ...\r\n",
      "Unpacking libfribidi0:amd64 (0.19.7-1+b1) ...\r\n",
      "Selecting previously unselected package libass5:amd64.\r\n",
      "Preparing to unpack .../26-libass5_1%3a0.13.4-2_amd64.deb ...\r\n",
      "Unpacking libass5:amd64 (1:0.13.4-2) ...\r\n",
      "Selecting previously unselected package libbluray1:amd64.\r\n",
      "Preparing to unpack .../27-libbluray1_1%3a0.9.3-3_amd64.deb ...\r\n",
      "Unpacking libbluray1:amd64 (1:0.9.3-3) ...\r\n",
      "Selecting previously unselected package libchromaprint1:amd64.\r\n",
      "Preparing to unpack .../28-libchromaprint1_1.4.2-1_amd64.deb ...\r\n",
      "Unpacking libchromaprint1:amd64 (1.4.2-1) ...\r\n",
      "Selecting previously unselected package libgme0:amd64.\r\n",
      "Preparing to unpack .../29-libgme0_0.6.0-4_amd64.deb ...\r\n",
      "Unpacking libgme0:amd64 (0.6.0-4) ...\r\n",
      "Selecting previously unselected package libmpg123-0:amd64.\r\n",
      "Preparing to unpack .../30-libmpg123-0_1.23.8-1+b1_amd64.deb ...\r\n",
      "Unpacking libmpg123-0:amd64 (1.23.8-1+b1) ...\r\n",
      "Selecting previously unselected package libopenmpt0:amd64.\r\n",
      "Preparing to unpack .../31-libopenmpt0_0.2.7386~beta20.3-3+deb9u3_amd64.deb ...\r\n",
      "Unpacking libopenmpt0:amd64 (0.2.7386~beta20.3-3+deb9u3) ...\r\n",
      "Selecting previously unselected package libssh-gcrypt-4:amd64.\r\n",
      "Preparing to unpack .../32-libssh-gcrypt-4_0.7.3-2+deb9u2_amd64.deb ...\r\n",
      "Unpacking libssh-gcrypt-4:amd64 (0.7.3-2+deb9u2) ...\r\n",
      "Selecting previously unselected package libavformat57:amd64.\r\n",
      "Preparing to unpack .../33-libavformat57_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavformat57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libavresample3:amd64.\r\n",
      "Preparing to unpack .../34-libavresample3_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavresample3:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libbs2b0:amd64.\r\n",
      "Preparing to unpack .../35-libbs2b0_3.1.0+dfsg-2.2_amd64.deb ...\r\n",
      "Unpacking libbs2b0:amd64 (3.1.0+dfsg-2.2) ...\r\n",
      "Selecting previously unselected package libebur128-1:amd64.\r\n",
      "Preparing to unpack .../36-libebur128-1_1.2.2-2_amd64.deb ...\r\n",
      "Unpacking libebur128-1:amd64 (1.2.2-2) ...\r\n",
      "Selecting previously unselected package libflite1:amd64.\r\n",
      "Preparing to unpack .../37-libflite1_2.0.0-release-3+b1_amd64.deb ...\r\n",
      "Unpacking libflite1:amd64 (2.0.0-release-3+b1) ...\r\n",
      "Selecting previously unselected package libtbb2:amd64.\r\n",
      "Preparing to unpack .../38-libtbb2_4.3~20150611-2_amd64.deb ...\r\n",
      "Unpacking libtbb2:amd64 (4.3~20150611-2) ...\r\n",
      "Selecting previously unselected package libopencv-core2.4v5:amd64.\r\n",
      "Preparing to unpack .../39-libopencv-core2.4v5_2.4.9.1+dfsg1-2_amd64.deb ...\r\n",
      "Unpacking libopencv-core2.4v5:amd64 (2.4.9.1+dfsg1-2) ...\r\n",
      "Selecting previously unselected package libopencv-imgproc2.4v5:amd64.\r\n",
      "Preparing to unpack .../40-libopencv-imgproc2.4v5_2.4.9.1+dfsg1-2_amd64.deb ...\r\n",
      "Unpacking libopencv-imgproc2.4v5:amd64 (2.4.9.1+dfsg1-2) ...\r\n",
      "Selecting previously unselected package libpostproc54:amd64.\r\n",
      "Preparing to unpack .../41-libpostproc54_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libpostproc54:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libsamplerate0:amd64.\r\n",
      "Preparing to unpack .../42-libsamplerate0_0.1.8-8+b2_amd64.deb ...\r\n",
      "Unpacking libsamplerate0:amd64 (0.1.8-8+b2) ...\r\n",
      "Selecting previously unselected package librubberband2:amd64.\r\n",
      "Preparing to unpack .../43-librubberband2_1.8.1-7_amd64.deb ...\r\n",
      "Unpacking librubberband2:amd64 (1.8.1-7) ...\r\n",
      "Selecting previously unselected package libswscale4:amd64.\r\n",
      "Preparing to unpack .../44-libswscale4_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libswscale4:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libpgm-5.2-0:amd64.\r\n",
      "Preparing to unpack .../45-libpgm-5.2-0_5.2.122~dfsg-2_amd64.deb ...\r\n",
      "Unpacking libpgm-5.2-0:amd64 (5.2.122~dfsg-2) ...\r\n",
      "Selecting previously unselected package libsodium18:amd64.\r\n",
      "Preparing to unpack .../46-libsodium18_1.0.11-2_amd64.deb ...\r\n",
      "Unpacking libsodium18:amd64 (1.0.11-2) ...\r\n",
      "Selecting previously unselected package libzmq5:amd64.\r\n",
      "Preparing to unpack .../47-libzmq5_4.2.1-4+deb9u1_amd64.deb ...\r\n",
      "Unpacking libzmq5:amd64 (4.2.1-4+deb9u1) ...\r\n",
      "Selecting previously unselected package libavfilter6:amd64.\r\n",
      "Preparing to unpack .../48-libavfilter6_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavfilter6:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package libcaca0:amd64.\r\n",
      "Preparing to unpack .../49-libcaca0_0.99.beta19-2+b2_amd64.deb ...\r\n",
      "Unpacking libcaca0:amd64 (0.99.beta19-2+b2) ...\r\n",
      "Selecting previously unselected package libcdio13:amd64.\r\n",
      "Preparing to unpack .../50-libcdio13_0.83-4.3+b1_amd64.deb ...\r\n",
      "Unpacking libcdio13:amd64 (0.83-4.3+b1) ...\r\n",
      "Selecting previously unselected package libcdio-cdda1:amd64.\r\n",
      "Preparing to unpack .../51-libcdio-cdda1_0.83-4.3+b1_amd64.deb ...\r\n",
      "Unpacking libcdio-cdda1:amd64 (0.83-4.3+b1) ...\r\n",
      "Selecting previously unselected package libcdio-paranoia1:amd64.\r\n",
      "Preparing to unpack .../52-libcdio-paranoia1_0.83-4.3+b1_amd64.deb ...\r\n",
      "Unpacking libcdio-paranoia1:amd64 (0.83-4.3+b1) ...\r\n",
      "Selecting previously unselected package libusb-1.0-0:amd64.\r\n",
      "Preparing to unpack .../53-libusb-1.0-0_2%3a1.0.21-1_amd64.deb ...\r\n",
      "Unpacking libusb-1.0-0:amd64 (2:1.0.21-1) ...\r\n",
      "Selecting previously unselected package libdc1394-22:amd64.\r\n",
      "Preparing to unpack .../54-libdc1394-22_2.2.5-1_amd64.deb ...\r\n",
      "Unpacking libdc1394-22:amd64 (2.2.5-1) ...\r\n",
      "Selecting previously unselected package libiec61883-0:amd64.\r\n",
      "Preparing to unpack .../55-libiec61883-0_1.2.0-2_amd64.deb ...\r\n",
      "Unpacking libiec61883-0:amd64 (1.2.0-2) ...\r\n",
      "Selecting previously unselected package libjack-jackd2-0:amd64.\r\n",
      "Preparing to unpack .../56-libjack-jackd2-0_1.9.10+20150825git1ed50c92~dfsg-5_amd64.deb ...\r\n",
      "Unpacking libjack-jackd2-0:amd64 (1.9.10+20150825git1ed50c92~dfsg-5) ...\r\n",
      "Selecting previously unselected package libopenal-data.\r\n",
      "Preparing to unpack .../57-libopenal-data_1%3a1.17.2-4_all.deb ...\r\n",
      "Unpacking libopenal-data (1:1.17.2-4) ...\r\n",
      "Selecting previously unselected package libsndio6.1:amd64.\r\n",
      "Preparing to unpack .../58-libsndio6.1_1.1.0-3_amd64.deb ...\r\n",
      "Unpacking libsndio6.1:amd64 (1.1.0-3) ...\r\n",
      "Selecting previously unselected package libopenal1:amd64.\r\n",
      "Preparing to unpack .../59-libopenal1_1%3a1.17.2-4+b2_amd64.deb ...\r\n",
      "Unpacking libopenal1:amd64 (1:1.17.2-4+b2) ...\r\n",
      "Selecting previously unselected package libsdl2-2.0-0:amd64.\r\n",
      "Preparing to unpack .../60-libsdl2-2.0-0_2.0.5+dfsg1-2_amd64.deb ...\r\n",
      "Unpacking libsdl2-2.0-0:amd64 (2.0.5+dfsg1-2) ...\r\n",
      "Selecting previously unselected package libavdevice57:amd64.\r\n",
      "Preparing to unpack .../61-libavdevice57_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking libavdevice57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package ffmpeg.\r\n",
      "Preparing to unpack .../62-ffmpeg_7%3a3.2.12-1~deb9u1_amd64.deb ...\r\n",
      "Unpacking ffmpeg (7:3.2.12-1~deb9u1) ...\r\n",
      "Selecting previously unselected package i965-va-driver:amd64.\r\n",
      "Preparing to unpack .../63-i965-va-driver_1.7.3-1_amd64.deb ...\r\n",
      "Unpacking i965-va-driver:amd64 (1.7.3-1) ...\r\n",
      "Selecting previously unselected package libaacs0:amd64.\r\n",
      "Preparing to unpack .../64-libaacs0_0.8.1-2_amd64.deb ...\r\n",
      "Unpacking libaacs0:amd64 (0.8.1-2) ...\r\n",
      "Selecting previously unselected package libbdplus0:amd64.\r\n",
      "Preparing to unpack .../65-libbdplus0_0.1.2-2_amd64.deb ...\r\n",
      "Unpacking libbdplus0:amd64 (0.1.2-2) ...\r\n",
      "Selecting previously unselected package libvdpau-va-gl1:amd64.\r\n",
      "Preparing to unpack .../66-libvdpau-va-gl1_0.4.2-1_amd64.deb ...\r\n",
      "Unpacking libvdpau-va-gl1:amd64 (0.4.2-1) ...\r\n",
      "Selecting previously unselected package mesa-va-drivers:amd64.\r\n",
      "Preparing to unpack .../67-mesa-va-drivers_13.0.6-1+b2_amd64.deb ...\r\n",
      "Unpacking mesa-va-drivers:amd64 (13.0.6-1+b2) ...\r\n",
      "Selecting previously unselected package mesa-vdpau-drivers:amd64.\r\n",
      "Preparing to unpack .../68-mesa-vdpau-drivers_13.0.6-1+b2_amd64.deb ...\r\n",
      "Unpacking mesa-vdpau-drivers:amd64 (13.0.6-1+b2) ...\r\n",
      "Selecting previously unselected package va-driver-all:amd64.\r\n",
      "Preparing to unpack .../69-va-driver-all_1.7.3-2_amd64.deb ...\r\n",
      "Unpacking va-driver-all:amd64 (1.7.3-2) ...\r\n",
      "Selecting previously unselected package vdpau-driver-all:amd64.\r\n",
      "Preparing to unpack .../70-vdpau-driver-all_1.1.1-6_amd64.deb ...\r\n",
      "Unpacking vdpau-driver-all:amd64 (1.1.1-6) ...\r\n",
      "Setting up libpgm-5.2-0:amd64 (5.2.122~dfsg-2) ...\r\n",
      "Setting up libraw1394-11:amd64 (2.1.2-1+b1) ...\r\n",
      "Setting up libcdio13:amd64 (0.83-4.3+b1) ...\r\n",
      "Setting up libx264-148:amd64 (2:0.148.2748+git97eaef2-1) ...\r\n",
      "Setting up libebur128-1:amd64 (1.2.2-2) ...\r\n",
      "Setting up libaacs0:amd64 (0.8.1-2) ...\r\n",
      "Setting up libnuma1:amd64 (2.0.11-2.1) ...\r\n",
      "Setting up libtbb2:amd64 (4.3~20150611-2) ...\r\n",
      "Setting up libflite1:amd64 (2.0.0-release-3+b1) ...\r\n",
      "Setting up libcdio-cdda1:amd64 (0.83-4.3+b1) ...\r\n",
      "Setting up libsoxr0:amd64 (0.1.2-2) ...\r\n",
      "Setting up libssh-gcrypt-4:amd64 (0.7.3-2+deb9u2) ...\r\n",
      "Setting up libxss1:amd64 (1:1.2.2-1) ...\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libcuda.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104 is empty, not checked.\r\n",
      "Setting up libvdpau1:amd64 (1.1.1-6) ...\r\n",
      "Setting up libshine3:amd64 (3.1.0-5) ...\r\n",
      "Setting up libopencv-core2.4v5:amd64 (2.4.9.1+dfsg1-2) ...\r\n",
      "Setting up libiec61883-0:amd64 (1.2.0-2) ...\r\n",
      "Setting up libspeex1:amd64 (1.2~rc1.2-1+b2) ...\r\n",
      "Setting up libxvidcore4:amd64 (2:1.3.4-1+b2) ...\r\n",
      "Setting up libvpx4:amd64 (1.6.1-3+deb9u1) ...\r\n",
      "Setting up libopus0:amd64 (1.2~alpha2-1) ...\r\n",
      "Setting up libva1:amd64 (1.7.3-2) ...\r\n",
      "Setting up libopenal-data (1:1.17.2-4) ...\r\n",
      "Setting up libbs2b0:amd64 (3.1.0+dfsg-2.2) ...\r\n",
      "Setting up i965-va-driver:amd64 (1.7.3-1) ...\r\n",
      "Setting up libusb-1.0-0:amd64 (2:1.0.21-1) ...\r\n",
      "Setting up libcrystalhd3:amd64 (1:0.0~git20110715.fdd2f19-12) ...\r\n",
      "Processing triggers for libc-bin (2.24-11+deb9u3) ...\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libcuda.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104 is empty, not checked.\r\n",
      "Setting up libsnappy1v5:amd64 (1.1.3-3) ...\r\n",
      "Setting up libbluray1:amd64 (1:0.9.3-3) ...\r\n",
      "Setting up mesa-va-drivers:amd64 (13.0.6-1+b2) ...\r\n",
      "Setting up libavc1394-0:amd64 (0.5.4-4+b1) ...\r\n",
      "Setting up libzvbi-common (0.2.35-13) ...\r\n",
      "Setting up libfribidi0:amd64 (0.19.7-1+b1) ...\r\n",
      "Setting up libcdio-paranoia1:amd64 (0.83-4.3+b1) ...\r\n",
      "Setting up libva-drm1:amd64 (1.7.3-2) ...\r\n",
      "Setting up libgme0:amd64 (0.6.0-4) ...\r\n",
      "Setting up libsodium18:amd64 (1.0.11-2) ...\r\n",
      "Setting up libbdplus0:amd64 (0.1.2-2) ...\r\n",
      "Setting up libzvbi0:amd64 (0.2.35-13) ...\r\n",
      "Setting up libsamplerate0:amd64 (0.1.8-8+b2) ...\r\n",
      "Setting up libsndio6.1:amd64 (1.1.0-3) ...\r\n",
      "Setting up libtheora0:amd64 (1.1.1+dfsg.1-14+b1) ...\r\n",
      "Setting up libmpg123-0:amd64 (1.23.8-1+b1) ...\r\n",
      "Setting up libslang2:amd64 (2.3.1-5) ...\r\n",
      "Setting up libopencv-imgproc2.4v5:amd64 (2.4.9.1+dfsg1-2) ...\r\n",
      "Setting up libva-x11-1:amd64 (1.7.3-2) ...\r\n",
      "Setting up libx265-95:amd64 (2.1-2+b2) ...\r\n",
      "Setting up libzmq5:amd64 (4.2.1-4+deb9u1) ...\r\n",
      "Setting up libavutil55:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libopenmpt0:amd64 (0.2.7386~beta20.3-3+deb9u3) ...\r\n",
      "Setting up libvdpau-va-gl1:amd64 (0.4.2-1) ...\r\n",
      "Setting up mesa-vdpau-drivers:amd64 (13.0.6-1+b2) ...\r\n",
      "Setting up libdc1394-22:amd64 (2.2.5-1) ...\r\n",
      "Setting up libswresample2:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libass5:amd64 (1:0.13.4-2) ...\r\n",
      "Setting up librubberband2:amd64 (1.8.1-7) ...\r\n",
      "Setting up libsdl2-2.0-0:amd64 (2.0.5+dfsg1-2) ...\r\n",
      "Setting up libswscale4:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up va-driver-all:amd64 (1.7.3-2) ...\r\n",
      "Setting up libpostproc54:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libjack-jackd2-0:amd64 (1.9.10+20150825git1ed50c92~dfsg-5) ...\r\n",
      "Setting up libopenal1:amd64 (1:1.17.2-4+b2) ...\r\n",
      "Setting up libcaca0:amd64 (0.99.beta19-2+b2) ...\r\n",
      "Setting up vdpau-driver-all:amd64 (1.1.1-6) ...\r\n",
      "Setting up libavresample3:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libavcodec57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libchromaprint1:amd64 (1.4.2-1) ...\r\n",
      "Setting up libavformat57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libavfilter6:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up libavdevice57:amd64 (7:3.2.12-1~deb9u1) ...\r\n",
      "Setting up ffmpeg (7:3.2.12-1~deb9u1) ...\r\n",
      "Processing triggers for libc-bin (2.24-11+deb9u3) ...\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libcuda.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.410.104 is empty, not checked.\r\n",
      "ldconfig: File /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.410.104 is empty, not checked.\r\n"
     ]
    }
   ],
   "source": [
    "# Set-up the virtual display environment\n",
    "!apt-get update\n",
    "!apt-get install python-opengl -y\n",
    "!apt install xvfb -y\n",
    "!pip install pyvirtualdisplay\n",
    "!pip install piglet\n",
    "!apt-get install ffmpeg -y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "3a08acc9ed518d2f31ee70e064ad1ce377631726"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Display cmd_param=['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '1400x900x24', ':1001'] cmd=['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '1400x900x24', ':1001'] oserror=None return_code=None stdout=\"None\" stderr=\"None\" timeout_happened=False>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Start the virtual monitor\n",
    "from pyvirtualdisplay import Display\n",
    "display = Display(visible=0, size=(1400, 900))\n",
    "display.start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "a12ab3b72f2d3b17c4f960baa70d6f98dc3b47aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/working\r\n",
      "__notebook__.ipynb\r\n",
      "__output__.json\r\n",
      "openaigym.episode_batch.0.19.stats.json\r\n",
      "openaigym.manifest.0.19.manifest.json\r\n",
      "openaigym.video.0.19.video000000.meta.json\r\n",
      "openaigym.video.0.19.video000000.mp4\r\n"
     ]
    }
   ],
   "source": [
    "# play a random game and create video\n",
    "env = gym.make(\"MsPacmanNoFrameskip-v4\")\n",
    "monitor_dir = os.getcwd()\n",
    "\n",
    "#Setup a wrapper to be able to record a video of the game\n",
    "record_video = True\n",
    "should_record = lambda i: record_video\n",
    "env = wrappers.Monitor(env, monitor_dir, video_callable=should_record, force=True)\n",
    "\n",
    "#Play a random game\n",
    "state = env.reset()\n",
    "done = False\n",
    "while not done:\n",
    "  action = env.action_space.sample() #random action, replace by the prediction of the model\n",
    "  state, reward, done, _ = env.step(action)\n",
    "\n",
    "record_video = False\n",
    "env.close() \n",
    "\n",
    "# download videos\n",
    "#from google.colab import files\n",
    "#import glob\n",
    "os.chdir(monitor_dir) # change directory to get the files\n",
    "!pwd #show file path\n",
    "!ls # show directory content\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "64d28244b7a5943b633d219ebb70d98308e4b7b4"
   },
   "source": [
    "## Play video\n",
    "\n",
    "The MP4 file is already generated and is part of the output files. Go to your navigation bar at the top/left of the screen to the tab Output and click on the .mp4 file. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "39ce9444329ff0437492947fa233b049ab01ed70"
   },
   "source": [
    "## Reinforcement learning training\n",
    "After the first publication of DQN many deeplearning Reinforcement Learning algorithms have been invented/tried, Some main ones in chronological order: DQN, Double DQN, Duelling DQN, Deep Deterministic Policy Gradient, Continuous DQN (CDQN or NAF) , A2C/A3C, Proximal Policy Optimization Algorithms, ARS, etc, etc. \n",
    "\n",
    "For the actual (long, millions of game screens) training I would like to recommend [colaboratory](https://colab.research.google.com/notebooks/welcome.ipynb) solutions, since Kaggle is owned by Alphabet I presume they don't mind I refer to Google colab solutions. Will keep posting some links over here:\n",
    "\n",
    "Bipedal walker  with Augmented Random Search: \n",
    "* Colab: https://colab.research.google.com/drive/1NxvslFQ6RDLBirlOYH4b77zabSW_x5DM   \n",
    "* Video: \thttps://www.youtube.com/watch?v=NJBgeOF5CnM  \n",
    "\n",
    "Ms Pacman with Proximal Policy Optimization:   \n",
    "* Colab: https://colab.research.google.com/drive/1aSoqbO_wysvciYfDv4hJbL6ZS7lXFnMN   \n",
    "* video after training 20 Million game screens:  https://youtu.be/um2XX5bktMA  )  \n",
    "\n",
    "\n",
    "Hoped you liked my notebook (upvote top right), my way to conribute back to this fantastic Kaggle platform and community."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}