learn-by-example-reinforcement-learning-with-gym.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "collapsed": true
   },
   "source": [
    "Welcome to my third notebook on Kaggle. I did record my notes so it might help others in their journey to understand Neural Networks by examples (in this case Reinforcement Learning with Gym from OpenAI).  \n",
    "Reinforcement learning is the process of learning by interacting with an environment. Reinforcement learning is also blessed with a lot of history and hence terminology, that does not always make it easier. So I have tried to put it into context so I can understand and hopefully others as well. It does not intend to give a complete history lesson but the key concepts are important to understand because they are often referred to in modern neural network reinforcement learning. Have fun!  \n",
    "\n",
    "This notebook is my way to contribute back to the Kaggle platform and A.I. community.\n",
    "\n",
    "I would like to acknowledge this tutorial for providing ideas and code, learning by example:  \n",
    "https://www.oreilly.com/learning/introduction-to-reinforcement-learning-and-openai-gym  \n",
    "Also special thanks to Move 37 course to enrich my understanding/knowledge:  \n",
    "https://www.theschool.ai/courses/move-37-course/  \n",
    "If you are new to Neural Networks you might want to have a look at my other notebooks:  \n",
    "https://www.kaggle.com/charel/learn-neural-networks-by-example-mnist-digits  \n",
    "https://www.kaggle.com/charel/learn-by-example-rnn-lstm-gru-time-series  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OK\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import gym # openAi gym\n",
    "from gym import envs\n",
    "import numpy as np \n",
    "import datetime\n",
    "import keras \n",
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "from time import sleep\n",
    "\n",
    "from rl.agents.dqn import DQNAgent\n",
    "from rl.policy import LinearAnnealedPolicy, BoltzmannQPolicy, EpsGreedyQPolicy\n",
    "from rl.memory import SequentialMemory\n",
    "from rl.core import Processor\n",
    "from rl.callbacks import FileLogger, ModelIntervalCheckpoint\n",
    "print(\"OK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a371168b9fe9d579fd65f95a808998ba009127d7"
   },
   "source": [
    "# Gym  \n",
    "Gym is released by Open AI in 2016 (http://gym.openai.com/docs/). It is a toolkit for developing and comparing reinforcement learning algorithms. OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity.\n",
    "<img src=\"https://i.imgur.com/ria9HOm.jpg width=300\" width=\"800px\">\n",
    "\n",
    "Source: [OpenAI](https://openai.com/)\n",
    "\n",
    "In 2018 Gym-retro was released as its successor: https://blog.openai.com/gym-retro/\n",
    "\n",
    "\n",
    "There are many many games made available.  Let's review all the possible game environments in gym, see list below.  \n",
    "\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_uuid": "43f7aa21cb73928f74f64a6bb768c67784beddab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_values([EnvSpec(Copy-v0), EnvSpec(RepeatCopy-v0), EnvSpec(ReversedAddition-v0), EnvSpec(ReversedAddition3-v0), EnvSpec(DuplicatedInput-v0), EnvSpec(Reverse-v0), EnvSpec(CartPole-v0), EnvSpec(CartPole-v1), EnvSpec(MountainCar-v0), EnvSpec(MountainCarContinuous-v0), EnvSpec(Pendulum-v0), EnvSpec(Acrobot-v1), EnvSpec(LunarLander-v2), EnvSpec(LunarLanderContinuous-v2), EnvSpec(BipedalWalker-v2), EnvSpec(BipedalWalkerHardcore-v2), EnvSpec(CarRacing-v0), EnvSpec(Blackjack-v0), EnvSpec(KellyCoinflip-v0), EnvSpec(KellyCoinflipGeneralized-v0), EnvSpec(FrozenLake-v0), EnvSpec(FrozenLake8x8-v0), EnvSpec(CliffWalking-v0), EnvSpec(NChain-v0), EnvSpec(Roulette-v0), EnvSpec(Taxi-v2), EnvSpec(GuessingGame-v0), EnvSpec(HotterColder-v0), EnvSpec(Reacher-v2), EnvSpec(Pusher-v2), EnvSpec(Thrower-v2), EnvSpec(Striker-v2), EnvSpec(InvertedPendulum-v2), EnvSpec(InvertedDoublePendulum-v2), EnvSpec(HalfCheetah-v2), EnvSpec(Hopper-v2), EnvSpec(Swimmer-v2), EnvSpec(Walker2d-v2), EnvSpec(Ant-v2), EnvSpec(Humanoid-v2), EnvSpec(HumanoidStandup-v2), EnvSpec(FetchSlide-v1), EnvSpec(FetchPickAndPlace-v1), EnvSpec(FetchReach-v1), EnvSpec(FetchPush-v1), EnvSpec(HandReach-v0), EnvSpec(HandManipulateBlockRotateZ-v0), EnvSpec(HandManipulateBlockRotateParallel-v0), EnvSpec(HandManipulateBlockRotateXYZ-v0), EnvSpec(HandManipulateBlockFull-v0), EnvSpec(HandManipulateBlock-v0), EnvSpec(HandManipulateEggRotate-v0), EnvSpec(HandManipulateEggFull-v0), EnvSpec(HandManipulateEgg-v0), EnvSpec(HandManipulatePenRotate-v0), EnvSpec(HandManipulatePenFull-v0), EnvSpec(HandManipulatePen-v0), EnvSpec(FetchSlideDense-v1), EnvSpec(FetchPickAndPlaceDense-v1), EnvSpec(FetchReachDense-v1), EnvSpec(FetchPushDense-v1), EnvSpec(HandReachDense-v0), EnvSpec(HandManipulateBlockRotateZDense-v0), EnvSpec(HandManipulateBlockRotateParallelDense-v0), EnvSpec(HandManipulateBlockRotateXYZDense-v0), EnvSpec(HandManipulateBlockFullDense-v0), EnvSpec(HandManipulateBlockDense-v0), EnvSpec(HandManipulateEggRotateDense-v0), EnvSpec(HandManipulateEggFullDense-v0), EnvSpec(HandManipulateEggDense-v0), EnvSpec(HandManipulatePenRotateDense-v0), EnvSpec(HandManipulatePenFullDense-v0), EnvSpec(HandManipulatePenDense-v0), EnvSpec(AirRaid-v0), EnvSpec(AirRaid-v4), EnvSpec(AirRaidDeterministic-v0), EnvSpec(AirRaidDeterministic-v4), EnvSpec(AirRaidNoFrameskip-v0), EnvSpec(AirRaidNoFrameskip-v4), EnvSpec(AirRaid-ram-v0), EnvSpec(AirRaid-ram-v4), EnvSpec(AirRaid-ramDeterministic-v0), EnvSpec(AirRaid-ramDeterministic-v4), EnvSpec(AirRaid-ramNoFrameskip-v0), EnvSpec(AirRaid-ramNoFrameskip-v4), EnvSpec(Alien-v0), EnvSpec(Alien-v4), EnvSpec(AlienDeterministic-v0), EnvSpec(AlienDeterministic-v4), EnvSpec(AlienNoFrameskip-v0), EnvSpec(AlienNoFrameskip-v4), EnvSpec(Alien-ram-v0), EnvSpec(Alien-ram-v4), EnvSpec(Alien-ramDeterministic-v0), EnvSpec(Alien-ramDeterministic-v4), EnvSpec(Alien-ramNoFrameskip-v0), EnvSpec(Alien-ramNoFrameskip-v4), EnvSpec(Amidar-v0), EnvSpec(Amidar-v4), EnvSpec(AmidarDeterministic-v0), EnvSpec(AmidarDeterministic-v4), EnvSpec(AmidarNoFrameskip-v0), EnvSpec(AmidarNoFrameskip-v4), EnvSpec(Amidar-ram-v0), EnvSpec(Amidar-ram-v4), EnvSpec(Amidar-ramDeterministic-v0), EnvSpec(Amidar-ramDeterministic-v4), EnvSpec(Amidar-ramNoFrameskip-v0), EnvSpec(Amidar-ramNoFrameskip-v4), EnvSpec(Assault-v0), EnvSpec(Assault-v4), EnvSpec(AssaultDeterministic-v0), EnvSpec(AssaultDeterministic-v4), EnvSpec(AssaultNoFrameskip-v0), EnvSpec(AssaultNoFrameskip-v4), EnvSpec(Assault-ram-v0), EnvSpec(Assault-ram-v4), EnvSpec(Assault-ramDeterministic-v0), EnvSpec(Assault-ramDeterministic-v4), EnvSpec(Assault-ramNoFrameskip-v0), EnvSpec(Assault-ramNoFrameskip-v4), EnvSpec(Asterix-v0), EnvSpec(Asterix-v4), EnvSpec(AsterixDeterministic-v0), EnvSpec(AsterixDeterministic-v4), EnvSpec(AsterixNoFrameskip-v0), EnvSpec(AsterixNoFrameskip-v4), EnvSpec(Asterix-ram-v0), EnvSpec(Asterix-ram-v4), EnvSpec(Asterix-ramDeterministic-v0), EnvSpec(Asterix-ramDeterministic-v4), EnvSpec(Asterix-ramNoFrameskip-v0), EnvSpec(Asterix-ramNoFrameskip-v4), EnvSpec(Asteroids-v0), EnvSpec(Asteroids-v4), EnvSpec(AsteroidsDeterministic-v0), EnvSpec(AsteroidsDeterministic-v4), EnvSpec(AsteroidsNoFrameskip-v0), EnvSpec(AsteroidsNoFrameskip-v4), EnvSpec(Asteroids-ram-v0), EnvSpec(Asteroids-ram-v4), EnvSpec(Asteroids-ramDeterministic-v0), EnvSpec(Asteroids-ramDeterministic-v4), EnvSpec(Asteroids-ramNoFrameskip-v0), EnvSpec(Asteroids-ramNoFrameskip-v4), EnvSpec(Atlantis-v0), EnvSpec(Atlantis-v4), EnvSpec(AtlantisDeterministic-v0), EnvSpec(AtlantisDeterministic-v4), EnvSpec(AtlantisNoFrameskip-v0), EnvSpec(AtlantisNoFrameskip-v4), EnvSpec(Atlantis-ram-v0), EnvSpec(Atlantis-ram-v4), EnvSpec(Atlantis-ramDeterministic-v0), EnvSpec(Atlantis-ramDeterministic-v4), EnvSpec(Atlantis-ramNoFrameskip-v0), EnvSpec(Atlantis-ramNoFrameskip-v4), EnvSpec(BankHeist-v0), EnvSpec(BankHeist-v4), EnvSpec(BankHeistDeterministic-v0), EnvSpec(BankHeistDeterministic-v4), EnvSpec(BankHeistNoFrameskip-v0), EnvSpec(BankHeistNoFrameskip-v4), EnvSpec(BankHeist-ram-v0), EnvSpec(BankHeist-ram-v4), EnvSpec(BankHeist-ramDeterministic-v0), EnvSpec(BankHeist-ramDeterministic-v4), EnvSpec(BankHeist-ramNoFrameskip-v0), EnvSpec(BankHeist-ramNoFrameskip-v4), EnvSpec(BattleZone-v0), EnvSpec(BattleZone-v4), EnvSpec(BattleZoneDeterministic-v0), EnvSpec(BattleZoneDeterministic-v4), EnvSpec(BattleZoneNoFrameskip-v0), EnvSpec(BattleZoneNoFrameskip-v4), EnvSpec(BattleZone-ram-v0), EnvSpec(BattleZone-ram-v4), EnvSpec(BattleZone-ramDeterministic-v0), EnvSpec(BattleZone-ramDeterministic-v4), EnvSpec(BattleZone-ramNoFrameskip-v0), EnvSpec(BattleZone-ramNoFrameskip-v4), EnvSpec(BeamRider-v0), EnvSpec(BeamRider-v4), EnvSpec(BeamRiderDeterministic-v0), EnvSpec(BeamRiderDeterministic-v4), EnvSpec(BeamRiderNoFrameskip-v0), EnvSpec(BeamRiderNoFrameskip-v4), EnvSpec(BeamRider-ram-v0), EnvSpec(BeamRider-ram-v4), EnvSpec(BeamRider-ramDeterministic-v0), EnvSpec(BeamRider-ramDeterministic-v4), EnvSpec(BeamRider-ramNoFrameskip-v0), EnvSpec(BeamRider-ramNoFrameskip-v4), EnvSpec(Berzerk-v0), EnvSpec(Berzerk-v4), EnvSpec(BerzerkDeterministic-v0), EnvSpec(BerzerkDeterministic-v4), EnvSpec(BerzerkNoFrameskip-v0), EnvSpec(BerzerkNoFrameskip-v4), EnvSpec(Berzerk-ram-v0), EnvSpec(Berzerk-ram-v4), EnvSpec(Berzerk-ramDeterministic-v0), EnvSpec(Berzerk-ramDeterministic-v4), EnvSpec(Berzerk-ramNoFrameskip-v0), EnvSpec(Berzerk-ramNoFrameskip-v4), EnvSpec(Bowling-v0), EnvSpec(Bowling-v4), EnvSpec(BowlingDeterministic-v0), EnvSpec(BowlingDeterministic-v4), EnvSpec(BowlingNoFrameskip-v0), EnvSpec(BowlingNoFrameskip-v4), EnvSpec(Bowling-ram-v0), EnvSpec(Bowling-ram-v4), EnvSpec(Bowling-ramDeterministic-v0), EnvSpec(Bowling-ramDeterministic-v4), EnvSpec(Bowling-ramNoFrameskip-v0), EnvSpec(Bowling-ramNoFrameskip-v4), EnvSpec(Boxing-v0), EnvSpec(Boxing-v4), EnvSpec(BoxingDeterministic-v0), EnvSpec(BoxingDeterministic-v4), EnvSpec(BoxingNoFrameskip-v0), EnvSpec(BoxingNoFrameskip-v4), EnvSpec(Boxing-ram-v0), EnvSpec(Boxing-ram-v4), EnvSpec(Boxing-ramDeterministic-v0), EnvSpec(Boxing-ramDeterministic-v4), EnvSpec(Boxing-ramNoFrameskip-v0), EnvSpec(Boxing-ramNoFrameskip-v4), EnvSpec(Breakout-v0), EnvSpec(Breakout-v4), EnvSpec(BreakoutDeterministic-v0), EnvSpec(BreakoutDeterministic-v4), EnvSpec(BreakoutNoFrameskip-v0), EnvSpec(BreakoutNoFrameskip-v4), EnvSpec(Breakout-ram-v0), EnvSpec(Breakout-ram-v4), EnvSpec(Breakout-ramDeterministic-v0), EnvSpec(Breakout-ramDeterministic-v4), EnvSpec(Breakout-ramNoFrameskip-v0), EnvSpec(Breakout-ramNoFrameskip-v4), EnvSpec(Carnival-v0), EnvSpec(Carnival-v4), EnvSpec(CarnivalDeterministic-v0), EnvSpec(CarnivalDeterministic-v4), EnvSpec(CarnivalNoFrameskip-v0), EnvSpec(CarnivalNoFrameskip-v4), EnvSpec(Carnival-ram-v0), EnvSpec(Carnival-ram-v4), EnvSpec(Carnival-ramDeterministic-v0), EnvSpec(Carnival-ramDeterministic-v4), EnvSpec(Carnival-ramNoFrameskip-v0), EnvSpec(Carnival-ramNoFrameskip-v4), EnvSpec(Centipede-v0), EnvSpec(Centipede-v4), EnvSpec(CentipedeDeterministic-v0), EnvSpec(CentipedeDeterministic-v4), EnvSpec(CentipedeNoFrameskip-v0), EnvSpec(CentipedeNoFrameskip-v4), EnvSpec(Centipede-ram-v0), EnvSpec(Centipede-ram-v4), EnvSpec(Centipede-ramDeterministic-v0), EnvSpec(Centipede-ramDeterministic-v4), EnvSpec(Centipede-ramNoFrameskip-v0), EnvSpec(Centipede-ramNoFrameskip-v4), EnvSpec(ChopperCommand-v0), EnvSpec(ChopperCommand-v4), EnvSpec(ChopperCommandDeterministic-v0), EnvSpec(ChopperCommandDeterministic-v4), EnvSpec(ChopperCommandNoFrameskip-v0), EnvSpec(ChopperCommandNoFrameskip-v4), EnvSpec(ChopperCommand-ram-v0), EnvSpec(ChopperCommand-ram-v4), EnvSpec(ChopperCommand-ramDeterministic-v0), EnvSpec(ChopperCommand-ramDeterministic-v4), EnvSpec(ChopperCommand-ramNoFrameskip-v0), EnvSpec(ChopperCommand-ramNoFrameskip-v4), EnvSpec(CrazyClimber-v0), EnvSpec(CrazyClimber-v4), EnvSpec(CrazyClimberDeterministic-v0), EnvSpec(CrazyClimberDeterministic-v4), EnvSpec(CrazyClimberNoFrameskip-v0), EnvSpec(CrazyClimberNoFrameskip-v4), EnvSpec(CrazyClimber-ram-v0), EnvSpec(CrazyClimber-ram-v4), EnvSpec(CrazyClimber-ramDeterministic-v0), EnvSpec(CrazyClimber-ramDeterministic-v4), EnvSpec(CrazyClimber-ramNoFrameskip-v0), EnvSpec(CrazyClimber-ramNoFrameskip-v4), EnvSpec(DemonAttack-v0), EnvSpec(DemonAttack-v4), EnvSpec(DemonAttackDeterministic-v0), EnvSpec(DemonAttackDeterministic-v4), EnvSpec(DemonAttackNoFrameskip-v0), EnvSpec(DemonAttackNoFrameskip-v4), EnvSpec(DemonAttack-ram-v0), EnvSpec(DemonAttack-ram-v4), EnvSpec(DemonAttack-ramDeterministic-v0), EnvSpec(DemonAttack-ramDeterministic-v4), EnvSpec(DemonAttack-ramNoFrameskip-v0), EnvSpec(DemonAttack-ramNoFrameskip-v4), EnvSpec(DoubleDunk-v0), EnvSpec(DoubleDunk-v4), EnvSpec(DoubleDunkDeterministic-v0), EnvSpec(DoubleDunkDeterministic-v4), EnvSpec(DoubleDunkNoFrameskip-v0), EnvSpec(DoubleDunkNoFrameskip-v4), EnvSpec(DoubleDunk-ram-v0), EnvSpec(DoubleDunk-ram-v4), EnvSpec(DoubleDunk-ramDeterministic-v0), EnvSpec(DoubleDunk-ramDeterministic-v4), EnvSpec(DoubleDunk-ramNoFrameskip-v0), EnvSpec(DoubleDunk-ramNoFrameskip-v4), EnvSpec(ElevatorAction-v0), EnvSpec(ElevatorAction-v4), EnvSpec(ElevatorActionDeterministic-v0), EnvSpec(ElevatorActionDeterministic-v4), EnvSpec(ElevatorActionNoFrameskip-v0), EnvSpec(ElevatorActionNoFrameskip-v4), EnvSpec(ElevatorAction-ram-v0), EnvSpec(ElevatorAction-ram-v4), EnvSpec(ElevatorAction-ramDeterministic-v0), EnvSpec(ElevatorAction-ramDeterministic-v4), EnvSpec(ElevatorAction-ramNoFrameskip-v0), EnvSpec(ElevatorAction-ramNoFrameskip-v4), EnvSpec(Enduro-v0), EnvSpec(Enduro-v4), EnvSpec(EnduroDeterministic-v0), EnvSpec(EnduroDeterministic-v4), EnvSpec(EnduroNoFrameskip-v0), EnvSpec(EnduroNoFrameskip-v4), EnvSpec(Enduro-ram-v0), EnvSpec(Enduro-ram-v4), EnvSpec(Enduro-ramDeterministic-v0), EnvSpec(Enduro-ramDeterministic-v4), EnvSpec(Enduro-ramNoFrameskip-v0), EnvSpec(Enduro-ramNoFrameskip-v4), EnvSpec(FishingDerby-v0), EnvSpec(FishingDerby-v4), EnvSpec(FishingDerbyDeterministic-v0), EnvSpec(FishingDerbyDeterministic-v4), EnvSpec(FishingDerbyNoFrameskip-v0), EnvSpec(FishingDerbyNoFrameskip-v4), EnvSpec(FishingDerby-ram-v0), EnvSpec(FishingDerby-ram-v4), EnvSpec(FishingDerby-ramDeterministic-v0), EnvSpec(FishingDerby-ramDeterministic-v4), EnvSpec(FishingDerby-ramNoFrameskip-v0), EnvSpec(FishingDerby-ramNoFrameskip-v4), EnvSpec(Freeway-v0), EnvSpec(Freeway-v4), EnvSpec(FreewayDeterministic-v0), EnvSpec(FreewayDeterministic-v4), EnvSpec(FreewayNoFrameskip-v0), EnvSpec(FreewayNoFrameskip-v4), EnvSpec(Freeway-ram-v0), EnvSpec(Freeway-ram-v4), EnvSpec(Freeway-ramDeterministic-v0), EnvSpec(Freeway-ramDeterministic-v4), EnvSpec(Freeway-ramNoFrameskip-v0), EnvSpec(Freeway-ramNoFrameskip-v4), EnvSpec(Frostbite-v0), EnvSpec(Frostbite-v4), EnvSpec(FrostbiteDeterministic-v0), EnvSpec(FrostbiteDeterministic-v4), EnvSpec(FrostbiteNoFrameskip-v0), EnvSpec(FrostbiteNoFrameskip-v4), EnvSpec(Frostbite-ram-v0), EnvSpec(Frostbite-ram-v4), EnvSpec(Frostbite-ramDeterministic-v0), EnvSpec(Frostbite-ramDeterministic-v4), EnvSpec(Frostbite-ramNoFrameskip-v0), EnvSpec(Frostbite-ramNoFrameskip-v4), EnvSpec(Gopher-v0), EnvSpec(Gopher-v4), EnvSpec(GopherDeterministic-v0), EnvSpec(GopherDeterministic-v4), EnvSpec(GopherNoFrameskip-v0), EnvSpec(GopherNoFrameskip-v4), EnvSpec(Gopher-ram-v0), EnvSpec(Gopher-ram-v4), EnvSpec(Gopher-ramDeterministic-v0), EnvSpec(Gopher-ramDeterministic-v4), EnvSpec(Gopher-ramNoFrameskip-v0), EnvSpec(Gopher-ramNoFrameskip-v4), EnvSpec(Gravitar-v0), EnvSpec(Gravitar-v4), EnvSpec(GravitarDeterministic-v0), EnvSpec(GravitarDeterministic-v4), EnvSpec(GravitarNoFrameskip-v0), EnvSpec(GravitarNoFrameskip-v4), EnvSpec(Gravitar-ram-v0), EnvSpec(Gravitar-ram-v4), EnvSpec(Gravitar-ramDeterministic-v0), EnvSpec(Gravitar-ramDeterministic-v4), EnvSpec(Gravitar-ramNoFrameskip-v0), EnvSpec(Gravitar-ramNoFrameskip-v4), EnvSpec(Hero-v0), EnvSpec(Hero-v4), EnvSpec(HeroDeterministic-v0), EnvSpec(HeroDeterministic-v4), EnvSpec(HeroNoFrameskip-v0), EnvSpec(HeroNoFrameskip-v4), EnvSpec(Hero-ram-v0), EnvSpec(Hero-ram-v4), EnvSpec(Hero-ramDeterministic-v0), EnvSpec(Hero-ramDeterministic-v4), EnvSpec(Hero-ramNoFrameskip-v0), EnvSpec(Hero-ramNoFrameskip-v4), EnvSpec(IceHockey-v0), EnvSpec(IceHockey-v4), EnvSpec(IceHockeyDeterministic-v0), EnvSpec(IceHockeyDeterministic-v4), EnvSpec(IceHockeyNoFrameskip-v0), EnvSpec(IceHockeyNoFrameskip-v4), EnvSpec(IceHockey-ram-v0), EnvSpec(IceHockey-ram-v4), EnvSpec(IceHockey-ramDeterministic-v0), EnvSpec(IceHockey-ramDeterministic-v4), EnvSpec(IceHockey-ramNoFrameskip-v0), EnvSpec(IceHockey-ramNoFrameskip-v4), EnvSpec(Jamesbond-v0), EnvSpec(Jamesbond-v4), EnvSpec(JamesbondDeterministic-v0), EnvSpec(JamesbondDeterministic-v4), EnvSpec(JamesbondNoFrameskip-v0), EnvSpec(JamesbondNoFrameskip-v4), EnvSpec(Jamesbond-ram-v0), EnvSpec(Jamesbond-ram-v4), EnvSpec(Jamesbond-ramDeterministic-v0), EnvSpec(Jamesbond-ramDeterministic-v4), EnvSpec(Jamesbond-ramNoFrameskip-v0), EnvSpec(Jamesbond-ramNoFrameskip-v4), EnvSpec(JourneyEscape-v0), EnvSpec(JourneyEscape-v4), EnvSpec(JourneyEscapeDeterministic-v0), EnvSpec(JourneyEscapeDeterministic-v4), EnvSpec(JourneyEscapeNoFrameskip-v0), EnvSpec(JourneyEscapeNoFrameskip-v4), EnvSpec(JourneyEscape-ram-v0), EnvSpec(JourneyEscape-ram-v4), EnvSpec(JourneyEscape-ramDeterministic-v0), EnvSpec(JourneyEscape-ramDeterministic-v4), EnvSpec(JourneyEscape-ramNoFrameskip-v0), EnvSpec(JourneyEscape-ramNoFrameskip-v4), EnvSpec(Kangaroo-v0), EnvSpec(Kangaroo-v4), EnvSpec(KangarooDeterministic-v0), EnvSpec(KangarooDeterministic-v4), EnvSpec(KangarooNoFrameskip-v0), EnvSpec(KangarooNoFrameskip-v4), EnvSpec(Kangaroo-ram-v0), EnvSpec(Kangaroo-ram-v4), EnvSpec(Kangaroo-ramDeterministic-v0), EnvSpec(Kangaroo-ramDeterministic-v4), EnvSpec(Kangaroo-ramNoFrameskip-v0), EnvSpec(Kangaroo-ramNoFrameskip-v4), EnvSpec(Krull-v0), EnvSpec(Krull-v4), EnvSpec(KrullDeterministic-v0), EnvSpec(KrullDeterministic-v4), EnvSpec(KrullNoFrameskip-v0), EnvSpec(KrullNoFrameskip-v4), EnvSpec(Krull-ram-v0), EnvSpec(Krull-ram-v4), EnvSpec(Krull-ramDeterministic-v0), EnvSpec(Krull-ramDeterministic-v4), EnvSpec(Krull-ramNoFrameskip-v0), EnvSpec(Krull-ramNoFrameskip-v4), EnvSpec(KungFuMaster-v0), EnvSpec(KungFuMaster-v4), EnvSpec(KungFuMasterDeterministic-v0), EnvSpec(KungFuMasterDeterministic-v4), EnvSpec(KungFuMasterNoFrameskip-v0), EnvSpec(KungFuMasterNoFrameskip-v4), EnvSpec(KungFuMaster-ram-v0), EnvSpec(KungFuMaster-ram-v4), EnvSpec(KungFuMaster-ramDeterministic-v0), EnvSpec(KungFuMaster-ramDeterministic-v4), EnvSpec(KungFuMaster-ramNoFrameskip-v0), EnvSpec(KungFuMaster-ramNoFrameskip-v4), EnvSpec(MontezumaRevenge-v0), EnvSpec(MontezumaRevenge-v4), EnvSpec(MontezumaRevengeDeterministic-v0), EnvSpec(MontezumaRevengeDeterministic-v4), EnvSpec(MontezumaRevengeNoFrameskip-v0), EnvSpec(MontezumaRevengeNoFrameskip-v4), EnvSpec(MontezumaRevenge-ram-v0), EnvSpec(MontezumaRevenge-ram-v4), EnvSpec(MontezumaRevenge-ramDeterministic-v0), EnvSpec(MontezumaRevenge-ramDeterministic-v4), EnvSpec(MontezumaRevenge-ramNoFrameskip-v0), EnvSpec(MontezumaRevenge-ramNoFrameskip-v4), EnvSpec(MsPacman-v0), EnvSpec(MsPacman-v4), EnvSpec(MsPacmanDeterministic-v0), EnvSpec(MsPacmanDeterministic-v4), EnvSpec(MsPacmanNoFrameskip-v0), EnvSpec(MsPacmanNoFrameskip-v4), EnvSpec(MsPacman-ram-v0), EnvSpec(MsPacman-ram-v4), EnvSpec(MsPacman-ramDeterministic-v0), EnvSpec(MsPacman-ramDeterministic-v4), EnvSpec(MsPacman-ramNoFrameskip-v0), EnvSpec(MsPacman-ramNoFrameskip-v4), EnvSpec(NameThisGame-v0), EnvSpec(NameThisGame-v4), EnvSpec(NameThisGameDeterministic-v0), EnvSpec(NameThisGameDeterministic-v4), EnvSpec(NameThisGameNoFrameskip-v0), EnvSpec(NameThisGameNoFrameskip-v4), EnvSpec(NameThisGame-ram-v0), EnvSpec(NameThisGame-ram-v4), EnvSpec(NameThisGame-ramDeterministic-v0), EnvSpec(NameThisGame-ramDeterministic-v4), EnvSpec(NameThisGame-ramNoFrameskip-v0), EnvSpec(NameThisGame-ramNoFrameskip-v4), EnvSpec(Phoenix-v0), EnvSpec(Phoenix-v4), EnvSpec(PhoenixDeterministic-v0), EnvSpec(PhoenixDeterministic-v4), EnvSpec(PhoenixNoFrameskip-v0), EnvSpec(PhoenixNoFrameskip-v4), EnvSpec(Phoenix-ram-v0), EnvSpec(Phoenix-ram-v4), EnvSpec(Phoenix-ramDeterministic-v0), EnvSpec(Phoenix-ramDeterministic-v4), EnvSpec(Phoenix-ramNoFrameskip-v0), EnvSpec(Phoenix-ramNoFrameskip-v4), EnvSpec(Pitfall-v0), EnvSpec(Pitfall-v4), EnvSpec(PitfallDeterministic-v0), EnvSpec(PitfallDeterministic-v4), EnvSpec(PitfallNoFrameskip-v0), EnvSpec(PitfallNoFrameskip-v4), EnvSpec(Pitfall-ram-v0), EnvSpec(Pitfall-ram-v4), EnvSpec(Pitfall-ramDeterministic-v0), EnvSpec(Pitfall-ramDeterministic-v4), EnvSpec(Pitfall-ramNoFrameskip-v0), EnvSpec(Pitfall-ramNoFrameskip-v4), EnvSpec(Pong-v0), EnvSpec(Pong-v4), EnvSpec(PongDeterministic-v0), EnvSpec(PongDeterministic-v4), EnvSpec(PongNoFrameskip-v0), EnvSpec(PongNoFrameskip-v4), EnvSpec(Pong-ram-v0), EnvSpec(Pong-ram-v4), EnvSpec(Pong-ramDeterministic-v0), EnvSpec(Pong-ramDeterministic-v4), EnvSpec(Pong-ramNoFrameskip-v0), EnvSpec(Pong-ramNoFrameskip-v4), EnvSpec(Pooyan-v0), EnvSpec(Pooyan-v4), EnvSpec(PooyanDeterministic-v0), EnvSpec(PooyanDeterministic-v4), EnvSpec(PooyanNoFrameskip-v0), EnvSpec(PooyanNoFrameskip-v4), EnvSpec(Pooyan-ram-v0), EnvSpec(Pooyan-ram-v4), EnvSpec(Pooyan-ramDeterministic-v0), EnvSpec(Pooyan-ramDeterministic-v4), EnvSpec(Pooyan-ramNoFrameskip-v0), EnvSpec(Pooyan-ramNoFrameskip-v4), EnvSpec(PrivateEye-v0), EnvSpec(PrivateEye-v4), EnvSpec(PrivateEyeDeterministic-v0), EnvSpec(PrivateEyeDeterministic-v4), EnvSpec(PrivateEyeNoFrameskip-v0), EnvSpec(PrivateEyeNoFrameskip-v4), EnvSpec(PrivateEye-ram-v0), EnvSpec(PrivateEye-ram-v4), EnvSpec(PrivateEye-ramDeterministic-v0), EnvSpec(PrivateEye-ramDeterministic-v4), EnvSpec(PrivateEye-ramNoFrameskip-v0), EnvSpec(PrivateEye-ramNoFrameskip-v4), EnvSpec(Qbert-v0), EnvSpec(Qbert-v4), EnvSpec(QbertDeterministic-v0), EnvSpec(QbertDeterministic-v4), EnvSpec(QbertNoFrameskip-v0), EnvSpec(QbertNoFrameskip-v4), EnvSpec(Qbert-ram-v0), EnvSpec(Qbert-ram-v4), EnvSpec(Qbert-ramDeterministic-v0), EnvSpec(Qbert-ramDeterministic-v4), EnvSpec(Qbert-ramNoFrameskip-v0), EnvSpec(Qbert-ramNoFrameskip-v4), EnvSpec(Riverraid-v0), EnvSpec(Riverraid-v4), EnvSpec(RiverraidDeterministic-v0), EnvSpec(RiverraidDeterministic-v4), EnvSpec(RiverraidNoFrameskip-v0), EnvSpec(RiverraidNoFrameskip-v4), EnvSpec(Riverraid-ram-v0), EnvSpec(Riverraid-ram-v4), EnvSpec(Riverraid-ramDeterministic-v0), EnvSpec(Riverraid-ramDeterministic-v4), EnvSpec(Riverraid-ramNoFrameskip-v0), EnvSpec(Riverraid-ramNoFrameskip-v4), EnvSpec(RoadRunner-v0), EnvSpec(RoadRunner-v4), EnvSpec(RoadRunnerDeterministic-v0), EnvSpec(RoadRunnerDeterministic-v4), EnvSpec(RoadRunnerNoFrameskip-v0), EnvSpec(RoadRunnerNoFrameskip-v4), EnvSpec(RoadRunner-ram-v0), EnvSpec(RoadRunner-ram-v4), EnvSpec(RoadRunner-ramDeterministic-v0), EnvSpec(RoadRunner-ramDeterministic-v4), EnvSpec(RoadRunner-ramNoFrameskip-v0), EnvSpec(RoadRunner-ramNoFrameskip-v4), EnvSpec(Robotank-v0), EnvSpec(Robotank-v4), EnvSpec(RobotankDeterministic-v0), EnvSpec(RobotankDeterministic-v4), EnvSpec(RobotankNoFrameskip-v0), EnvSpec(RobotankNoFrameskip-v4), EnvSpec(Robotank-ram-v0), EnvSpec(Robotank-ram-v4), EnvSpec(Robotank-ramDeterministic-v0), EnvSpec(Robotank-ramDeterministic-v4), EnvSpec(Robotank-ramNoFrameskip-v0), EnvSpec(Robotank-ramNoFrameskip-v4), EnvSpec(Seaquest-v0), EnvSpec(Seaquest-v4), EnvSpec(SeaquestDeterministic-v0), EnvSpec(SeaquestDeterministic-v4), EnvSpec(SeaquestNoFrameskip-v0), EnvSpec(SeaquestNoFrameskip-v4), EnvSpec(Seaquest-ram-v0), EnvSpec(Seaquest-ram-v4), EnvSpec(Seaquest-ramDeterministic-v0), EnvSpec(Seaquest-ramDeterministic-v4), EnvSpec(Seaquest-ramNoFrameskip-v0), EnvSpec(Seaquest-ramNoFrameskip-v4), EnvSpec(Skiing-v0), EnvSpec(Skiing-v4), EnvSpec(SkiingDeterministic-v0), EnvSpec(SkiingDeterministic-v4), EnvSpec(SkiingNoFrameskip-v0), EnvSpec(SkiingNoFrameskip-v4), EnvSpec(Skiing-ram-v0), EnvSpec(Skiing-ram-v4), EnvSpec(Skiing-ramDeterministic-v0), EnvSpec(Skiing-ramDeterministic-v4), EnvSpec(Skiing-ramNoFrameskip-v0), EnvSpec(Skiing-ramNoFrameskip-v4), EnvSpec(Solaris-v0), EnvSpec(Solaris-v4), EnvSpec(SolarisDeterministic-v0), EnvSpec(SolarisDeterministic-v4), EnvSpec(SolarisNoFrameskip-v0), EnvSpec(SolarisNoFrameskip-v4), EnvSpec(Solaris-ram-v0), EnvSpec(Solaris-ram-v4), EnvSpec(Solaris-ramDeterministic-v0), EnvSpec(Solaris-ramDeterministic-v4), EnvSpec(Solaris-ramNoFrameskip-v0), EnvSpec(Solaris-ramNoFrameskip-v4), EnvSpec(SpaceInvaders-v0), EnvSpec(SpaceInvaders-v4), EnvSpec(SpaceInvadersDeterministic-v0), EnvSpec(SpaceInvadersDeterministic-v4), EnvSpec(SpaceInvadersNoFrameskip-v0), EnvSpec(SpaceInvadersNoFrameskip-v4), EnvSpec(SpaceInvaders-ram-v0), EnvSpec(SpaceInvaders-ram-v4), EnvSpec(SpaceInvaders-ramDeterministic-v0), EnvSpec(SpaceInvaders-ramDeterministic-v4), EnvSpec(SpaceInvaders-ramNoFrameskip-v0), EnvSpec(SpaceInvaders-ramNoFrameskip-v4), EnvSpec(StarGunner-v0), EnvSpec(StarGunner-v4), EnvSpec(StarGunnerDeterministic-v0), EnvSpec(StarGunnerDeterministic-v4), EnvSpec(StarGunnerNoFrameskip-v0), EnvSpec(StarGunnerNoFrameskip-v4), EnvSpec(StarGunner-ram-v0), EnvSpec(StarGunner-ram-v4), EnvSpec(StarGunner-ramDeterministic-v0), EnvSpec(StarGunner-ramDeterministic-v4), EnvSpec(StarGunner-ramNoFrameskip-v0), EnvSpec(StarGunner-ramNoFrameskip-v4), EnvSpec(Tennis-v0), EnvSpec(Tennis-v4), EnvSpec(TennisDeterministic-v0), EnvSpec(TennisDeterministic-v4), EnvSpec(TennisNoFrameskip-v0), EnvSpec(TennisNoFrameskip-v4), EnvSpec(Tennis-ram-v0), EnvSpec(Tennis-ram-v4), EnvSpec(Tennis-ramDeterministic-v0), EnvSpec(Tennis-ramDeterministic-v4), EnvSpec(Tennis-ramNoFrameskip-v0), EnvSpec(Tennis-ramNoFrameskip-v4), EnvSpec(TimePilot-v0), EnvSpec(TimePilot-v4), EnvSpec(TimePilotDeterministic-v0), EnvSpec(TimePilotDeterministic-v4), EnvSpec(TimePilotNoFrameskip-v0), EnvSpec(TimePilotNoFrameskip-v4), EnvSpec(TimePilot-ram-v0), EnvSpec(TimePilot-ram-v4), EnvSpec(TimePilot-ramDeterministic-v0), EnvSpec(TimePilot-ramDeterministic-v4), EnvSpec(TimePilot-ramNoFrameskip-v0), EnvSpec(TimePilot-ramNoFrameskip-v4), EnvSpec(Tutankham-v0), EnvSpec(Tutankham-v4), EnvSpec(TutankhamDeterministic-v0), EnvSpec(TutankhamDeterministic-v4), EnvSpec(TutankhamNoFrameskip-v0), EnvSpec(TutankhamNoFrameskip-v4), EnvSpec(Tutankham-ram-v0), EnvSpec(Tutankham-ram-v4), EnvSpec(Tutankham-ramDeterministic-v0), EnvSpec(Tutankham-ramDeterministic-v4), EnvSpec(Tutankham-ramNoFrameskip-v0), EnvSpec(Tutankham-ramNoFrameskip-v4), EnvSpec(UpNDown-v0), EnvSpec(UpNDown-v4), EnvSpec(UpNDownDeterministic-v0), EnvSpec(UpNDownDeterministic-v4), EnvSpec(UpNDownNoFrameskip-v0), EnvSpec(UpNDownNoFrameskip-v4), EnvSpec(UpNDown-ram-v0), EnvSpec(UpNDown-ram-v4), EnvSpec(UpNDown-ramDeterministic-v0), EnvSpec(UpNDown-ramDeterministic-v4), EnvSpec(UpNDown-ramNoFrameskip-v0), EnvSpec(UpNDown-ramNoFrameskip-v4), EnvSpec(Venture-v0), EnvSpec(Venture-v4), EnvSpec(VentureDeterministic-v0), EnvSpec(VentureDeterministic-v4), EnvSpec(VentureNoFrameskip-v0), EnvSpec(VentureNoFrameskip-v4), EnvSpec(Venture-ram-v0), EnvSpec(Venture-ram-v4), EnvSpec(Venture-ramDeterministic-v0), EnvSpec(Venture-ramDeterministic-v4), EnvSpec(Venture-ramNoFrameskip-v0), EnvSpec(Venture-ramNoFrameskip-v4), EnvSpec(VideoPinball-v0), EnvSpec(VideoPinball-v4), EnvSpec(VideoPinballDeterministic-v0), EnvSpec(VideoPinballDeterministic-v4), EnvSpec(VideoPinballNoFrameskip-v0), EnvSpec(VideoPinballNoFrameskip-v4), EnvSpec(VideoPinball-ram-v0), EnvSpec(VideoPinball-ram-v4), EnvSpec(VideoPinball-ramDeterministic-v0), EnvSpec(VideoPinball-ramDeterministic-v4), EnvSpec(VideoPinball-ramNoFrameskip-v0), EnvSpec(VideoPinball-ramNoFrameskip-v4), EnvSpec(WizardOfWor-v0), EnvSpec(WizardOfWor-v4), EnvSpec(WizardOfWorDeterministic-v0), EnvSpec(WizardOfWorDeterministic-v4), EnvSpec(WizardOfWorNoFrameskip-v0), EnvSpec(WizardOfWorNoFrameskip-v4), EnvSpec(WizardOfWor-ram-v0), EnvSpec(WizardOfWor-ram-v4), EnvSpec(WizardOfWor-ramDeterministic-v0), EnvSpec(WizardOfWor-ramDeterministic-v4), EnvSpec(WizardOfWor-ramNoFrameskip-v0), EnvSpec(WizardOfWor-ramNoFrameskip-v4), EnvSpec(YarsRevenge-v0), EnvSpec(YarsRevenge-v4), EnvSpec(YarsRevengeDeterministic-v0), EnvSpec(YarsRevengeDeterministic-v4), EnvSpec(YarsRevengeNoFrameskip-v0), EnvSpec(YarsRevengeNoFrameskip-v4), EnvSpec(YarsRevenge-ram-v0), EnvSpec(YarsRevenge-ram-v4), EnvSpec(YarsRevenge-ramDeterministic-v0), EnvSpec(YarsRevenge-ramDeterministic-v4), EnvSpec(YarsRevenge-ramNoFrameskip-v0), EnvSpec(YarsRevenge-ramNoFrameskip-v4), EnvSpec(Zaxxon-v0), EnvSpec(Zaxxon-v4), EnvSpec(ZaxxonDeterministic-v0), EnvSpec(ZaxxonDeterministic-v4), EnvSpec(ZaxxonNoFrameskip-v0), EnvSpec(ZaxxonNoFrameskip-v4), EnvSpec(Zaxxon-ram-v0), EnvSpec(Zaxxon-ram-v4), EnvSpec(Zaxxon-ramDeterministic-v0), EnvSpec(Zaxxon-ramDeterministic-v4), EnvSpec(Zaxxon-ramNoFrameskip-v0), EnvSpec(Zaxxon-ramNoFrameskip-v4), EnvSpec(CubeCrash-v0), EnvSpec(CubeCrashSparse-v0), EnvSpec(CubeCrashScreenBecomesBlack-v0), EnvSpec(MemorizeDigits-v0)])\n"
     ]
    }
   ],
   "source": [
    "print(envs.registry.all())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "1ee2e2c8c9e58a4de97d163f262691ad3273a311"
   },
   "source": [
    "Let's start learning using some build-in games in Gym that do not require additonal installs. Let's start with a very basic game called Taxi.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "80a92d7ae756846e1c72846c47209899395175c6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|R: | : :\u001b[35mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m|\u001b[43m \u001b[0m: |B: |\n",
      "+---------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('Taxi-v2')\n",
    "env.reset()\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "24fa3bba1a709611174be39b512dcdddcbae66a4"
   },
   "source": [
    "# The Taxi game\n",
    "There are four designated locations in the grid world indicated by R(ed), B(lue), G(reen), and Y(ellow). When the episode starts, the taxi starts off at a random square and the passenger is at a random location. The taxi drive to the passenger's location, pick up the passenger, drive to the passenger's destination (another one of the four specified locations), and then drop off the passenger. Once the passenger is dropped off, the episode ends. The taxi cannot pass thru a wall.\n",
    "\n",
    "Actions: \n",
    "There are 6 discrete deterministic actions:\n",
    "    - 0: move south\n",
    "    - 1: move north\n",
    "    - 2: move east \n",
    "    - 3: move west \n",
    "    - 4: pickup passenger\n",
    "    - 5: dropoff passenger\n",
    "    \n",
    "Rewards: \n",
    "There is a reward of -1 for each action and an additional reward of +20 for delievering the passenger. There is a reward of -10 for executing actions \"pickup\" and \"dropoff\" illegally.\n",
    "    \n",
    "Rendering:\n",
    "    - blue: passenger\n",
    "    - magenta: destination\n",
    "    - yellow: empty taxi\n",
    "    - green: full taxi\n",
    "    - other letters: locations\n",
    "\n",
    "\n",
    "https://gym.openai.com/envs/Taxi-v2/\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7650831ece2de8bea1595b76d446c6dec9a80ba1"
   },
   "source": [
    "# Interacting with the Gym environment  \n",
    "Source: [OpenAI](https://openai.com/)  \n",
    "OPenAI/gym makes it relative straightforward to interact with the game.  \n",
    "<img src=\"https://cdn-images-1.medium.com/max/800/1*7Ae4mf9gVvpuMgenwtf8wA.png\">\n",
    "\n",
    "Each timestep, the agent chooses an action, and the environment returns an observation and a reward.  \n",
    "\n",
    "*observation, reward, done, info = env.step(action) *  \n",
    "* observation (object): an environment-specific object representing your observation of the environment. For example, pixel data from a camera, joint angles and joint velocities of a robot, or the board state in a board game line Taxi.\n",
    "* reward (float): amount of reward achieved by the previous action. The scale varies between environments, but the goal is always to increase your total reward.\n",
    "* done (boolean): whether it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and done being True indicates the episode has terminated. (For example, perhaps the pole tipped too far, or you lost your last life.)\n",
    "* info (dict): ignore, diagnostic information useful for debugging. Official evaluations of your agent are not allowed to use this for learning.  \n",
    "\n",
    "Let's first do some random steps in the game so you see how the game looks like\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_uuid": "77dd8855ce714bfaf0f50fe0a2c1c6d4069b37b7",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[43mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[43mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (Pickup)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[43mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (Dropoff)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : : : :\u001b[43m \u001b[0m|\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : : :\u001b[43m \u001b[0m: |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : :\u001b[43m \u001b[0m: : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| :\u001b[43m \u001b[0m: : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "Reward: -24\n"
     ]
    }
   ],
   "source": [
    "# Let's first do some random steps in the game so you see how the game looks like\n",
    "\n",
    "rew_tot=0\n",
    "obs= env.reset()\n",
    "env.render()\n",
    "for _ in range(6):\n",
    "    action = env.action_space.sample() #take step using random action from possible actions (actio_space)\n",
    "    obs, rew, done, info = env.step(action) \n",
    "    rew_tot = rew_tot + rew\n",
    "    env.render()\n",
    "#Print the reward of these random action\n",
    "print(\"Reward: %r\" % rew_tot)    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "ba589475e35bf258a87335d43c0cbee595f8bc7d"
   },
   "source": [
    "# Action \n",
    "Action (a): the input the agent provides to the environment. So what are the action commands the agents can give to the enironment? The env.action_space will tell you\n",
    "\n",
    "What is the meaning of the actions? For the deep learning algorithm it should not matter, it should sort it out independent of the meaning of the action. But for humans it is handy to have the description, so we can understand the actions.   \n",
    "\n",
    "In case of the Taxi game [0..5]:  \n",
    "* 0: move south\n",
    "* 1: move north\n",
    "* 2: move east \n",
    "* 3: move west \n",
    "* 4: pickup passenger\n",
    "* 5: dropoff passenger\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "500b6e1fabcceccf19ad494149d287fa82bf65c5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Discrete(6)\n",
      "Possible actions: [0..5]\n"
     ]
    }
   ],
   "source": [
    "# action space has 6 possible actions, the meaning of the actions is nice to know for us humans but the neural network will figure it out\n",
    "print(env.action_space)\n",
    "NUM_ACTIONS = env.action_space.n\n",
    "print(\"Possible actions: [0..%a]\" % (NUM_ACTIONS-1))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "6bcee395c56b5c161e43ce36832c0c3673e8cd3f"
   },
   "source": [
    "# State  \n",
    "State (s): This represents the board state of the game and in gym returned it is returned as observation. State: a numeric representation of what the agent is observing at a particular moment of time in the environment.  \n",
    "In case of Taxi the observation is an integer, 500 different states are possible that translate to a nice graphic visual format with the render function. Note that this is specific for the Taxi game, in case of e.g. an Atari game the observation is the game screen with many coloured pixels.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "eeeb040b8bee18af1c5c070b573912eabe4e139f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Discrete(500)\n",
      "\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: |\u001b[43m \u001b[0m: :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[34;1mR\u001b[0m: | : :G|\n",
      "| : : : : |\n",
      "| :\u001b[43m \u001b[0m: : : |\n",
      "| | : | : |\n",
      "|\u001b[35mY\u001b[0m| : |B: |\n",
      "+---------+\n",
      "  (West)\n"
     ]
    }
   ],
   "source": [
    "print(env.observation_space)\n",
    "print()\n",
    "env.env.s=42 # some random number, you might recognize it\n",
    "env.render()\n",
    "env.env.s = 222 # and some other\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "13c4565e8ab253c842d69d1f3d02973b3c0d95f5"
   },
   "source": [
    "# Markov decision process(MDP)\n",
    "The Taxi game is an example of an [Markov decision process ](https://en.wikipedia.org/wiki/Markov_decision_process). The game can be described in states, possible actions in a state (leading to a next state with a certain probability) with a reward.\n",
    "\n",
    "The word Markov refers to [Markovian property](https://en.wikipedia.org/wiki/Markov_property) which means that the state is independent of any previous states history, not on the sequence of events that preceded it. The current state encapsulates all that is needed to decide the future actions, no memory needed.\n",
    "\n",
    "[small video to explain Markov model](https://www.youtube.com/watch?v=EqUfuT3CC8s)  \n",
    "\n",
    "In terms of Reinforcement Learning the Environment is modelled as a markov model and the agent needs to take actions in this environment to maximize the amount of reward. Since the agent sees only the outside of the environment (the effects of it actions) it is often referred to as the hidden markov model which needs to be learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "b2a61906ac33e831e11d0c70b011478c8c2f3124"
   },
   "source": [
    "# Policy\n",
    "Policy (π): The strategy that the agent employs to determine next action 'a'  in state 's'. Note that it does not state if it is a good or bad policy, it is a policy.   The policy is normally noted with the greek letter π.\n",
    "Optimal policy (π*), policy which maximizes the expected reward. Among all the policies taken, the optimal policy is the one that optimizes to maximize the amount of reward received or expected to receive over a lifetime.  \n",
    "\n",
    "So how do we find the optimal policy (π*) that is maximize our reward (and win the game) given the Taxi environment with the Markov model .  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "65441f069d3f6afa07cde0ce37527d956288f601"
   },
   "source": [
    "# Bellman equation\n",
    "We will make use of the basic Bellman equation for deterministic environments to solve a problem that is described as a Markov model, see figure below:\n",
    "\n",
    "<img src=\"https://wikimedia.org/api/rest_v1/media/math/render/svg/06039a80d0b3ef5b1eee6f05b9ae77867cda3026\">\n",
    "\n",
    "Source: [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Q-l%C3%A6ring_formel_1.png)\n",
    "\n",
    "where\n",
    "* R(s,a) = Reward of action a in state s\n",
    "* P(s'|s,a)= Probability of going to state s' given action a in state s. The Taxi game actions are deterministic (no such a thing as if I want to go north there is an 80% chance to go north and 10% chance to go west and 10% chance to go east). so the probability that selected action will lead to expected state is 100%. So ignore it for this game, it is always 1.\n",
    "* γ = Discount factor gamma, how much discount is applicable for the future rewards. It must be between 0 and 1. The higher gamma the higher the focus on long term rewards\n",
    "\n",
    "The value iteration algorithm makes use of the equation in the form of:  \n",
    "* Value V(s): The expected long-term return with discount, as opposed to the short-term reward R. Vπ(s) is defined as the expected long-term return of the current state sunder policy π.  \n",
    "\n",
    "The Q learning algorithm makes use of the equation in the form of:   \n",
    "* Q-matrix or action-value Q(s,a): Q-matrix is similar to Value, except that it takes an extra parameter, the action a. Qπ(s, a) refers to the long-term return of the current state s, taking action a under policy π."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "00d5812c19f9bb264e4c8859d4ea7084bbcbd2c4"
   },
   "source": [
    "# Value iteration algorithm\n",
    " So' let's start first in \"memory lane\", the early days of reinforcement learning. Value iteration is the \"hello world\" of reinforcement learning methods to find an optimal policy to maximize reward for e.g. a Markov decision process problem.   \n",
    "\n",
    "The value iteration is centred around the game states. The core of the idea is to calculate the value (expected long-term maximum result) of each state. The algorithm loops over all states (s) and possible actions (a) to explore rewards of a given action and calculates the maximum possible action/reward and stores it in V[s]. The algorithm iterates/repeats until V[s] is not (significantly) improving anymore. The Optimal policy (π*) is then to take every time the action to go state with the highest value. This value iteration algorithm is an example of what is referred to as [dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming) (DP) in literature. There are other DP techniques to solve this like policy iteration, etc but it can also be solved by a recursive program (a function that calls itself, look at the Bellman equation, it is a recursive definition).  \n",
    "Anyhow, lets see how this value iteration works.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "60c2f7e729ac11e8696694eafa4ac9ed944d4665"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "74  iterations done\n"
     ]
    }
   ],
   "source": [
    "# Value iteration algorithem\n",
    "NUM_ACTIONS = env.action_space.n\n",
    "NUM_STATES = env.observation_space.n\n",
    "V = np.zeros([NUM_STATES]) # The Value for each state\n",
    "Pi = np.zeros([NUM_STATES], dtype=int)  # Our policy with we keep updating to get the optimal policy\n",
    "gamma = 0.9 # discount factor\n",
    "significant_improvement = 0.01\n",
    "\n",
    "def best_action_value(s):\n",
    "    # finds the highest value action (max_a) in state s\n",
    "    best_a = None\n",
    "    best_value = float('-inf')\n",
    "\n",
    "    # loop through all possible actions to find the best current action\n",
    "    for a in range (0, NUM_ACTIONS):\n",
    "        env.env.s = s\n",
    "        s_new, rew, done, info = env.step(a) #take the action\n",
    "        v = rew + gamma * V[s_new]\n",
    "        if v > best_value:\n",
    "            best_value = v\n",
    "            best_a = a\n",
    "    return best_a\n",
    "\n",
    "iteration = 0\n",
    "while True:\n",
    "    # biggest_change is referred to by the mathematical symbol delta in equations\n",
    "    biggest_change = 0\n",
    "    for s in range (0, NUM_STATES):\n",
    "        old_v = V[s]\n",
    "        action = best_action_value(s) #choosing an action with the highest future reward\n",
    "        env.env.s = s # goto the state\n",
    "        s_new, rew, done, info = env.step(action) #take the action\n",
    "        V[s] = rew + gamma * V[s_new] #Update Value for the state using Bellman equation\n",
    "        Pi[s] = action\n",
    "        biggest_change = max(biggest_change, np.abs(old_v - V[s]))\n",
    "    iteration += 1\n",
    "    if biggest_change < significant_improvement:\n",
    "        print (iteration,' iterations done')\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_uuid": "373b433ca66d1dee88071f55b414f5344ab80ecd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[34;1mG\u001b[0m|\n",
      "| : : :\u001b[43m \u001b[0m: |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | :\u001b[43m \u001b[0m:\u001b[34;1mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[34;1m\u001b[43mG\u001b[0m\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (East)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :\u001b[42mG\u001b[0m|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (Pickup)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : : : :\u001b[42m_\u001b[0m|\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : : :\u001b[42m_\u001b[0m: |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| : :\u001b[42m_\u001b[0m: : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m: | : :G|\n",
      "| :\u001b[42m_\u001b[0m: : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35mR\u001b[0m:\u001b[42m_\u001b[0m| : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (North)\n",
      "+---------+\n",
      "|\u001b[35m\u001b[42mR\u001b[0m\u001b[0m: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|\u001b[35m\u001b[42mR\u001b[0m\u001b[0m: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |B: |\n",
      "+---------+\n",
      "  (Dropoff)\n",
      "Reward: 11\n"
     ]
    }
   ],
   "source": [
    "# Let's see how the algorithm solves the taxi game\n",
    "rew_tot=0\n",
    "obs= env.reset()\n",
    "env.render()\n",
    "done=False\n",
    "while done != True: \n",
    "    action = Pi[obs]\n",
    "    obs, rew, done, info = env.step(action) #take step using selected action\n",
    "    rew_tot = rew_tot + rew\n",
    "    env.render()\n",
    "#Print the reward of these actions\n",
    "print(\"Reward: %r\" % rew_tot)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "db36d8f42216986bd8778dae5c86bdb022193322"
   },
   "source": [
    "# Model vs Model-free based methods\n",
    "It nicely solves it. Still it feels a bit like cheating in terms of reinforcement learning. We have to know all environment states/transitions upfront so the algorithm works. In RL literature they refer to it as model based methods.\n",
    "What if not all states are known upfront and we need to find out while we are learning. Hence we enter the realm of model-free based methods.\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "2f99c26b731ef1829788435ba4a3600b9f531546"
   },
   "source": [
    "# Basic Q-learning algorithm  \n",
    "The[ Q-learning](https://en.wikipedia.org/wiki/Q-learning) algorithm is centred around the actor (in this case the Taxi) and starts exploring based on trial-and-error to update its knowledge about the model and hence path to the best reward.\n",
    "The core of the idea is the Q-matrix Q(s, a). It contains the maximum discounted future reward when we perform action a in state s. Or in other words Q(s, a) gives estimates the best course of action a in state s. Q-learning learns by trail and error and updates its policy (Q-matrix) based on reward.  to state it simple: the best it can do given a state it is in.  \n",
    "After every step we update Q(s,a) using the reward, and the max Q value for new state resulting from the action. This update is done using the action value formula (based upon the Bellman equation) and allows state-action pairs to be updated in a recursive fashion (based on future values).   \n",
    "The Bellman equation is extended with a learning rate (if you put learning rate = 1 it comes back to the basic Bellman equation) :\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/f/f0/Q-l%C3%A6ring_formel_1.png/800px-Q-l%C3%A6ring_formel_1.png\">\n",
    "\n",
    "Source: [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:Q-l%C3%A6ring_formel_1.png)\n",
    "\n",
    "Note: [Temporal difference learning ](https://en.wikipedia.org/wiki/Temporal_difference_learning) and [Sarsa](https://en.wikipedia.org/wiki/State%E2%80%93action%E2%80%93reward%E2%80%93state%E2%80%93action) algorithems explored simular value expressions. Q-learning was the basis for Deep Q-learning (Deep referring to Neural Network technology). so, let's see how the Q-learning algorithm works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_uuid": "0a8bf6c408b00cdc614614159f001d63617f1db5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 50 Total Reward: -61\n",
      "Episode 100 Total Reward: -93\n",
      "Episode 150 Total Reward: -12\n",
      "Episode 200 Total Reward: 6\n",
      "Episode 250 Total Reward: 5\n",
      "Episode 300 Total Reward: 9\n",
      "Episode 350 Total Reward: 5\n",
      "Episode 400 Total Reward: 7\n",
      "Episode 450 Total Reward: 10\n",
      "Episode 500 Total Reward: 4\n",
      "Episode 550 Total Reward: 7\n",
      "Episode 600 Total Reward: 13\n",
      "Episode 650 Total Reward: 6\n",
      "Episode 700 Total Reward: 6\n",
      "Episode 750 Total Reward: 12\n",
      "Episode 800 Total Reward: 7\n",
      "Episode 850 Total Reward: 9\n",
      "Episode 900 Total Reward: 7\n",
      "Episode 950 Total Reward: 10\n",
      "Episode 1000 Total Reward: 6\n"
     ]
    }
   ],
   "source": [
    "NUM_ACTIONS = env.action_space.n\n",
    "NUM_STATES = env.observation_space.n\n",
    "Q = np.zeros([NUM_STATES, NUM_ACTIONS]) #You could also make this dynamic if you don't know all games states upfront\n",
    "gamma = 0.9 # discount factor\n",
    "alpha = 0.9 # learning rate\n",
    "for episode in range(1,1001):\n",
    "    done = False\n",
    "    rew_tot = 0\n",
    "    obs = env.reset()\n",
    "    while done != True:\n",
    "            action = np.argmax(Q[obs]) #choosing the action with the highest Q value \n",
    "            obs2, rew, done, info = env.step(action) #take the action\n",
    "            Q[obs,action] += alpha * (rew + gamma * np.max(Q[obs2]) - Q[obs,action]) #Update Q-marix using Bellman equation\n",
    "            #Q[obs,action] = rew + gamma * np.max(Q[obs2]) # same equation but with learning rate = 1 returns the basic Bellman equation\n",
    "            rew_tot = rew_tot + rew\n",
    "            obs = obs2   \n",
    "    if episode % 50 == 0:\n",
    "        print('Episode {} Total Reward: {}'.format(episode,rew_tot))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "90fa1d8270ca776ecc23badc8ef43e22a71e7e52"
   },
   "source": [
    "So, what is the magic, how does it solve it? \n",
    "\n",
    "The Q-matrix is initialized with zero's. So initially it starts moving randomly until it hits a state/action with rewards or state/actions with a penalty. For understanding, let's simplify the problem that it needs to go to a certain drop-off position to get a reward. So random moves get no rewards but by luck (brute force enough tries) the state/action is found where a reward is given. So next game the immediate actions preceding this state/action will direct toward it by use of the Q-Matrix. The next iteration the actions before that, etc, etc. In other words, it solves \"the puzzle\" backwards from end-result (drop-off passenger) towards steps to be taken to get there in a iterative fashion.  \n",
    "\n",
    "Note that in case of the Taxi game there is a reward of -1 for each action. So if in a state the algorithm explored eg south which let to no value the Q-matrix is updated to -1 so next iteration (because values were initialized on 0) it will try an action that is not yet tried and still on 0. So also by design it encourages systematic exploration of states and actions \n",
    "\n",
    "If you put the learning rate on 1 the game also is solved. Reason is that there is only one reward (dropoff passenger), so the algorithm will find it whatever learning rate. In case a game has more reward places the learning rate determines if it should prioritize longer term or shorter term rewards\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "8e4ac4c4a5aea696de188db3159b880193994709"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+\n",
      "|R: | :\u001b[43m \u001b[0m:G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : :\u001b[43m \u001b[0m: |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : :\u001b[43m \u001b[0m: |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : :\u001b[43m \u001b[0m: : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| :\u001b[43m \u001b[0m: : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "|\u001b[43m \u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (West)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "|\u001b[43m \u001b[0m| : | : |\n",
      "|\u001b[34;1mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[34;1m\u001b[43mY\u001b[0m\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|\u001b[42mY\u001b[0m| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (Pickup)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "|\u001b[42m_\u001b[0m| : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (North)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "|\u001b[42m_\u001b[0m: : : : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (North)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| :\u001b[42m_\u001b[0m: : : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (East)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : :\u001b[42m_\u001b[0m: : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (East)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : :\u001b[42m_\u001b[0m: |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (East)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : |\u001b[42m_\u001b[0m: |\n",
      "|Y| : |\u001b[35mB\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35m\u001b[42mB\u001b[0m\u001b[0m: |\n",
      "+---------+\n",
      "  (South)\n",
      "+---------+\n",
      "|R: | : :G|\n",
      "| : : : : |\n",
      "| : : : : |\n",
      "| | : | : |\n",
      "|Y| : |\u001b[35m\u001b[42mB\u001b[0m\u001b[0m: |\n",
      "+---------+\n",
      "  (Dropoff)\n",
      "Reward: 5\n"
     ]
    }
   ],
   "source": [
    "# Let's see how the algorithm solves the taxi game by following the policy to take actions delivering max value\n",
    "\n",
    "rew_tot=0\n",
    "obs= env.reset()\n",
    "env.render()\n",
    "done=False\n",
    "while done != True: \n",
    "    action = np.argmax(Q[obs])\n",
    "    obs, rew, done, info = env.step(action) #take step using selected action\n",
    "    rew_tot = rew_tot + rew\n",
    "    env.render()\n",
    "#Print the reward of these actions\n",
    "print(\"Reward: %r\" % rew_tot)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "a3ff1014ef78d57017e4966acdf3fc235fa7eb86"
   },
   "source": [
    "# exploration vs. exploitation\n",
    "\n",
    "The taxi game has one place with the reward: dropoff passenger +20. What if it also had a reward: stop at coffee shop +2 points. If the trial-and-error found that value first and in subsequent iterations started to optimize the route to it, how do we know it would also find the bigger reward of dropoff passenger?  \n",
    "What if the the Taxi game actions are not deterministic (eg due to slippery roads I want to go north there is an 80% chance to go north and 10% chance to go west and 10% chance to go east), how would we ensure we still find the optimal policy?\n",
    "\n",
    "Our algorithm to exploit action = np.argmax(Q[obs]) time and time again will not cope with these more complex environments. In Reinforcement literature this is called the crucial tradeoff between \"exploitation\" and \"exploration\".\n",
    "* Exploitation: Make the best decision given current information (Go to the restaurent you know you like)\n",
    "* Exploration: Gather more information (go to a new restaurent to find out if you like it)\n",
    "\n",
    "Some approaches:  \n",
    "\n",
    "* Epsilon Greedy  \n",
    "We exploit the current situation with probability 1 — epsilon and explore a new option with probability epsilon,  the rates of exploration and exploitation are fixed\n",
    "* Epsilon-Decreasing  \n",
    "We exploit the current situation with probability 1 — epsilon and explore a new option with probability epsilon, with epsilon decreasing over time. \n",
    "* Thompson sampling: the rates of exploration and exploitation are dynamically updated with respect to the entire probability distribution of each arm\n",
    "* Epsilon-Decreasing with Softmax\n",
    "We exploit the current situation with probability 1 — epsilon and explore a new option with probability epsilon, with epsilon decreasing over time.  In the case of exploring a new option, we don’t just pick an option at random, but instead we estimate the outcome of each option, and then pick based on that (this is the softmax part).  \n",
    "* Etc  \n",
    "\n",
    "The code below I tried with epsilon-greedy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f7d5a310b7ba811af48c10e8bee933501dd10cf9"
   },
   "source": [
    "A nice place to test this is in the game [Frozen lakes](https://gym.openai.com/envs/FrozenLake-v0/) of OpenAI/Gym.  \n",
    "\n",
    " Short description: \"Winter is here. You and your friends were tossing around a frisbee at the park when you made a wild throw that left the frisbee out in the middle of the lake. The water is mostly frozen, but there are a few holes where the ice has melted. If you step into one of those holes, you'll fall into the freezing water. At this time, there's an international frisbee shortage, so it's absolutely imperative that you navigate across the lake and retrieve the disc. However, the ice is slippery, so you won't always move in the direction you intend.\"  \n",
    "\n",
    "Notice that the game is not deterministic anymore: \"won't always move in the direction you intend\". Note it is really slippery, the chance you move into the direction you want is actually not that big\n",
    "\n",
    "S- Start  \n",
    "G - Goal  \n",
    "F- Frozen (safe)  \n",
    "H- Hole (dead)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "7ea12b3040c3d1c9629a9a47931ad6aa616f93a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "# Let's show the game layout\n",
    "\n",
    "env = gym.make('FrozenLake-v0')\n",
    "rew_tot=0\n",
    "obs= env.reset()\n",
    "env.render()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "8b14f7d3622c01a3e55a6c1bab79cc65b2fc61dd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 5000 avarage reward: 0.0\n",
      "Episode 10000 avarage reward: 0.0\n",
      "Episode 15000 avarage reward: 0.24\n",
      "Episode 20000 avarage reward: 0.52\n",
      "Episode 25000 avarage reward: 0.78\n",
      "Episode 30000 avarage reward: 0.6\n",
      "Episode 35000 avarage reward: 0.67\n",
      "Episode 40000 avarage reward: 0.69\n",
      "Episode 45000 avarage reward: 0.71\n",
      "Episode 50000 avarage reward: 0.77\n",
      "Episode 55000 avarage reward: 0.64\n",
      "Episode 60000 avarage reward: 0.75\n",
      "Episode 65000 avarage reward: 0.73\n",
      "Episode 70000 avarage reward: 0.63\n",
      "Episode 75000 avarage reward: 0.72\n",
      "Episode 80000 avarage reward: 0.68\n",
      "Episode 85000 avarage reward: 0.7\n",
      "Episode 90000 avarage reward: 0.75\n",
      "Episode 95000 avarage reward: 0.76\n",
      "Episode 100000 avarage reward: 0.8\n",
      "Episode 105000 avarage reward: 0.8\n",
      "Episode 110000 avarage reward: 0.75\n",
      "Episode 115000 avarage reward: 0.69\n",
      "Episode 120000 avarage reward: 0.74\n",
      "Episode 125000 avarage reward: 0.7\n",
      "Episode 130000 avarage reward: 0.7\n",
      "Episode 135000 avarage reward: 0.7\n",
      "Episode 140000 avarage reward: 0.76\n",
      "Episode 145000 avarage reward: 0.76\n",
      "Episode 150000 avarage reward: 0.76\n",
      "Episode 155000 avarage reward: 0.71\n",
      "Episode 160000 avarage reward: 0.76\n",
      "Episode 165000 avarage reward: 0.73\n",
      "Episode 170000 avarage reward: 0.81\n",
      "Frozen lake solved\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('FrozenLake-v0')\n",
    "env.reset()\n",
    "NUM_ACTIONS = env.action_space.n\n",
    "NUM_STATES = env.observation_space.n\n",
    "Q = np.zeros([NUM_STATES, NUM_ACTIONS]) #You could also make this dynamic if you don't know all games states upfront\n",
    "gamma = 0.95 # discount factor\n",
    "alpha = 0.01 # learning rate\n",
    "epsilon = 0.1 #\n",
    "for episode in range(1,500001):\n",
    "    done = False\n",
    "    obs = env.reset()\n",
    "    while done != True:\n",
    "        if np.random.rand(1) < epsilon:\n",
    "            # exploration with a new option with probability epsilon, the epsilon greedy approach\n",
    "            action = env.action_space.sample()\n",
    "        else:\n",
    "            # exploitation\n",
    "            action = np.argmax(Q[obs])\n",
    "        obs2, rew, done, info = env.step(action) #take the action\n",
    "        Q[obs,action] += alpha * (rew + gamma * np.max(Q[obs2]) - Q[obs,action]) #Update Q-marix using Bellman equation\n",
    "        obs = obs2   \n",
    "        \n",
    "    if episode % 5000 == 0:\n",
    "        #report every 5000 steps, test 100 games to get avarage point score for statistics and verify if it is solved\n",
    "        rew_average = 0.\n",
    "        for i in range(100):\n",
    "            obs= env.reset()\n",
    "            done=False\n",
    "            while done != True: \n",
    "                action = np.argmax(Q[obs])\n",
    "                obs, rew, done, info = env.step(action) #take step using selected action\n",
    "                rew_average += rew\n",
    "        rew_average=rew_average/100\n",
    "        print('Episode {} avarage reward: {}'.format(episode,rew_average))\n",
    "        \n",
    "        if rew_average > 0.8:\n",
    "            # FrozenLake-v0 defines \"solving\" as getting average reward of 0.78 over 100 consecutive trials.\n",
    "            # Test it on 0.8 so it is not a one-off lucky shot solving it\n",
    "            print(\"Frozen lake solved\")\n",
    "            break\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "570a20a4336cb40b424d9a5f68420b61c5dcb0a4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "\u001b[41mF\u001b[0mFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "SFFF\n",
      "FHFH\n",
      "F\u001b[41mF\u001b[0mFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "H\u001b[41mF\u001b[0mFG\n",
      "  (Right)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HF\u001b[41mF\u001b[0mG\n",
      "  (Down)\n",
      "SFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFF\u001b[41mG\u001b[0m\n",
      "Reward: 1.0\n"
     ]
    }
   ],
   "source": [
    "# Let's see how the algorithm solves the frozen-lakes game\n",
    "\n",
    "rew_tot=0.\n",
    "obs= env.reset()\n",
    "done=False\n",
    "while done != True: \n",
    "    action = np.argmax(Q[obs])\n",
    "    obs, rew, done, info = env.step(action) #take step using selected action\n",
    "    rew_tot += rew\n",
    "    env.render()\n",
    "\n",
    "print(\"Reward:\", rew_tot)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8ecaf0ce4d16b049d6390226c4cf39a8e3a7db9c"
   },
   "source": [
    "Analyzing the moves. It looks like that if you want to move rigth there is a significant chance you move up or down, simular if you want to move up there is a significant chance you move left or right, etc. So the algorithm learned that if you are on the frozen tile left column second row and you want to move down it is risky to give the down command because you could move to the right into the hole. So it gives the left command because if will keep you on the tile or move you up or down, but not to thr right.  \n",
    "Or in other words, the algorithm learned to take that actions with the least risk to (accidently slip) drown into a hole. Also interesting to se it learned as first move to go left, this to avoid you move right which is the more dangerous road.  \n",
    "\n",
    "Note: there is no 100% score possible. By consitently moving away from a hole you can safely traverse all fields except 1 (second row, thrid column) on which you could glide into due to slippery ice.  \n",
    "\n",
    "Also good to notice the algorithm uses tenthousands of iterations to find the optimal policy, while this is a 4 by 4 playing field..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "67649be9ed271e555a4a45e0da7d73472dda09c2"
   },
   "source": [
    "# DQN\n",
    "Nice, isn't it. In case of 500 observation states and 6 actions (Taxi game) or 16 observations and 4 moves (Frozen-lake game) the Q-matrix is a manageable matrix. Imagine you got a full Atari game screen of pixels as an observation and it becomes quickly visible the Q-matrix solution will not cope. Also the Q-learning agent does not have the ability to estimate value for unseen states, it has no clue which action to take and goes back to random action as best.  \n",
    "\n",
    "To deal with these problems, Deep Q-Network (DQN) removes the two-dimensional Q-matrix by introducing a Neural Network. So it leverages a Neural Network to estimate the Q-value function. The input for the network is the current game state, while the output is the corresponding Q-value for each of the actions.  \n",
    "![alt text](https://cdn-images-1.medium.com/max/800/1*3ZgGbUpEyAZb9POWijRq4Q.png)\n",
    "\n",
    "In 2014 Google DeepMind published a [paper](https://arxiv.org/pdf/1312.5602.pdf) titled \"Playing Atari with Deep Reinforcement Learning\" that can play Atari 2600 games at expert human levels. This was the first breakthruogh in applying deep neural networks for reinforcement learning.\n",
    "\n",
    "![alt text](https://cdn-images-1.medium.com/max/800/1*vUMIoHkl-PuIjbTqbtn8dA.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "8624390292f07aa1a37c052431e217308080e421",
    "collapsed": true
   },
   "source": [
    "# Reinforcement learning developments\n",
    "After the first publication of DQN many deeplearning Reinforcement Learning algorithms have been invented/tried, Some main ones in chronological order:  DQN, Double DQN, Duelling DQN, Deep Deterministic Policy Gradient, Continuous DQN (CDQN or NAF) ,  A2C/A3C,  Proximal Policy Optimization Algorithms, etc, etc. Let's see if we can also try them out on Kaggle and have some fun.  \n",
    "In my next notebook you will find the instructions to install gym[atari] so you are able to unleash the power op A.I. on these iconic Atari games like breakout, pacman, spaceinvaders, etc.\n",
    "\n",
    "https://www.kaggle.com/charel/learn-by-example-reinforcement-learning-gym-setup  \n",
    "\n",
    "\n",
    "Hoped you liked my notebook (upvote top right), my way to conribute back to this fantastic Kaggle platform and community.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}