intro-to-reinforcement-learning-with-openai-gym.ipynb

{"cells":[{"metadata":{"_uuid":"3c4f5211fc0d2e74b0d6aabe9fa8e07164ee07c5"},"cell_type":"markdown","source":"# **Introduction to Reinforcement Learning with Taxi V2 OpenAI Gym**\n\nWe shall use the Taxi V2 Open AI gym library. \nThe Documentation can be found at: https://github.com/openai/gym/blob/master/gym/envs/toy_text/taxi.py"},{"metadata":{"trusted":true,"_uuid":"0f45fbd74eba00eef91dfbb511f73ab06dbc667d"},"cell_type":"code","source":"import gym # openAi gym\nfrom gym import envs\nimport numpy as np \nimport datetime\nimport keras \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nimport pandas as pd \nfrom time import sleep\n\nfrom rl.agents.dqn import DQNAgent\nfrom rl.policy import LinearAnnealedPolicy, BoltzmannQPolicy, EpsGreedyQPolicy\nfrom rl.memory import SequentialMemory\nfrom rl.core import Processor\nfrom rl.callbacks import FileLogger, ModelIntervalCheckpoint\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5f62ddd2db99429cd125112bda541723c858f419"},"cell_type":"markdown","source":"# **1. Intro to taxi game environment**\n\nThe aim of the taxi game is to make sure the taxi can get to the passenger, pick him up and bring him to the drop-off location in the fastest way possible."},{"metadata":{"_uuid":"8d0578768abc4cbf53c4a20d250ef8e39c99497c"},"cell_type":"markdown","source":"**Representations**\n\n- | --> WALL (Can't pass through, will remain in the same position if tries to move through wall)\n\n- Yellow --> Taxi Current Location\n\n- Blue --> Pick up Location\n\n- Purple --> Drop-off Location\n\n- Green --> Taxi turn green once passenger board\n\n- Letters --> Locations\n"},{"metadata":{"trusted":true,"_uuid":"b321b04f669571df210d6676265b9273d3017a9d"},"cell_type":"code","source":"env = gym.make('Taxi-v2')\nenv.reset()\nenv.render()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d85e4518e9981ab9ecb7beac0529864d55474393"},"cell_type":"markdown","source":"Total Number of states in the environment = **500** (0 to 499)"},{"metadata":{"trusted":true,"_uuid":"e0004dc2b02965be1f47ec64998578b91a1e819f"},"cell_type":"code","source":"env.observation_space.n   #Total no. of states","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ddf91afb34e02d9a3ca72442516b2c9ac09482b3"},"cell_type":"markdown","source":"**Actions (6 in total)**\n\n- 0: move south\n- 1: move north\n- 2: move east \n- 3: move west \n- 4: pickup passenger\n- 5: dropoff passenger"},{"metadata":{"trusted":true,"_uuid":"b3dfff01881abd8bc8c4fd6ade6199e0be8fd0fa"},"cell_type":"code","source":"env.action_space.n   #Total no. of actions","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e4a0664e73e126d75e1139ed1a1f6e303704088a"},"cell_type":"code","source":"env.env.s = 122\nenv.render()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d23b96ffb38b239dcc551d81ae3a6c211c3db73"},"cell_type":"code","source":"env.step(3)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"201acf86a6308d8220cb1f474af31c8f64450ab9"},"cell_type":"markdown","source":"Each timestep, the agent chooses an action, and the environment returns an observation and a reward.\n\nThe 4 elements returned are:\n\n-  **Observation (object)**: the state the environment is in or an environment-specific object representing your observation of the environment.\n-  **Reward (float)**: Reward achieved by the previous action. \n    -  +20: Last step when we successfully pick up a passenger and drop them off at their desired location\n    -  -1: for each step in order for the agent to try and find the quickest solution possible\n    -  -10: every time you incorrectly pick up or drop off a passenger\n-  **Done (boolean)**: whether it’s time to reset the environment again. Most (but not all) tasks are divided up into well-defined episodes, and done being True indicates the episode has terminated. (For example, you lost your last life.)\n-  **Info (dict)**: Can be ignored, diagnostic information useful for debugging. Official evaluations of your agent are not allowed to use this for learning."},{"metadata":{"trusted":true,"_uuid":"3ed8fa4353fb0bb7c2f831099d6d228c421829bf"},"cell_type":"code","source":"env.render()  #view state","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4dca645ec054b91775aecd06acdde55c3450ce7e"},"cell_type":"markdown","source":"The function (env.P) below can be used to see the relevant states and rewards for each action taken in that particular state."},{"metadata":{"trusted":true,"_uuid":"0a45d784b8e8a22c54bed3a3f6f918021a721e1c"},"cell_type":"code","source":"env.reset()\nenv.env.P[300]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd7f8693f05c30915cba7f0a712ea24b82035363"},"cell_type":"markdown","source":"# **2. Random Search**\nLet's start with the simplest way to train our agent to complete this task. The agent would just take random steps at every state until he completes the task (picking the passenger and dropping him off at the drop-off location). "},{"metadata":{"trusted":true,"_uuid":"bd8addbf5b06d604052757c34c35f803c280684a"},"cell_type":"code","source":"env = gym.make('Taxi-v2')\nrandom_policy = np.ones([env.env.nS, env.env.nA]) / env.env.nA\ndef random_policy_steps_count():\n    state = env.reset()\n    counter = 0\n    reward = None\n    while reward != 20:\n        state, reward, done, info = env.step(env.action_space.sample())  \n        counter += 1\n    return counter\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b64e536320cff64dc87c7c099e66bdecf766c18"},"cell_type":"code","source":"counts = [random_policy_steps_count() for i in range(1000)]\nsns.distplot(counts)\nplt.title(\"Distribution of number of steps needed\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3eceba707f008db9e21a360dffe38da71ffa2f0c"},"cell_type":"code","source":"print(\"An agent using Random search takes about an average of \" + str(int(np.mean(counts)))\n      + \" steps to successfully complete its mission.\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89d733d85d462d668e7dafd27a5210e3fa95ecb8"},"cell_type":"markdown","source":"Clearly, it is not the most efficient way to complete this. Let's try to use policy iteration and value iteration method to more effectively let our agent complete this task. "},{"metadata":{"_uuid":"eed867980763903739024c63c58babae83661f3d"},"cell_type":"markdown","source":"# **3. Policy Iteration/Value Iteration**\n\n**Policy Iteration** --> The algorithm redefines the policy at each step and improve the policy, and then compute the value according to this new policy until the policy converges.\n\n**Value Iteration** --> The algorithm computes the optimal state value function by iteratively improving the estimate of V(s). The algorithm initialize V(s) to arbitrary random values. It repeatedly updates the Q(s, a) and V(s) values until they converge. \n\n\nPolicy iteration and Value Iteration are guranteed to convege to the optimal policy and it often takes less iterations for policy iteration to converge than the value-iteration algorithm.\n\nBoth value-iteration and policy-iteration algorithms can be used for offline planning where the agent is assumed to have prior knowledge about the effects of its actions on the environment (they assume the Markov Decision model is known). Comparing to each other, policy-iteration is computationally efficient as it often takes considerably fewer number of iterations to converge although each iteration is more computationally expensive."},{"metadata":{"_uuid":"71f05229183918a96cccc686710e5e48ef024cb2"},"cell_type":"markdown","source":"**3.1 Functions for policy evaluation, policy iteration and value iteration.**"},{"metadata":{"trusted":true,"_uuid":"a14285c0077a95ba7a24ec0b966916b86c406081"},"cell_type":"code","source":"def policy_eval(policy, env, discount_factor=1.0, theta=0.00001):\n    \"\"\"\n    Evaluate a policy given an environment and a full description of the environment's dynamics.\n    \n    Args:\n        policy: [S, A] shaped matrix representing the policy.\n        env: OpenAI env. env.P represents the transition probabilities of the environment.\n            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n            env.nS is a number of states in the environment. \n            env.nA is a number of actions in the environment.\n        theta: We stop evaluation once our value function change is less than theta for all states.\n        discount_factor: Gamma discount factor.\n    \n    Returns:\n        Vector of length env.nS representing the value function.\n    \"\"\"\n    # Start with a random (all 0) value function\n    V = np.zeros(env.env.nS)\n    while True:\n        # TODO: Implement!\n        delta = 0  #delta = change in value of state from one iteration to next\n       \n        for state in range(env.env.nS):  #for all states\n            val = 0  #initiate value as 0\n            \n            for action,act_prob in enumerate(policy[state]): #for all actions/action probabilities\n                for prob,next_state,reward,done in env.env.P[state][action]:  #transition probabilities,state,rewards of each action\n                    val += act_prob * prob * (reward + discount_factor * V[next_state])  #eqn to calculate\n            delta = max(delta, np.abs(val-V[state]))\n            V[state] = val\n        if delta < theta:  #break if the change in value is less than the threshold (theta)\n            break\n    return np.array(V)\n\ndef policy_iteration(env, policy_eval_fn=policy_eval, discount_factor=1.0):\n    \"\"\"\n    Policy Improvement Algorithm. Iteratively evaluates and improves a policy\n    until an optimal policy is found.\n    \n    Args:\n        env: The OpenAI envrionment.\n        policy_eval_fn: Policy Evaluation function that takes 3 arguments:\n            policy, env, discount_factor.\n        discount_factor: gamma discount factor.\n        \n    Returns:\n        A tuple (policy, V). \n        policy is the optimal policy, a matrix of shape [S, A] where each state s\n        contains a valid probability distribution over actions.\n        V is the value function for the optimal policy.\n        \n    \"\"\"\n    def one_step_lookahead(state, V):\n        \"\"\"\n        Helper function to calculate the value for all action in a given state.\n        \n        Args:\n            state: The state to consider (int)\n            V: The value to use as an estimator, Vector of length env.nS\n        \n        Returns:\n            A vector of length env.nA containing the expected value of each action.\n        \"\"\"\n        A = np.zeros(env.env.nA)\n        for a in range(env.env.nA):\n            for prob, next_state, reward, done in env.env.P[state][a]:\n                A[a] += prob * (reward + discount_factor * V[next_state])\n        return A\n    # Start with a random policy\n    policy = np.ones([env.env.nS, env.env.nA]) / env.env.nA\n\n    while True:\n        # Implement this!\n        curr_pol_val = policy_eval_fn(policy, env, discount_factor)  #eval current policy\n        policy_stable = True  #Check if policy did improve (Set it as True first)\n        for state in range(env.env.nS):  #for each states\n            chosen_act = np.argmax(policy[state])  #best action (Highest prob) under current policy\n            act_values = one_step_lookahead(state,curr_pol_val)  #use one step lookahead to find action values\n            best_act = np.argmax(act_values) #find best action\n            if chosen_act != best_act:\n                policy_stable = False  #Greedily find best action\n            policy[state] = np.eye(env.env.nA)[best_act]  #update \n        if policy_stable:\n            return policy, curr_pol_val\n    \n    return policy, np.zeros(env.env.nS)\n\ndef value_iteration(env, theta=0.0001, discount_factor=1.0):\n    \"\"\"\n    Value Iteration Algorithm.\n    \n    Args:\n        env: OpenAI env. env.P represents the transition probabilities of the environment.\n            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n            env.nS is a number of states in the environment. \n            env.nA is a number of actions in the environment.\n        theta: We stop evaluation once our value function change is less than theta for all states.\n        discount_factor: Gamma discount factor.\n        \n    Returns:\n        A tuple (policy, V) of the optimal policy and the optimal value function.        \n    \"\"\"\n    \n    def one_step_lookahead(state, V):\n        \"\"\"\n        Helper function to calculate the value for all action in a given state.\n        \n        Args:\n            state: The state to consider (int)\n            V: The value to use as an estimator, Vector of length env.nS\n        \n        Returns:\n            A vector of length env.nA containing the expected value of each action.\n        \"\"\"\n        A = np.zeros(env.env.nA)\n        for act in range(env.env.nA):\n            for prob, next_state, reward, done in env.env.P[state][act]:\n                A[act] += prob * (reward + discount_factor*V[next_state])\n        return A\n    \n    V = np.zeros(env.env.nS)\n    while True:\n        delta = 0  #checker for improvements across states\n        for state in range(env.env.nS):\n            act_values = one_step_lookahead(state,V)  #lookahead one step\n            best_act_value = np.max(act_values) #get best action value\n            delta = max(delta,np.abs(best_act_value - V[state]))  #find max delta across all states\n            V[state] = best_act_value  #update value to best action value\n        if delta < theta:  #if max improvement less than threshold\n            break\n    policy = np.zeros([env.env.nS, env.env.nA])\n    for state in range(env.env.nS):  #for all states, create deterministic policy\n        act_val = one_step_lookahead(state,V)\n        best_action = np.argmax(act_val)\n        policy[state][best_action] = 1\n        \n    \n    # Implement!\n    return policy, V","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07005b7bf617aa822c551f62165c592b4cfedf2e"},"cell_type":"markdown","source":"Let's evaluate the random policy. "},{"metadata":{"trusted":true,"_uuid":"9485c7389a2f19316b2522b4ca146fafd5da0adb"},"cell_type":"code","source":"env = gym.make('Taxi-v2')\nrandom_policy = np.ones([env.env.nS, env.env.nA]) / env.env.nA\npolicy_eval(random_policy,env,discount_factor=0.95)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5fc5845a3b61562c906f7d382f5f2f34cf52ba27"},"cell_type":"markdown","source":"Now let's use policy iteration to improve our policy. "},{"metadata":{"trusted":true,"_uuid":"9f5860638276898669d306fd24463196cdfa8ca4"},"cell_type":"code","source":"pol_iter_policy = policy_iteration(env,policy_eval,discount_factor=0.99)\npol_iter_policy[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1234eb5581646898799b7230bede11c65bfd1e2b"},"cell_type":"markdown","source":"Next, we use value iteration to improve our policy."},{"metadata":{"trusted":true,"_uuid":"f2ddadb26d1665b1184022e1e4fd84034adf3916"},"cell_type":"code","source":"val_iter_policy = value_iteration(env,discount_factor=0.99)\nval_iter_policy[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0dae910a3baf043edf1149ebf88f6c2ee994b9ac"},"cell_type":"code","source":"for x in range(len(pol_iter_policy[0])):\n    if not (pol_iter_policy[0][x] == val_iter_policy[0][x]).all():\n        print(\"Not the same Policy\")\n        break\nprint(\"Same Policy\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b09936fb6979464425d799859d916b924b03fa80"},"cell_type":"markdown","source":"We can see that policy-iterated and value-iterated policies would converge to the same policy. "},{"metadata":{"trusted":true,"_uuid":"a9cef08320b520222a94c7fb9637fe0983f49c85"},"cell_type":"code","source":"def count(policy):\n    curr_state = env.reset()\n    counter = 0\n    reward = None\n    while reward != 20:\n        state, reward, done, info = env.step(np.argmax(policy[curr_state]))  \n        curr_state = state\n        counter += 1\n    return counter","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a13fa862a0c5ba303ad344146f22dc932211d44b"},"cell_type":"code","source":"pol_count = count(pol_iter_policy[0])\npol_counts = [count(pol_iter_policy[0]) for i in range(10000)]\nprint(\"An agent using a policy which has been improved using policy-iterated takes about an average of \" + str(int(np.mean(pol_counts)))\n      + \" steps to successfully complete its mission.\")\nsns.distplot(pol_counts)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6c534c76018e7e2e50d7e395a43a91984efaff50"},"cell_type":"code","source":"val_count = count(val_iter_policy[0])\nval_counts = [count(val_iter_policy[0]) for i in range(10000)]\nprint(\"An agent using a policy which has been value-iterated takes about an average of \" + str(int(np.mean(val_counts)))\n      + \" steps to successfully complete its mission.\")\nsns.distplot(val_counts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5470f4214e35665927617825d3831633a4092212"},"cell_type":"markdown","source":"**We can see that the improved policy is definitely much more efficient than the random search.**"},{"metadata":{"_uuid":"279fdaba96989912e3edc7d3473a88ddef6500ef"},"cell_type":"markdown","source":"# **4. Now, let's implement Q-learning with epsilon-greedy method** \n\nThe algorithm for Q-learning is as follows:\n\n- Initialize the Q-table by all zeros.\n- Start exploring actions: For each state, select any one among all possible actions for the current state (S).\n- Travel to the next state (S') as a result of that action (a).\n- For all possible actions from the state (S') select the one with the highest Q-value with probability (1-epsilon) and select a random action with probability (epsilon). This is to balance the exploration and exploitation actions.\n- Update Q-table values using the equation:  **Q(state,action) <- (1−α) * Q(state,action) + α(reward + γ * max<sub>a</sub>Q(next state,all actions))**\n- Set the next state as the current state.\n- If goal state is reached, then end and repeat the process."},{"metadata":{"trusted":true,"_uuid":"eac8b1cafdbc5dd6a6f83f37e4b6f1e9981e7332"},"cell_type":"code","source":"import random\nfrom IPython.display import clear_output\n\ndef Q_learning_train(env,alpha,gamma,epsilon,episodes): \n    \"\"\"Q Learning Algorithm with epsilon greedy \n\n    Args:\n        env: Environment \n        alpha: Learning Rate --> Extent to which our Q-values are being updated in every iteration.\n        gamma: Discount Rate --> How much importance we want to give to future rewards\n        epsilon: Probability of selecting random action instead of the 'optimal' action\n        episodes: No. of episodes to train on\n\n    Returns:\n        Q-learning Trained policy\n\n    \"\"\"\n    %%time\n    \"\"\"Training the agent\"\"\"\n\n    # For plotting metrics\n    all_epochs = []\n    all_penalties = []\n    \n    #Initialize Q table of 500 x 6 size (500 states and 6 actions) with all zeroes\n    q_table = np.zeros([env.observation_space.n, env.action_space.n])  \n    \n    for i in range(1, episodes+1):\n        state = env.reset()\n\n        epochs, penalties, reward, = 0, 0, 0\n        done = False\n\n        while not done:\n            if random.uniform(0, 1) < epsilon:\n                action = env.action_space.sample() # Explore action space randomly\n            else:\n                action = np.argmax(q_table[state]) # Exploit learned values by choosing optimal values\n\n            next_state, reward, done, info = env.step(action) \n\n            old_value = q_table[state, action]\n            next_max = np.max(q_table[next_state])\n\n            new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)\n            q_table[state, action] = new_value\n\n            if reward == -10:\n                penalties += 1\n\n            state = next_state\n            epochs += 1\n\n        if i % 100 == 0:\n            clear_output(wait=True)\n            print(f\"Episode: {i}\")\n    # Start with a random policy\n    policy = np.ones([env.env.nS, env.env.nA]) / env.env.nA\n\n    for state in range(env.env.nS):  #for each states\n        best_act = np.argmax(q_table[state]) #find best action\n        policy[state] = np.eye(env.env.nA)[best_act]  #update \n        \n    print(\"Training finished.\\n\")\n    return policy, q_table","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4d1731a715762ac303703946d2d67c1a31e7230b"},"cell_type":"code","source":"env = gym.make('Taxi-v2')\nenv.reset()\nQ_learn_pol = Q_learning_train(env,0.2,0.95,0.1,100000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"68e98ba01e503d5df27d93f502e03cb82f79f2dc"},"cell_type":"code","source":"Q_Learning_counts = count(Q_learn_pol[0])\nQ_counts = [count(Q_learn_pol[0]) for i in range(1000)]\nprint(\"An agent using a policy which has been improved using Q-learning takes about an average of \" + str(int(np.mean(Q_counts)))\n      + \" steps to successfully complete its mission.\")\nsns.distplot(Q_counts)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cda548224d3bc9bae9dcc4feeeba27150467eae1"},"cell_type":"markdown","source":"# **5. Let's watch how our optimal policies works in action**"},{"metadata":{"trusted":true,"_uuid":"3f74eb8c19f6a13da3e5a504488030b3046e7c3c"},"cell_type":"code","source":"def view_policy(policy):\n    curr_state = env.reset()\n    counter = 0\n    reward = None\n    while reward != 20:\n        state, reward, done, info = env.step(np.argmax(policy[0][curr_state])) \n        curr_state = state\n        counter += 1\n        env.env.s = curr_state\n        env.render()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4553e3873675023d4c229b69718927523a00685"},"cell_type":"code","source":"view_policy(pol_iter_policy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ede64a2e93602ca827d1bf5df4fa764030128ebd"},"cell_type":"code","source":"view_policy(val_iter_policy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e69eac5e043e6ce56e499f1bead7f800389eaa0"},"cell_type":"code","source":"view_policy(Q_learn_pol)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cab257b4ec990f66acf5d2fcf59c7c155ab6b4dc"},"cell_type":"markdown","source":"We can also view it animated by running the function below. "},{"metadata":{"trusted":true,"_uuid":"52b4079c117a61c15bb53da6d099542e6f2d8a39"},"cell_type":"code","source":"from IPython.display import clear_output\n\nfrom matplotlib import animation\nfrom IPython.display import display\n\n\ndef view_policy_anim(policy):\n    penalties, reward = 0, 0\n\n    frames = [] # for animation\n\n    done = False\n    curr_state = env.reset()\n    while not done:\n        action = np.argmax(policy[0][curr_state])\n        state, reward, done, info = env.step(action)\n        curr_state = state\n        if reward == -10:\n            penalties += 1\n\n        # Put each rendered frame into dict for animation\n        frames.append({\n            'frame': env.render(mode='ansi'),\n            'state': state,\n            'action': action,\n            'reward': reward\n            }\n        )\n    def print_frames(frames):\n        for i, frame in enumerate(frames):\n            clear_output(wait=True)\n            print(frame['frame'].getvalue())\n            print(f\"Timestep: {i + 1}\")\n            print(f\"State: {frame['state']}\")\n            print(f\"Action: {frame['action']}\")\n            print(f\"Reward: {frame['reward']}\")\n            sleep(.2)\n\n    print_frames(frames)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"513cad49ef1bd1bc2998e839931700bb3b85427a"},"cell_type":"code","source":"view_policy_anim(val_iter_policy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6f1087fb3a6088849d5073c6ca39cfbe5e417588"},"cell_type":"code","source":"view_policy_anim(pol_iter_policy)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c728056c17ffb40e59a936cce999139f4b77e13e"},"cell_type":"code","source":"view_policy_anim(Q_learn_pol)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1e0d1f18ade5caaba44d12f92ec0eb841cb5291"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ad4f7583d845415f884da2f90f1a0b7457197af6"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f458e9a2833f9a2ffbbcef2dce9e8f0cc7781a3"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}