reinforcement-learning-for-meal-planning-in-python.ipynb

{"cells":[{"metadata":{"_uuid":"6624fb3ca54ae50cf2ca77fc2cd0c04ff713ac35"},"cell_type":"markdown","source":"# Reinforcement Learning for Meal Planning based on Meeting a Set Budget and Personal Preferences"},{"metadata":{"_uuid":"c53ce7c333e0bf6f357b3efe990a6e044257e831"},"cell_type":"markdown","source":"## Aim\n\nWhen food shopping, there are many different products for the same ingredient to choose from in supermarkets. Some are less expensive, others are of higher quality. I would like to create a model that, for the required ingredients, can select the optimal products required to make a meal that is both:\n\n1. Within my budget\n\n2. Meets my personal preferences\n\nTo do this, I will first build a very simple model that can recommend the products that are below my budgets before introducing my preferences.\n\nThe reason we use a model is so that we could, in theory, scale the problem to consider more and more ingredients and products that would cause the problemt to then be beyond the possibility of any mental calculations.\n"},{"metadata":{"_uuid":"309d875421b01fe337f15f24283f9a6eb8456d7d"},"cell_type":"markdown","source":"## Method\n\nTo achieve this, I will be bulding a simple reinforcement learning model and use Montel Carlo learning to find the optimal combination of products.\n\nFirst, let us formally define the parts of our model as a Markov Decision Process:\n\n- We have a finite number of ingredients required to make any meal and are considered to be our **States**\n- There are the finite possible products for each ingredient and are therefore the **Actions of each state**\n- Our preferences become the **Individual Rewards** for selecting each product, we will cover this in more detail later\n\nMonte Carlo learning takes the combined the quality of each step towards reaching an end goal and requires that, in order to assess the quality of any step, we must wait and see the outcome of the whole combination. \n\nMonte Carlo is often avoided due to the time required to go through the whole process before being able to learn. However, in our problem it is required as our final check when establishing whether the combination of products selected is good or bad is to add up the real cost of those selected and check whether or not this is below or above our budget. Futhermore, at least at this stage, we will not be considering more than a few ingredients and so the time taken is not significant in this regard.\n\n"},{"metadata":{"_uuid":"1ced4abbbe247e40ed55618e1600a63399215966"},"cell_type":"markdown","source":"## Sample Data\n\nFor this demonstration, I have created some sample data for a meal where we have 4 ingredients and 9 products, as shown in the digram below. \n\nWe need to select one product for each ingredient in the meal.\n\nThis means we have 2 x 2 x 2 x 3 = 24 possible selections of products for the 4 ingredients.\n\nI have also included the real cost for each product and V_0. \n\nV_0 is simply the initial quality of each product to meet our requirements and we set this to 0 for each.\n"},{"metadata":{"_uuid":"bece19a36a9588090a5f84b242fb31f712986bdb"},"cell_type":"markdown","source":"![States1.png](attachment:States1.png)","attachments":{"States1.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAA2gAAAHnCAYAAADNb34ZAAAAAXNSR0ICQMB9xQAAAAlwSFlzAAAXEgAAFxIBZ5/SUgAAABl0RVh0U29mdHdhcmUATWljcm9zb2Z0IE9mZmljZX/tNXEAACB/SURBVHja7d0vdOrK2gfgLZFIJBJZWYmsrKxEIiuRXFWJrEQikZXISmQlshJZ2ZvcM5yTOyeB8C8k4dlrPev7bkvh7Lz7TeeXTGb+/Oc///kDAADA7TkIAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgOQgAAAACWhMP2J8/48Qqx9Dxgcb2dTcxK+jtZWIaPCceHTNo5TlglFgkvhK/GetwftD70Mz+Tnv7o+B3/Hvmd3w6xn8Q0Jp5Av+JTtw7344RNLa3pwV9XeQrnPA7jh80vv+H6e/wkr2fDuh6jhs0pr/7R/5+T20Tk1v+jle84xP4voIOHSdoZG/PTziBpxaOHzS27zuJtxP6fuX4QWP6fHji7/fUJvEkoNW/yKsDhXx3nOCuAlpq6hhCI8PZ+tRBm2MIdxHQdpYCWn0L3CtRwG9TnqA1Aa0fDebS588+TW+GVvT8256pTemU56fwWMMw9P5SQIPWBLSP8PWs1/D1ojH+m4BWzwKPo0L9hIeJ4wI+OV7QroCWeV2v4FmVvuMIjen3wZ7pTIM9P/cQLtKY4gjNDmjzPa8f7Vlv4kFAq1+B4yvni3CFrXTRgWYHtPDapedPodH9Psvp4Z+yF1rSO2uOI7QzoIWfGRSEtMqmOipeueLmrQDzEqY9bXOmR5jmCO0NaHmvHTiO0Ihe7xQMvGaODwhomZ8rupBTyRhf8coVd1JUoILB2ovjBu0LaGFwt/EMGrRqsOYiCwho8c/1Ci7mVPIok+KVK+66aGntgmmOlt6GlgW0sGBA3nOnY8cQGtPrU6sygoBW8meXt1q5WfEOF2ewLz0XTHP8Mc0RGh/QPsPWGquC5bjTPp84ftD4XrfoBwhoeT87u9WWWop3uDjTQ8+YFZzwR44fNHrQdki6THfP8YNG9frC4l4goJX82cmtzheKd7g4X4cKUzDN8cPxg1YHtJ1XxxAa0+srm82DgFbyZ8cCWj2L+ljm4cA90xy7jiM0NqBlpzh+Hbqb5jhCI3rdHTQQ0MoGtLdbrfiqeMcVpnAJ/YIBnsUDoLkBrV9wov8seB7NKnBQ/16fegYNBLSSP7uwSEg9i7o5Y9qTkz60LKCF13YK7qhZMATq3+svVnEEAa3kz+ZdkH0W0OpX0FNYRABaFNDC6ye214BG9vpjwe/qB8cHBLTMzz3cclyveMWFeb9QQHt1PKF1AW2U8/qlYwm17/Vuwe9q/QsCWvbnlre82654xYX5jorynVkwoEjetKdPxxNaF9DeLRQCje335al30cIU5wfHEdob0AouwlZ600Xx8gvzdMpDgeltz4KC9h1XaEdAC1Okfsqs8ArUst8HBT2cfu3lwKAtvVj77ThC+wJaGMd/FIzlv4sWChTQbjtYG5T82ZXFA6AVPT8LK75lFe2XtnYcoVE9P9vzaMJnuEue7fv/m1XjGEKjA9o6+t1+aDud9OLNsNL/bsX7VyHz9jT7OuLn8za1+3JsoXEBrazKT9zA2T3fy/ldX9bWMYRGB7Rjf8ePKv/vVrx/FfLpnGdL9kxztEcStC+gbTyPAo0OaR8n9L0VW+E+Atr6VuN3xft3IWfnLr9bMM3RptVQ374fH7vH4S2uqAFX6f9Rybtp31Zmhsb1d6fgmdN9F14Xt54Zo3j/LmQ/mpc6PPE9XjPv8Wo/NKh97z+GK217OVbQ2nNAuu/Rc/i9vQwXYuanjgWAWo3thyV0a/PfrHAAAAACGgAAAAIaAACAgFb/A3LeUpx7Ob7Q/n53XOE++t0xhfscz1fR/4oooIF+1+eg3/U66HcBDQAAAAENAABAQAMAAEBAAwAAENAAAAAQ0AAAAAQ0AAAABDQAAAABDQAAAAENAABAQAMAAEBAAwAAENAAAAAQ0AAAAAQ0AAAABDQAAAABDQAAAAENAABAQAMAAEBAAwAAENAcBAAAAAENAAAAAQ0AAEBAAwAAQEADAAAQ0AAAABDQTjoof/50Ei+JaWIVeUuMEl3HCvQ7oN8B/S6gXa+QvcR74ifxW8IiMXDsQL8D+h3Q7wLaZYuZpuhtyULG0hTecRxBvwP6HdDvAtr5xXw/sZBZK9MiQL8D+h3Q7wLa6YVM56Z+XKCYO2sncdDvgH4H9LuAdlpBZxcs5t/JW7OAfgf0O6DfBbTjivl0hWLuTDUM6HdAvwP6XUArV8z0Vuj3FQuarhrT0zig3wH9Duh3Ae1wQV+vWMydmeYB/Q7od0C/C2iHC7qpoKA/HigG/Q7od0C/C2j7i9mvoJg7zxoI9Dug3wH9LqAVF/S1woLONRHod0C/A/pdQCsu6LzCgn5pItDvgH4H9LuAVlzQVYUF/dZEoN8B/Q7odwGtuKBfFRb0VxOBfgf0O6DfBbTigm6cwEG/63fQ7/od9LuAVo+CVnlL9EcTgX4H9Dug3wW04oIuKizoRhOBfgf0O6DfBbTigk4rLOhSE4F+B/Q7oN8FtOKCDiss6FgTgX4H9Dug3wW0/UXdVlTQviYC/Q7od0C/C2j7Czoz/QH0u34H/a7fQb/Xrd/vtaC9dEWWKxf0QfOAfgf0O6DfBbRyRX1zdQ30u34H/a7fQb/Xqd/vuaCdxOcVivmdJnpNA/od0O+AfhfQjitqPxTgYhvZpavKaBjQ74B+B/S7gHZ6UTcXKObWyRv0O6DfAf0uoJ1f1G5idUYxvzw0DPod0O+AfhfQLlvYlyNvkaYp+9WxA/0O6HdAvwto1ylqJxR2UVDctIjLxChN6o4Z6HdAvwP6XUADAABoW8B0EAAAAAQ0AAAABDQAAAABDQAAAAENAABAQAMAAEBAAyo7mfy150i6p8gq3XfEMQEAuPOAlvzppYNEhYWb9F8/uwlkFZ/nuAMAAlp9g9lHZnD4rrjQzoCW/HlKfGc+69HxBwAEtPoMCieJn+zAMDFXXGhXQAsXYhZRr6eGjj8AIKDdfjD4kPjMGawJaNCygJb8GSW2Bf0uoAEAAtqNB4Jv0QBtHRYnENCgRQEtvGd8IWYeTXEU0AAAAe2Gg8BhZmCWXlF/DV+fCmjQuoA2z7zn5+55s+TPRkADAAS0egW090Q383UBDdoZ0NILMePo6wIaANR3TNALY/ahFdYbENDSK+BhimK6V9JXGHytwkBsXOLnO3nLawtoUM+AFr7+Hu6A/YQVV9PFfR5Kvmcn5+sCGgDUd0yQnQEzckxqGtBCgv4qeMg/6+uUZbMFNKhfQAsXY3729Pv4xM8S0KC955KXcCFnpb/h/gJaOjYI/b+61610qixUdgW2n8xds3nOIgDfx94OFdCgXgEtOjlvM3fN45A2EdCA5M8g2tM0NXVs4D4CWjgHfDoHVBvQfsKgapx9dizz/WF0pf1VQINGB7SdWfS8aLwyY9r3PQEN7vr8MSm42y6gQcsDWnh8aeoccJuA9lDiNbNMQRYCGjQ+oI0KXtuLlsqfHPlZAhq047zxWDCLRkCDOwho4RzwFV203Qpo9SrmU3ZJbQENGh3QRkf07LeABnd3zhhG54zPMMVpLqBB+wNazjngI5wDVgJafQd4GwENmhvQSrz+IToxd474LAENmn/OGGXumI0KBnYCGrQ3oE0z54DnzNcFtBoUcJDZJ+FFQIP7CGjhZ7IBbSCgQaPPAd3wu3wSfie/5D1zHv3+n8SvEdCgMT0/CH0+iVdbLBnQ0umNrznnAAHtBsVMpzEuSiy3L6BB+wPa9pSgJaBB7QZpn3t+n8/3BbWc9xPQoN49/7xnC51p2YC25/0FtAqL2S25D5qABvcT0DYCGjS678cH9jrM7nHaKfmeAhrUt+cnJfp9GQhodQ5oIZyts+ErnNQf9gzwBDRof0DLDuz6Aho0qufj50hXYbpTP0xdeol+9/dLvq+ABvXs+eechX3GYWrzUxTKfgW0+ge0UXQVrVtigCegQYsDWtj35KhAJ6BBrXo+u6n0ougOWXjG5PuI9xXQoJ49v4rCWSfnNS85d9UFtJoGtHmZ/Y4ENGhPQDv0zEl04ebYfhfQ4Lb9/hT1e++C7y2gQf16Pr5j3t/z2mcBrRkBLXugx3teNxTQoDUBbX1gBbfs1feZgAaN6vdZ9nmTC7+3gAb16/nsOHtV4vULAa3+AS17Il8UvOYpWtFNQINmB7TCkBb168+xV98FNLh5v6+vNYAS0KCWPV9qNlzB6wW0mga055wld5/DMyhPUYA7GNAye6xkraJBYfz9Rw0GlQW0TRTAFmFK4zjnIeLZnvcc5PTyNLqYM8/5fkdN4Kr9vjl18CWgQSN7fnVMzwtoDQho4WCX2fssG7Q2JQaBx3BXDaoLaPPoGbMii31hKrrTdoyhmsBV+/37Wv0moEHze15Aa05A64Q7X9uCaVDD8LrdcylfBe/TLbnnSuxNg8FVe7yX6bf38LXHaCrUznfJKRLjEwPaQE3gqv2evYP2LKDBXfX8i4DWkoAWHfhhWIbzOX4+JQSw4aEV4IDahrTHnK8PQs+/HLPfGVDbXi+1+JeABq3s+dGRfSygNSGgAQCNHqzNrzVDRUCDWvb8Ip4lI6AJaABAfQZr02jKcueC7y2gQf16fpLt+QOvfYgeaRLQBDQA4MqDtV70PPh0z2vTZ1G/yj66IKBBI3r+teB1g+h5NQFNQAMAKhqwxVvkzLLPmIaB2iwzqOsLaNDono+3yRln9zEN/3ub2WJHQBPQAIAKB2u9cGcsb5XWeLXlbc6iYIMTV2kdO/5wk57v59wd+81ZrfkzLAq2N6CFvZBPOQeMWn+s/YMDAE4csHVL7HOaDt4ecn52eOLgbOrYw01D2npPf76HrbWGJQLay4nngEnrj7N/bADAmYO2x7BwyCJMT0q95W27kfmZ3f6o0yP1HXO4ab93w92vaabf59l+D6/Z9exgzzlgeoKegAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoDkIAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoLX5oPz500m8JKaJVeQtMUp0HSvQ74B+B/S7gHa9QvYS74mfxG8Jy8SjYwf6HdDvgH4X0C5bzEliW7KQeYWtZQIH9Dvod/0O+r05/a6Qf93+XJ5YyKyvRF9zgH4H9Dug3wW004rZTawvUMydjZM46HdAvwP6XUA7raDLCxYzm7w7Ggb0O6DfAf0uoJUv5vQKxdxZaBjQ74B+B/S7gFaumL0jVnY5ldWfQL8D+h3Q7wJaiYLOrlzM1FrzgH4H9Dug3wW0wwXdVlDQXw8Ug34H9Dug3wW0/cUcVlTM1KsGAv0O6HdAvwtoxQV9q7CgS00E+h3Q74B+F9CKC7qssKAbTQT6HdDvgH4X0IoL+llhQX80Eeh3QL8D+l1AKy7opsKC/moi0O+Afgf0u4BWXNC1Ezjod/0O+l2/g34X0OpR0A9z1EG/63fQ7/od9Ltn0OpR0PcKC/qpiUC/A/od0O8CWnFBnyss6FQTgX4H9Dug3wW04oJ209VYKiroUBOBfgf0O6DfBbT9Ra1i74RvDQT6HdDvgH4X0A4X9KGCgo40D+h3QL8D+l1Au33qXmsc0O+Afgf0u4BWvqDdK21yl86HHWga0O+Afgf0u4B2XFEfr/CA4YuGAf0O6HdAvwtopxd1e6GkPdIooN8B/Q7odwHtvKL2003ozijm1pK7oN8B/Q7odwHtsoV9SZfTPDJlv6XzXx0/0O+Afgf0u4B2+aJ2QmEXBcXdhhVjRk7coN8B/Q7odwENAACgjQHTQQAAABDQAAAAENAAAAAENAAAAAQ0AAAAAQ0AAAABDQAAQEADAABAQAMAABDQAAAAENAAAAAENAAAAAQ0AAAAAQ1g/4nlz5/HxNCxAAAQ0IDbhrNJ4jd4d0wAAO4ooCV/OonnxDTxkViF/3+U6CsyVN6T00xAm1/4vfuJcWIWen0RPu8pPRc4/gCAgHa7QWAvHfwlfjKDwVj6vYlCQ7MDWjpdMgSy3z2+0qmVagAACGjVDwAHie2Bwdrvta7iA9UFtOTP6xG9nnpSBwBAQKt2ADjKDMa+w3Snp3CVfRiegdkatEErAlr2ztlnCGy7Xn8K0xyzvb4x3REAENCqHQA+hwA22fOafjT9caXg0MiAtkys901fjBYmSY3UAgAQ0Oo3UMwO2rYKDs0LaEd87lfmc2dqAQAIaMcPqLqZqUqpwYXffxhdVe8pOtwuoIU727t+71z4c+eZz/1QCwC46XigE343r2y7U/OAFkLZOExZKnrI//0SYSr58xC9r+dS4AYBLWxc/ZnT6+tLTUcMz6Lu3nehFtDYc0j2Qk7XMYFG9/Lfv/NPPAfc5di96kL1omlI+6TPmD2c+XlP2SX3NQtUH9CiRX2KTC/wuQsbZENjzxvD0MPfBeOBmbAG7Q1omXNA3irt6Xnh7Z7OAVUXahTtWTSOpjhOooU9Ps78vPfMey01C1Qe0LbR6oq7FRhn4X9nT8DjMz6zE33Ws1pAI84X6ayaj5IXbtNB2tBxg/YEtHDz5phzwF3sd1p1oYZhRbbhntc8RsUYnPhZ8YDtRbNA5QGt8I5WGJhlpzpvzvjM5ygUms4MzThfvOZsOP8RziOLnAs53/obWhXQJjnngOWec8DXPZwD6lrQz3OvhIdbofZFgtsHtNme1/ai1z6f8HmdKOi9qQM05nyxG5yt8vYrDf39Hp0n9Di0J6DtxgwfeXfHwjkg3u/0VUC7TUHn5xQhvesWTZUcahS4SUBblXj96pxnx6Krby7GQLPOF4MyF2aiC7dWaYX2BLTBoXF6zoXY1j+21ISANj3yZzvRgG+uSeBmAW1e4vWvp550w0qt2YsxT2oArTyvjO1rCu0LaEe8V/Zi7JeAdt3CPYaT7jT4COHq+4yAtoiupnc1CdQ6oI2OueMWnfi/rdwItQtSs/C7fLvb/+jchXuifU2tygz1CWDT8MzYJtzpnu3uiF04oI0u8cy6gFZ8gDshBW9KrtgyPeK936KFAgYaCGof0J6OPemGBUayW3asTG2Em/b9oGCvw6z0+32DM2hFzw8LlsT/+/nzcF64VEDLzrZZC2iXD2fxUpqbMLiaZe6krY8NaNH0hx9TnaAxAS27AuNnyfPIKtrsuuvYw816/jGaarzry1VOaJuf+BnvnkGD2vT8S07Pb3JmwX1dMKDN7+nxpaoL+h7d4XoqUYRpifcdCWfQ2ICWvbiyKBHOPoQzqFXPr6MBWT/6fi/ze/3jxM/4tm0O1KLfO1E/psHsIfr+JO+u2pmfub2n582rLuqmzMqKxwS0KJylRhoIGhXQ3ksuyR+Hs82p06WAi/X7S7RHWXfPa59OGVhFn/FjOjPctOdHUc/3S47Pfy/4mfZBu2BBSz8oWDagCWdQ+4D2dWDAFl8VGwtn0Kh+v+oehOF50+9D5wigsp7/POImyvjcgBbuwG/v7Q76zQJa0aAtvG5zqPg54exV40DtAtpu88lOiZN37lWxnHC2Fc6gNv2eHTgNrvD+82OeUQWu3vO/ZXv+Eqs4hhUi7+7501ueyN9yvv+QsyLMNOd1w1NXegQqD2i7Z8Ueotc9xys+lRigbeP3AW7W671LLQBQ8P6TaGqjlZmhQT1/bkDLWZ29L6BVM3D7yKzc+Fmw6kteQFuXXKI/9mNBAai0z+c5y2zHqzztpiz2Dv0yONJSPeCqvf54raXvo+fOLAwC9ev572sGtHh65L0tAHiL4i4PDKpmUUHzAtrnGYO2u0nfUIOANgwDre2BvZG6ZU7uR/pSD7hqr19lb7Jw3vgxSwZq1/PDY3r+1IAW7Y96l48x3bLAH5k7Ybudx5/C97vh+6u81R7D1Kjpiaz+BNft76fQux+7CyLhTthbpufTu+SLsPFkp0TgO4Ur7nDdXs9eTd9e6D3jRx1mjjXUpucfrj3FMYwh7v4CjX9wAMApg7VemcW/zghnrd+MFtrc88cGtHAO+HGBRkADAE4fsGUHU8/CGbS+50tvb3XMHTfnAAENALjMYG1ZdmGe8PjCoMTA7G6W0oa29nzo902ZgCacCWgAwOUGaw9lHuYPz6ulg7VV9PVBHM48Kw617vmnQyushqmN/1pxveD9BsKZgAYAXHbAFq/OvAwLAA3DXmbZ78cBbVWwFcch74493Kzn19EWVvOwYvMwLAi2zexdtjkQ0OKV2VclvQloAAD5A6zeEdvfvEQ/uzljK42e4w836fl+tGdxnnV43epAQDu1/39bfYz9QwMALjBoe9uz52G6rcZDzs+sThycbU2FhJv2e7egf9OLLu+7/szeQS94n/WJ54BNq4+vf2QAwAUHbr0w1WlHkIL29vsg0+v9gtcMz92G4+6Oq4MAAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAIaAAAAAhoAAAAAhoAAAACGgAAgIAGAACAgAYAACCgAQAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAUIuQ8+dPJ/GSmCZWkbfEKNEV0AAAAK4XzHqJ98RP4reEZeJRQAMAALhsOJsktiWDWV5Q6wpoAAAA5wWzTghYv2f6SvQFNAAAgNPCWTexvkA429ncOqQpLAAA0NSAtrxgOMveSesIaAAAAOXD2fQK4WxnIaABAACUC2e9I1ZqPNVNVndUYAAAoGkBbXblcJZaC2gAAACHA9q2goD2e4sFQxQYAABoUjgbVhTOUq8CGgAAQHFAe6swoC0FNAAAgOKAtqwwoG0ENAAAgOKA9llhQPsR0AAAAIoD2qbCgPYroAEAABQHtLWABgAAUI+A9uEZNAAAgHoEtPcKA9qngAYAAFAc0J4rDGhTAQ0AAKA4oHXT1RUrCmhDAQ0AAGB/SKtiL7Tvm/zdFBgAAGhYQHuoIKCNBDQAAIByIe2ad9HWN/t7KS4AANDAgNa90qbV6fNtAwENAADguJD2eIUFQ15u+ndSWAAAoOEhbXuhO2ejm/99FBUAAGh4SOunm0qfEc62t1hSX0ADAADaHNRe0uXxj7xr9pY+z1abv4NCAgAALQppnRDUFgVhbRtWgBzVKZgJaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAABHhow/f7qJTeI38eWYCGgAAMDtAlo/hLP/KfkzD4lxYho8Jx4FNAAAgAoCWvKnF8LYd/b1kXViKKABAABcKaCFO2bbPcEs9iKgAQAAXCegTTOv+Uq8pnfKwvNr6f+dRwEtvcvWE9AAAAAuH9AmYSGRpz3vM4lC2lhAAwAAuHBAK/k+nWga5LuABgAAUByi0oU+OtcIaOG9Vpn3+hDQAAAA/j80PSUWmb3OdqstTsNdr0sGtHXmvWYCGgAAwD+B6fXAaouf6f5lF5zi+OMZNAAAgH8HpreclRXn4c5Z9k7X9kIBbR59VkdAAwAAhLO/njXL3s36yHn2bJi3t9kJn5XukbbMvMfPvtUeBTQAAODeAtoket6sU/C6frjbVSqghSmTq+AzZ8pk+vXh3Rxn/9gAAIASAS27IMjrgdcOjwhoqwPPs43T0CegAQAA/BOkssGpd+C1/RPvoK3ypkiGKY4TAQ0AABDOjlw2/xLL7Cd/XnKmPL4JaAAAwL0HtOyUxU0VAS3zXu9RSHsU0AAAAAEtLHdfcUBL90L7upfNqv2DAwAADoWkQdVTHKP3m2UXDhHQAACAew5o3WstElLy818FNAAAgH9CUnaT6vGB1z5fOKAtTHEEAAD4JyRlpxmu97zuIV4qv+B1jyU/d3BMOBTQAACAewhoD9E0x1nOa57z9jErCF3/C3rpz+z5zMfwmt17pYuFdAQ0AABASPv/qYb/W3I/LIM/jfYsW2eDVYmwtw2bVL+F95rl7IGW3kUbtP4Y+4cGAACUDGidnJAW+wiLiqwOTHE89D6/UeBr9f5nAhoAAHBqUHsNQewnmn74knnNW4nn1Qbhbtk6J5R9h5DX6mfOBDQAAOCSYe3hgu/Vu5c7ZQIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgAYAAICABgAAIKABAAAgoAEAAAhoAAAACGgAAAACGgAAAAIaAACAgOYgAAAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAAAgoAEAACCgAQAACGgAAAAIaAAAAAIaAAAAAhoAAICABgAAgIAGAADQVv8FxpKpqPlMrwYAAAAASUVORK5CYII="}}},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"33386d7b6704a1b51842be28a4a0857ee6f58b43"},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport time\n","execution_count":1,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d7d841000db7cfa3136e936d45492b1e38b87a0"},"cell_type":"code","source":"#data = pd.read_csv('C:\\\\Users\\\\kr015\\\\Documents\\\\Machine Learning\\Meal Plan\\\\Final\\\\SampleData.csv')\ndata = pd.read_csv(\"../input/SampleData.csv\")\n","execution_count":3,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40c2e12ef48d0c06bf23191d2791b4e81cbf9725"},"cell_type":"code","source":"data","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"c8f061ce951104cad898aa85cd07e6d16c0ebc13"},"cell_type":"markdown","source":"## Applying the Model in Theory\n\nFor now, I will not introduce any individual rewards for the procuts. Intead, I will simply focus on whether the combination of products selected is below our budget or not. This outcome is defined as the **Terminal Reward** of our problem.\n\nFor example, say we have a budget of £30, then the choice:\n\n$$a1 \\rightarrow b1 \\rightarrow c1 \\rightarrow d1$$\n    \nThen the real cost of this selection is:\n\n$$£10 + £8 + £3 + £8 = £29 < £30$$\n\nAnd therefore, our terminal reward is:\n\n$$R_T = +1$$\n    \nWhereas, \n\n$$a2 \\rightarrow b2 \\rightarrow c2 \\rightarrow d1$$\n    \nThen the real cost of this selection is:\n\n$$£6 + £11 + £7 + £8 = £32 > £30$$\n\nAnd therefore, our terminal reward is:\n\n$$R_T = -1$$\n    \n\nFor now, we are simply telling our model whether the choice is good or bad and will observe what this does to the results."},{"metadata":{"_uuid":"d8cb3d4390dcad6bd7744563357464a93a5db3f8"},"cell_type":"markdown","source":"### Model Learning\n\nSo how does our model actually learn? In short, we get our model to try out lots of combinations of proucts and at the end of each tell it whether its choice was good or bad. Over time, it will recognise that some products generally lead to getting a good outcome while others do not.\n\nWhat we end up creating are values for how good each product is, denoted V(a). We have already introduced the initial V(a) for each product but how do we reach go from these initial values to actually being able to make a decision?\n\nFor this, we need an **Update Rule**. This tells the model, after each time it has presented its choice of products and we have told it whether it's selection is good or bad, how to add this to our initial values. \n\nOur update rule is as follows:\n    \n$$V(a) \\leftarrow V(a) + \\alpha*(G - V(a))$$\n    \nThis may look unusual at first but in words we are simply updating the value of any action, V(a), by an amount that is either a little more if the outcome was good or a little less if the outcome was bad. \n\n\nG is the **Return** and is simply to total reward obtained. Currently in our example, this is simply the terminal reward (+1 or -1 accordingly). We will reintroduce this later when we include individual product rewards.\n\nAlpha, $\\alpha$, is the **Learning Rate** and we will demonstrate how this effects the results more later but just for now, the simple explination is: \"The learning rate determines to what extent newly acquired information overrides old information. A factor of 0 makes the agent learn nothing, while a factor of 1 makes the agent consider only the most recent information.\" (https://en.wikipedia.org/wiki/Q-learning)"},{"metadata":{"_uuid":"db4a670d7bcf337d4b8be6f29dee06729fa3ec15"},"cell_type":"markdown","source":"### Small Demo of Updating Values\n\nSo how do we actually use this with our model?\n\nLet us start with a table that has each product and its initial V_0(a):\n\n|Product|V_0(a)|\n|:-----:|:----:|\n|   a1  |   0  |\n|   a2  |   0  |\n|   b1  |   0  |\n|   b2  |   0  |\n|   c1  |   0  |\n|   c2  |   0  |\n|   d1  |   0  |\n|   d2  |   0  |\n|   d3  |   0  |\n\n---\nWe now pick a random selection of products, each cobination is known as an **episode**. We also set $\\alpha = 0.5$ for now just for simpicity in the calculations.\n\ne.g.\n\n$$a1 \\rightarrow b1 \\rightarrow c1 \\rightarrow d1$$ \n\nProvides:\n  \n$$Total Cost = £29 < £30$$\n\nTherefore:\n    \n$$R_T = +1$$\n    \n\n\nNow applying our update rule to a1:\n\n$$V_1(a1) <- V_0(a1) + \\alpha*( G - V_0(a1))$$\n    \n$$\\implies V_1(a1) <- 0 + 0.5*( 1 - 0) = 0.5$$\n\n\nTherefore, all actions that lead to this positive outcome are updated as well to produced the following table with V1(a):\n\n|Product|V_0(a)|V_1(a)|\n|:-----:|:----:|:----:|\n|   a1  |   0  |  0.5 |\n|   a2  |   0  |   0  |\n|   b1  |   0  |  0.5 |\n|   b2  |   0  |   0  |\n|   c1  |   0  |  0.5 |\n|   c2  |   0  |   0  |\n|   d1  |   0  |  0.5 |\n|   d2  |   0  |   0  |\n|   d3  |   0  |   0  |\n\n---\nSo let us pick another random episode:\n\n$$a1 \\rightarrow b2 \\rightarrow c2 \\rightarrow d1$$\n\nProvides:\n  \n$$Total Cost = £36 > £30$$\n\nTherefore:\n    \n$$R_T = -1$$\n    \n\n\nNow applying our update rule to a1:\n\n\n\n$$V_2(a1) \\leftarrow V_1(a1) + \\alpha * ( G - V_1(a1))$$\n\n  \n$$\\implies V_2(a1) \\leftarrow 0.5 + 0.5*( -1 - 0.5) = 0.5 - 0.75$$\n\n\n$$\\implies V_2(a1) \\leftarrow -0.25$$\n\n\n\nand likewise for d1.\n\nWhereas the updated value for b2 is:\n\n$$V_2(b2) \\leftarrow V_1(b2) + \\alpha*( G - V_1(b2))$$\n    \n$$\\implies V_2(b2) \\leftarrow 0 + 0.5*( -1 - 0) = -0.5$$\n    \nand likewise for c2.\n\n\nTherefore, we can add V2(a) to our table:\n\n|Product|V_0(a)|V_1(a)|V_2(a)|\n|:-----:|:----:|:----:|:----:|\n|   a1  |   0  |  0.5 | -0.25|\n|   a2  |   0  |   0  |   0  |\n|   b1  |   0  |  0.5 | 0.5  |\n|   b2  |   0  |   0  | -0.5 |\n|   c1  |   0  |  0.5 | 0.5  |\n|   c2  |   0  |   0  | -0.5 |\n|   d1  |   0  |  0.5 | -0.25|\n|   d2  |   0  |   0  |   0  |\n|   d3  |   0  |   0  |   0  |\n\n\n"},{"metadata":{"_uuid":"9664a75499ca9a2774619bfb109914535f4e962f"},"cell_type":"markdown","source":"## Action Selection\n\nYou may have noticed in the demo, I have simply randomly selected the products in each episode. We could do this but using a completely random selection process may mean that some actions are not selected often enough to know whether they are good or bad.\n\nSimilarly, if we went to other way and decided to select the products greedily, i.e. to ones that currently have the best value, we may miss one that is in fact better but never given a chance. For example, if we chose the best actions from V2(a) we would get a2, b1, c1 and d2 or d3 which both provide a positive terminal reward therefore, if we used a purely greedy selection process, we would never consider any other products as these continue to provide a positiv outcome. \n\nInstead, we implement **epsilon-greedy** action selection where we randomly select products with probablity $\\epsilon$, and greedily select products with probability $1-\\epsilon$ where:\n\n\n$$0 \\leq  \\epsilon \\leq  1$$\n    \nThis means that we are going reach the optimal choice of products quickly as we continue to test whether the 'good' products are in fact optimal but also leaves room for us to also explore other products occasionally just to make sure they aren't as good as our current choice. \n\n"},{"metadata":{"_uuid":"ae39f993f0b53cf5b7c17435ca06f3e255607cf3"},"cell_type":"markdown","source":"# Building and Applying our Model"},{"metadata":{"_uuid":"02cba2536832084bf9781a15123db1fac61a337a"},"cell_type":"markdown","source":"We are now ready to build a simple model as shown in the MCModelv1 function below.\n\nAlthough this seems complex, I have done nothing more than apply the methods previously discussed in such a way that we can vary the inputs and still obtain reqults. Admittedly, this was my first attempt at doing this and so my coding may not be perfectly written but should be sufficient for our requirements. \n\n\nTo calculate the terminal reward, we currently use the following condition to check if the total cost is less or more than our budget:\n\n    if(budget >= episode2['Real_Cost'].sum()):\n        Return = 1  \n    else:\n        Return = -1\n        \n\n"},{"metadata":{"collapsed":true,"trusted":true,"_uuid":"f2f37b0be5ab5ecd0b74a482c6ce1c4e75b1523e"},"cell_type":"code","source":"def MCModelv1(data, alpha, e, epsilon, budget, reward):\n    # Define the States\n    Ingredients = list(set(data['Ingredient']))\n    # Initialise V_0\n    V0 = data['V_0']\n    data['V'] = V0\n    output = []\n    output1 = []\n    output2 = []\n    actioninfull = []\n    #Interate over the number of episodes specified\n    for e in range(0,e):\n        \n        episode_run = []\n        #Introduce epsilon-greedy selection, we randomly select the first episode as V_0(a) = 0 for all actions\n        epsilon = epsilon\n        if e == 0:\n            for i in range(0,len(Ingredients)):\n                episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n            episode_run = episode_run.astype(int)\n        \n        else:\n            for i in range(0,len(Ingredients)):\n                greedyselection = np.random.random_integers(low = 1, high =10)\n                if greedyselection <= (epsilon)*10:\n                    episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n                else:\n                    data_I = data[data['Ingredient'] == (i+1)] \n                    MaxofVforI = data_I[data_I['V'] == data_I['V'].max() ]['Product']\n                    #If multiple max values, take first\n                    MaxofVforI = MaxofVforI.values[0]\n                    episode_run = np.append(episode_run, MaxofVforI)\n                    \n                episode_run = episode_run.astype(int)\n                \n               \n           \n        episode = pd.DataFrame({'Ingredient' : Ingredients, 'Product': episode_run})    \n        episode['Merged_label'] =  (episode['Ingredient']*10 + episode['Product']).astype(float)\n        data['QMerged_label'] = (data['QMerged_label']).astype(float)\n        data['Reward'] = reward\n        episode2 =  episode.merge(data[['QMerged_label','Real_Cost','Reward']], left_on='Merged_label',right_on='QMerged_label', how = 'inner')\n        data = data.drop('Reward',1)\n        \n        # Calculate our terminal reward\n        if(budget >= episode2['Real_Cost'].sum()):\n            Return = 1  \n        else:\n            Return = -1 \n        episode2 = episode2.drop('Reward',1)\n        episode2['Return'] = Return\n        \n        # Apply update rule to actions that were involved in obtaining terminal reward \n        data = data.merge(episode2[['Merged_label','Return']], left_on='QMerged_label',right_on='Merged_label', how = 'outer')\n        data['Return'] = data['Return'].fillna(0)\n        for v in range(0,len(data)):\n            if data.iloc[v,7] == 0:\n                data.iloc[v,5] = data.iloc[v,5] \n            else:\n                data.iloc[v,5]  = data.iloc[v,5]  + alpha*( (data.iloc[v,7]/len(Ingredients)) - data.iloc[v,5] )\n                \n        # Output table    \n        data = data.drop('Merged_label',1)\n        data = data.drop('Return',1)\n        \n        # Output is the Sum of V(a) for all episodes\n        output  = np.append(output, data.iloc[:,-1].sum())\n        \n        # Output 1 and 2 are the Sum of V(a) for for the cheapest actions and rest respectively\n        # I did this so we can copare how they converge whilst applying to such a small sample problem\n        output1 = np.append(output1, data.iloc[[1,2,4,8],-1].sum())\n        output2 = np.append(output2, data.iloc[[0,3,5,6,7],-1].sum())\n        \n        # Ouput to optimal action from the model based on highest V(a)\n        action = pd.DataFrame(data.groupby('Ingredient')['V'].max())\n        action2 = action.merge(data, left_on = 'V',right_on = 'V', how = 'inner')\n        action3 = action2[['Ingredient','Product']]\n        action3 = action3.groupby('Ingredient')['Product'].apply(lambda x :x.iloc[np.random.randint(0, len(x))])\n        \n        # Output the optimal action at each episode so we can see how this changes over time\n        actioninfull = np.append(actioninfull, action3)\n        actioninfull = actioninfull.astype(int)\n        \n        # Rename for clarity\n        SumofV = output\n        SumofVForCheapest = output1\n        SumofVForExpensive = output2\n        OptimalActions = action3\n        ActionsSelectedinTime = actioninfull\n        \n    return(SumofV, SumofVForCheapest, SumofVForExpensive, OptimalActions, data, ActionsSelectedinTime)\n\n\n","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"75bcc1a237213002de82b46ec4b5a34f1704a796"},"cell_type":"markdown","source":"### We now run our model with some sample variables:"},{"metadata":{"trusted":true,"_uuid":"d2b42e574790be8a6cb696f95c1bebd25696d21d"},"cell_type":"code","source":"alpha = 0.1\nnum_episodes = 100\nepsilon = 0.5\nbudget = 30\n\n# Currently not using a reward\nreward = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl = MCModelv1(data=data, alpha = alpha, e = num_episodes,epsilon = epsilon, budget = budget, reward = reward)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"8ba124da82c4b87c648da69f4d8b7e09f682ace6"},"cell_type":"markdown","source":"In our function, we have 6 outputs from the model.\n\n- Mdl[0]: Returns the Sum of all V(a) for each episode\n\n- Mdl[1]: Returns to Sum of V(a) for the cheapest products, possible to define due to the simplicity of our sample data\n\n- Mdl[2]: Returns the Sum of V(a) for the non-cheapest products\n\n- Mdl[3]: Returns the optimal actions of the final episode\n\n- Mdl[4]: Returns the data table with the final V(a) added for each product\n\n- Mdl[5]: Shows the optimal action at each epsiode\n\nThere is a lot to take away from these so let us go through each and establish what we can learn to improve our model.\n\n"},{"metadata":{"_uuid":"bcdb7a66bc747bec906d8f8e7bc2d2cb96999551"},"cell_type":"markdown","source":"#### Optimal actions of final episode\n\nFirst, lets see what the model suggest we should select. In this run it suggests actions, or products, that have a total cost below budget which is good.\n\nHowever, there is still more that we can check to help us understand what is going on.\n\n\nFirst, we can plot the total V for all actions and we see that this is converging which is ideal. We want our model to converge so that as we try more episodes we are 'zoning-in' on the optimal choice of products. The reason the output converges is because we are reducing the amount it learns each time by a factor of $\\alpha$, in this case 0.5. We will show later what happens if we vary this or don't apply this at all.\n\nWe have also plotted the sum of V for the produts we know are cheapest, based on being able to assess the small sample size, and the others seperately. Again, both are converging positively although the cheaper products appear to have slightly higher values.\n\n\n"},{"metadata":{"trusted":false,"_uuid":"5073a92e9ac087ee29ba16b6c89993c7a324dc1f"},"cell_type":"code","source":"print(Mdl[3])\n\nMdl[4]\n","execution_count":6,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"9ad315bedc05747fae1abfabacc8599b2ef5191e"},"cell_type":"code","source":"plt.plot(range(0,num_episodes), Mdl[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":7,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"959651b3ff1f479edd91fa7e272bf1a3e2688fea"},"cell_type":"code","source":"plt.plot(range(0,num_episodes), Mdl[1],range(0,num_episodes), Mdl[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"38f203c11214be3fb8657775f71a7fb50e583592"},"cell_type":"markdown","source":"#### So why is this happening and why did the model suggest the actions it did?\n\nTo understand that, we need to disect the suggestions made by the model at each episode and how this relates to our return.\n\nBelow, we have taken the optimal action for each state. We can see that the suggested actions do vary greatly between episodes and the model appears to decide which is wants to suggest very quickly.\n\nTherefore, I have plotted the total cost of the suggested actions at each episode and we can see the actions vary initially then smooth out and the resulting total cost is below our budget. This helps us understand what is going on greatly.\n\nSo far, all we have told the model is to provide a selection that is below budget and it has. It has simply found a answer that is below the budget as required.\n\nSo what is the next step? Before I introduce rewards I want to demonstrate what happens if I vary some of the parameters and what we can do if we decide to change what we want our model to suggest."},{"metadata":{"trusted":false,"_uuid":"3140ed63710d9e9730b7b0282635d3bf7e9884a9"},"cell_type":"code","source":"Ingredients = list(set(data['Ingredient']))\nactions = pd.DataFrame()\n\nfor a in range(0, len(Ingredients)):   \n    individualactions = []\n    for i in range(0,num_episodes):    \n        individualactions = np.append(individualactions, Mdl[5][a+(i*(len(Ingredients)))])\n    actions[a] = individualactions\n    plt.plot(range(0,num_episodes), actions[a])\n    plt.title('Product for Ingredient: ' + str(a+1))\n    plt.xlabel('Episode')\n    plt.ylabel('Product')\n    plt.show()    \n\n    \n","execution_count":9,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"17910040daf6184d63349fdb561f452dd6d5f575"},"cell_type":"code","source":"actions2 = actions\nactions2['Product1'] = actions2.iloc[:,0]+10\nactions2['Product2'] = actions2.iloc[:,1]+20\nactions2['Product3'] = actions2.iloc[:,2]+30\nactions2['Product4'] = actions2.iloc[:,3]+40\n\nactions3 = actions2.merge(data[['QMerged_label','Real_Cost']],left_on = 'Product1',right_on = 'QMerged_label', how = 'left')\nactions4 = actions3.merge(data[['QMerged_label','Real_Cost']],left_on = 'Product2',right_on = 'QMerged_label', how = 'left')\nactions5 = actions4.merge(data[['QMerged_label','Real_Cost']],left_on = 'Product3',right_on = 'QMerged_label', how = 'left')\nactions6 = actions5.merge(data[['QMerged_label','Real_Cost']],left_on = 'Product4',right_on = 'QMerged_label', how = 'left')\n\n\nactions6['Total_Cost'] = actions6.iloc[:,9] + actions6.iloc[:,11] + actions6.iloc[:,13] + actions6.iloc[:,15]\nactions6 = actions6.iloc[:,[0,1,2,3,-1]]\n\nactions6 = actions6.iloc[:num_episodes]\n\nplt.plot(range(0,num_episodes), actions6['Total_Cost'])\nplt.plot([0, num_episodes], [budget, budget], 'k-', lw=2)\nplt.title('Total Real Cost of Best Products at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Total Real Cost (£)')\nplt.ylim([0,budget+10])\nplt.show()","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"bcf9e3930b1d8a07e7f67c726fb3d43f8de861be"},"cell_type":"markdown","source":"## Effect of Changing Parameters and How to Change Model's Aim\n\n"},{"metadata":{"_uuid":"dc72dd6a45d9007a14616491715efdb139040728"},"cell_type":"markdown","source":"We have a few parameters that can be changed:\n\n1. The Budget\n\n2. Our learning rate, $\\alpha$\n\n3. Out action selection parameter, $\\epsilon$"},{"metadata":{"_uuid":"f5691f5ef2e745a445ad02cf0a53fcd34285423a"},"cell_type":"markdown","source":"#### Varying Budget\n\nFirst, lets observe what happens if we make our budget either impossibly low or high.\n\nA budget that means we only obtain a negative reward means that we will force our V to converge negatively whereas a budget that is too high will cause our V to converge positively as all actions are continually positive.\n\nThe latter seems like what we had in our first run, a lot of the episodes lead to positive outcomes and so many combinations of products are possible and there is little distinction between the cheapest products from the rest.\n\nIf instead we consider a budget that is reasonably low given the prices of the products, we can see a trend where the cheapest products look to be converging positively and the more expensive products converging negatively. However, the smoothness of these is far from ideal, both appear to be oscillating greatly between each episode.\n\nSo what can we do the reduce the 'spikiness' of the outputs? This leades us onto our next parameter, alpha."},{"metadata":{"trusted":false,"_uuid":"7a2f4dd36aace84bdea24c07685fb94530d5c0c1"},"cell_type":"code","source":"##### Make budget very small\nbudget2 = 5\n\n\nalpha2 = 0.1\nnum_episodes2 = 100\nepsilon2 = 0.5\n\n\n# Currently not using a reward\nreward2 = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl2 = MCModelv1(data=data, alpha = alpha2, e = num_episodes2,epsilon = epsilon2, budget = budget2, reward = reward2)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nplt.plot(range(0,num_episodes2), Mdl2[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":11,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"41edca465c3b169aa1c4f3fafc18e8cbd1091b70"},"cell_type":"code","source":"# Make budget very large\nbudget3 = 100\n\n\nalpha3 = 0.1\nnum_episodes3 = 100\nepsilon3 = 0.5\n\n\n# Currently not using a reward\nreward3 = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl3 = MCModelv1(data=data, alpha = alpha3, e = num_episodes3,epsilon = epsilon3, budget = budget3, reward = reward3)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nplt.plot(range(0,num_episodes), Mdl3[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":12,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"d8876e14166674f5aaceddcddf1f9f2c973640e5"},"cell_type":"code","source":"# Make budget reasonably small\nbudget4 = 23\n\n\nalpha4 = 0.1\nnum_episodes4 = 100\nepsilon4 = 0.5\n\n\n# Currently not using a reward\nreward4 = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl4 = MCModelv1(data=data, alpha = alpha4, e = num_episodes4,epsilon = epsilon4, budget = budget4, reward = reward4)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nplt.plot(range(0,num_episodes4), Mdl4[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n\n\nplt.plot(range(0,num_episodes4), Mdl4[1],range(0,num_episodes4), Mdl4[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"a8cea2a13870b3e05a9b927e6ec66961b2d06137"},"cell_type":"markdown","source":"## Varying Alpha"},{"metadata":{"_uuid":"50e27078fda9b990300e344521ba2fb11bb87893"},"cell_type":"markdown","source":"#### A good explination of what is going on with our outupt due to alpha is described by stack overflow user VishalTheBeast:\n    \n\"Learning rate tells the magnitude of step that is taken towards the solution.\n\nIt should not be too big a number as it may continuously oscillate around the minima and it should not be too small of a number else it will take a lot of time and iterations to reach the minima.\n\nThe reason why decay is advised in learning rate is because initially when we are at a totally random point in solution space we need to take big leaps towards the solution and later when we come close to it, we make small jumps and hence small improvements to finally reach the minima.\n\nAnalogy can be made as: in the game of golf when the ball is far away from the hole, the player hits it very hard to get as close as possible to the hole. Later when he reaches the flagged area, he choses a different stick to get accurate short shot.\n\nSo its not that he won't be able to put the ball in the hole without choosing the short shot stick, he may send the ball ahead of the target two or three times. But it would be best if he plays optimally and uses the right amount of power to reach the hole. Same is for decayed learning rate.\" \n    \n    \nhttps://stackoverflow.com/questions/33011825/learning-rate-of-a-q-learning-agent"},{"metadata":{"_uuid":"b4aee633829809e9139085a2d66f8f6bca335b15"},"cell_type":"markdown","source":"To better demonstrate the effect of varying our alpha, I will be using an animated plot created using Plot.ly.\n\nI have witten a more detailed guide on how to do this here:\n\nhttps://www.philiposbornedata.com/2018/03/01/creating-interactive-animation-for-parameter-optimisation-using-plot-ly/\n"},{"metadata":{"_uuid":"dcf0ecc4852b5307a2afa77f9513b4f9748a171b"},"cell_type":"markdown","source":"---\n\nIn our first animation, we vary alpha between 1 and 0.1. This enables us to see that as we reduce alpha our output smooths somewhat but it still pretty rough.\n\nTo investigate this further, I have then created a similar plot for alpha between 0.1 and 0.01. This emphasises the smoothing effect alpha has even more so.\n\nHowever, even though the results are smoothing out, they are no longer converging in 100 episodes and, furthermore, they output seems to alternate between each alpha. This is due to a combination of small alphas requireing more episodes to learn and out action selection paramter epsilon being 0.5. Essentially, the output is still being decided by randomness half of the time and so out results are not converging within the 100 episode frame. "},{"metadata":{"trusted":false,"_uuid":"5559fa135a0563ef9c988702c397b0a4f800a365"},"cell_type":"code","source":"from plotly.offline import init_notebook_mode, iplot, plot\nfrom IPython.display import display, HTML\nimport plotly\nimport plotly.plotly as py\ninit_notebook_mode(connected=True)","execution_count":30,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"7a3bc9503822bd6322be95c2f7fb80c17f0f4b84"},"cell_type":"code","source":"# Provide all parameters fixed except alpha\nbudget5 = 23\nnum_episodes5 = 100\nepsilon5 = 0.5\n\n# Currently not using a reward\nreward5 = [0,0,0,0,0,0,0,0,0]\n\nVforInteractiveGraphA = []    \nlA = []\nnum_episodes5_2 = []\nfor x in range(0, 10):\n    alpha5 = 1 - x/10\n    Mdl5 = MCModelv1(data=data, alpha = alpha5, e = num_episodes5,epsilon = epsilon5, budget = budget5, reward = reward5)\n    VforInteractiveGraphA = np.append(VforInteractiveGraphA, Mdl5[0])\n    for y in range(0, num_episodes5):\n        lA = np.append(lA,alpha5)\n        num_episodes5_2 = np.append(num_episodes5_2, y)\nVforInteractiveGraphA2 = pd.DataFrame(VforInteractiveGraphA,lA)\nVforInteractiveGraphA2['index1'] = VforInteractiveGraphA2.index\nVforInteractiveGraphA2['Episode'] = num_episodes5_2\nVforInteractiveGraphA2.columns = ['V', 'Alpha', 'Episode']\nVforInteractiveGraphA2 = VforInteractiveGraphA2[['Alpha','Episode', 'V']]\n\nVforInteractiveGraphA2.head()\n","execution_count":23,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e738e8a72a7c5dbb4287aa301ade51bb19c087d5"},"cell_type":"code","source":"VforInteractiveGraphA3 = VforInteractiveGraphA2\nVforInteractiveGraphA3['continent'] = 'Test'\nVforInteractiveGraphA3['country'] = 'Test2'\nVforInteractiveGraphA3['pop'] = 7000000.0\nVforInteractiveGraphA3.columns = ['year', 'lifeExp', 'gdpPercap', 'continent', 'country', 'pop']\n\nalphaforGraph = list(set(VforInteractiveGraphA3['year']))\nalphaforGraph = np.round(alphaforGraph,1)\nalphaforGraph = np.sort(alphaforGraph)[::-1]\nyears = np.round([(alphaforGraph) for alphaforGraph in alphaforGraph],1)\nyears","execution_count":25,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"e6bc2a910af8b9fe2b00a5b628929463742c4eb9"},"cell_type":"code","source":"%matplotlib inline\n\ndataset = VforInteractiveGraphA3\n\ncontinents = []\nfor continent in dataset['continent']:\n    if continent not in continents:\n        continents.append(continent)\n# make figure\nfigure = {\n    'data': [],\n    'layout': {},\n    'frames': []\n}\n\n# fill in most of layout\nfigure['layout']['title'] = \"Parameter Optimisation using Interactive Animation <br> PhilipOsborneData.com\"\nfigure['layout']['xaxis'] = {'title': 'Episode'}\nfigure['layout']['yaxis'] = {'title': 'Sum of V', 'type': 'linear'}\nfigure['layout']['hovermode'] = 'closest'\nfigure['layout']['sliders'] = {\n    'args': [\n        'transition', {\n            'duration': 400,\n            'easing': 'cubic-in-out'\n        }\n    ],\n    'initialValue': '1952',\n    'plotlycommand': 'animate',\n    'values': years,\n    'visible': True\n}\nfigure['layout']['updatemenus'] = [\n    {\n        'buttons': [\n            {\n                'args': [None, {'frame': {'duration': 500, 'redraw': False},\n                         'fromcurrent': True, 'transition': {'duration': 300, 'easing': 'quadratic-in-out'}}],\n                'label': 'Play',\n                'method': 'animate'\n            },\n            {\n                'args': [[None], {'frame': {'duration': 0, 'redraw': False}, 'mode': 'immediate',\n                'transition': {'duration': 0}}],\n                'label': 'Pause',\n                'method': 'animate'\n            }\n        ],\n        'direction': 'left',\n        'pad': {'r': 10, 't': 87},\n        'showactive': False,\n        'type': 'buttons',\n        'x': 0.1,\n        'xanchor': 'right',\n        'y': 0,\n        'yanchor': 'top'\n    }\n]\n\nsliders_dict = {\n    'active': 0,\n    'yanchor': 'top',\n    'xanchor': 'left',\n    'currentvalue': {\n        'font': {'size': 20},\n        'prefix': 'Alpha: ',\n        'visible': True,\n        'xanchor': 'right'\n    },\n    'transition': {'duration': 300, 'easing': 'cubic-in-out'},\n    'pad': {'b': 10, 't': 50},\n    'len': 0.9,\n    'x': 0.1,\n    'y': 0,\n    'steps': []\n}\n\n# make data\nyear = 1.0\nfor continent in continents:\n    dataset_by_year = dataset[np.round(dataset['year'],1) == np.round(year,1)]\n    dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n    data_dict = {\n        'x': list(dataset_by_year_and_cont['lifeExp']),\n        'y': list(dataset_by_year_and_cont['gdpPercap']),\n        'mode': 'markers',\n        'text': list(dataset_by_year_and_cont['country']),\n        'marker': {\n            'sizemode': 'area',\n            'sizeref': 200000,\n            'size': list(dataset_by_year_and_cont['pop'])\n        },\n        'name': continent\n    }\n    figure['data'].append(data_dict)\n\n# make frames\nfor year in years:\n    frame = {'data': [], 'name': str(year)}\n    for continent in continents:\n        dataset_by_year = dataset[np.round(dataset['year'],1) == np.round(year,1)]\n        dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n        data_dict = {\n            'x': list(dataset_by_year_and_cont['lifeExp']),\n            'y': list(dataset_by_year_and_cont['gdpPercap']),\n            'mode': 'markers',\n            'text': list(dataset_by_year_and_cont['country']),\n            'marker': {\n                'sizemode': 'area',\n                'sizeref': 200000,\n                'size': list(dataset_by_year_and_cont['pop'])\n            },\n            'name': continent\n        }\n        frame['data'].append(data_dict)\n\n    figure['frames'].append(frame)\n    slider_step = {'args': [\n        [year],\n        {'frame': {'duration': 300, 'redraw': False},\n         'mode': 'immediate',\n       'transition': {'duration': 300}}\n     ],\n     'label': year,\n     'method': 'animate'}\n    sliders_dict['steps'].append(slider_step)\n\n\nfigure['layout']['sliders'] = [sliders_dict]\n\niplot(figure)","execution_count":31,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"61c5aaabce291d42db50782e531753b373043abb"},"cell_type":"code","source":"# Provide all parameters fixed except alpha\nbudget6 = 23\nnum_episodes6 = 100\nepsilon6 = 0.5\n\n# Currently not using a reward\nreward6 = [0,0,0,0,0,0,0,0,0]\n\nVforInteractiveGraphA_2 = []    \nlA2 = []\nnum_episodes6_2 = []\nfor x in range(0, 10):\n    alpha6 = 0.1 - x/100\n    Mdl6 = MCModelv1(data=data, alpha = alpha6, e = num_episodes6,epsilon = epsilon6, budget = budget6, reward = reward6)\n    VforInteractiveGraphA_2 = np.append(VforInteractiveGraphA_2, Mdl6[0])\n    for y in range(0, num_episodes6):\n        lA2 = np.append(lA2,alpha6)\n        num_episodes6_2 = np.append(num_episodes6_2, y)\nVforInteractiveGraphA_22 = pd.DataFrame(VforInteractiveGraphA_2,lA2)\nVforInteractiveGraphA_22['index1'] = VforInteractiveGraphA_22.index\nVforInteractiveGraphA_22['Episode'] = num_episodes6_2\nVforInteractiveGraphA_22.columns = ['V', 'Alpha', 'Episode']\nVforInteractiveGraphA_22 = VforInteractiveGraphA_22[['Alpha','Episode', 'V']]\n\n\nVforInteractiveGraphA_23 = VforInteractiveGraphA_22\nVforInteractiveGraphA_23['continent'] = 'Test'\nVforInteractiveGraphA_23['country'] = 'Test2'\nVforInteractiveGraphA_23['pop'] = 7000000.0\nVforInteractiveGraphA_23.columns = ['year', 'lifeExp', 'gdpPercap', 'continent', 'country', 'pop']\n\nalphaforGraph2 = list(set(VforInteractiveGraphA_23['year']))\nalphaforGraph2 = np.round(alphaforGraph2,2)\nalphaforGraph2 = np.sort(alphaforGraph2)[::-1]\nyears = np.round([(alphaforGraph2) for alphaforGraph2 in alphaforGraph2],2)\nyears\n\n%matplotlib inline\n\ndataset = VforInteractiveGraphA_23\n\ncontinents = []\nfor continent in dataset['continent']:\n    if continent not in continents:\n        continents.append(continent)\n# make figure\nfigure = {\n    'data': [],\n    'layout': {},\n    'frames': []\n}\n\n# fill in most of layout\nfigure['layout']['title'] = \"Parameter Optimisation using Interactive Animation <br> PhilipOsborneData.com\"\nfigure['layout']['xaxis'] = {'title': 'Episode'}\nfigure['layout']['yaxis'] = {'title': 'Sum of V', 'type': 'linear'}\nfigure['layout']['hovermode'] = 'closest'\nfigure['layout']['sliders'] = {\n    'args': [\n        'transition', {\n            'duration': 400,\n            'easing': 'cubic-in-out'\n        }\n    ],\n    'initialValue': '1952',\n    'plotlycommand': 'animate',\n    'values': years,\n    'visible': True\n}\nfigure['layout']['updatemenus'] = [\n    {\n        'buttons': [\n            {\n                'args': [None, {'frame': {'duration': 500, 'redraw': False},\n                         'fromcurrent': True, 'transition': {'duration': 300, 'easing': 'quadratic-in-out'}}],\n                'label': 'Play',\n                'method': 'animate'\n            },\n            {\n                'args': [[None], {'frame': {'duration': 0, 'redraw': False}, 'mode': 'immediate',\n                'transition': {'duration': 0}}],\n                'label': 'Pause',\n                'method': 'animate'\n            }\n        ],\n        'direction': 'left',\n        'pad': {'r': 10, 't': 87},\n        'showactive': False,\n        'type': 'buttons',\n        'x': 0.1,\n        'xanchor': 'right',\n        'y': 0,\n        'yanchor': 'top'\n    }\n]\n\nsliders_dict = {\n    'active': 0,\n    'yanchor': 'top',\n    'xanchor': 'left',\n    'currentvalue': {\n        'font': {'size': 20},\n        'prefix': 'Alpha: ',\n        'visible': True,\n        'xanchor': 'right'\n    },\n    'transition': {'duration': 300, 'easing': 'cubic-in-out'},\n    'pad': {'b': 10, 't': 50},\n    'len': 0.9,\n    'x': 0.1,\n    'y': 0,\n    'steps': []\n}\n\n# make data\nyear = 1.0\nfor continent in continents:\n    dataset_by_year = dataset[np.round(dataset['year'],2) == np.round(year,2)]\n    dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n    data_dict = {\n        'x': list(dataset_by_year_and_cont['lifeExp']),\n        'y': list(dataset_by_year_and_cont['gdpPercap']),\n        'mode': 'markers',\n        'text': list(dataset_by_year_and_cont['country']),\n        'marker': {\n            'sizemode': 'area',\n            'sizeref': 200000,\n            'size': list(dataset_by_year_and_cont['pop'])\n        },\n        'name': continent\n    }\n    figure['data'].append(data_dict)\n\n# make frames\nfor year in years:\n    frame = {'data': [], 'name': str(year)}\n    for continent in continents:\n        dataset_by_year = dataset[np.round(dataset['year'],2) == np.round(year,2)]\n        dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n        data_dict = {\n            'x': list(dataset_by_year_and_cont['lifeExp']),\n            'y': list(dataset_by_year_and_cont['gdpPercap']),\n            'mode': 'markers',\n            'text': list(dataset_by_year_and_cont['country']),\n            'marker': {\n                'sizemode': 'area',\n                'sizeref': 200000,\n                'size': list(dataset_by_year_and_cont['pop'])\n            },\n            'name': continent\n        }\n        frame['data'].append(data_dict)\n\n    figure['frames'].append(frame)\n    slider_step = {'args': [\n        [year],\n        {'frame': {'duration': 300, 'redraw': False},\n         'mode': 'immediate',\n       'transition': {'duration': 300}}\n     ],\n     'label': year,\n     'method': 'animate'}\n    sliders_dict['steps'].append(slider_step)\n\n\nfigure['layout']['sliders'] = [sliders_dict]\n\niplot(figure)\n","execution_count":50,"outputs":[]},{"metadata":{"_uuid":"3f0915e417cad1dc8e14fafb7498f338990f71b5"},"cell_type":"markdown","source":"## Varying Epsilon"},{"metadata":{"_uuid":"bf3e04b106c2be44ddb929d5edf8682abe5b2166"},"cell_type":"markdown","source":"With the previous results in mind, we now fix alpha to be 0.05 and vary epsilone between 1 and 0 to show the effect of completely randomly selecting actions to selecting actions greedily.\n\nWe see that having a high epsilon creates very sporadic results. Therefore we should select something resonably small like 0.2. Although have epsilon equal to 0 looks good because of how smooth the curve is, as we mentioned earlier, this may lead us to a choice very quickly but may not be the best. We want some randomness so the model can explore other actions if needed."},{"metadata":{"scrolled":false,"trusted":false,"_uuid":"e5302372421bf7fb23cf8844e868338c7773e582"},"cell_type":"code","source":"# Provide all parameters fixed except alpha\nbudget7 = 23\nnum_episodes7 = 100\nalpha7  = 0.05\n\n# Currently not using a reward\nreward7 = [0,0,0,0,0,0,0,0,0]\n\nVforInteractiveGraphA_3 = []    \nlA3 = []\nnum_episodes7_3 = []\nfor x in range(0, 11):\n    epsilon7 = 1 - x/10\n    Mdl7 = MCModelv1(data=data, alpha = alpha7, e = num_episodes7,epsilon = epsilon7, budget = budget7, reward = reward7)\n    VforInteractiveGraphA_3 = np.append(VforInteractiveGraphA_3, Mdl7[0])\n    for y in range(0, num_episodes7):\n        lA3 = np.append(lA3,epsilon7)\n        num_episodes7_3 = np.append(num_episodes7_3, y)\nVforInteractiveGraphA_32 = pd.DataFrame(VforInteractiveGraphA_3,lA3)\nVforInteractiveGraphA_32['index1'] = VforInteractiveGraphA_32.index\nVforInteractiveGraphA_32['Episode'] = num_episodes7_3\nVforInteractiveGraphA_32.columns = ['V', 'Epsilon', 'Episode']\nVforInteractiveGraphA_32 = VforInteractiveGraphA_32[['Epsilon','Episode', 'V']]\n\n\nVforInteractiveGraphA_33 = VforInteractiveGraphA_32\nVforInteractiveGraphA_33['continent'] = 'Test'\nVforInteractiveGraphA_33['country'] = 'Test2'\nVforInteractiveGraphA_33['pop'] = 7000000.0\nVforInteractiveGraphA_33.columns = ['year', 'lifeExp', 'gdpPercap', 'continent', 'country', 'pop']\n\nepsilonforGraph3 = list(set(VforInteractiveGraphA_33['year']))\nepsilonforGraph3 = np.round(epsilonforGraph3,1)\nepsilonforGraph3 = np.sort(epsilonforGraph3)[::-1]\nyears = np.round([(epsilonforGraph3) for epsilonforGraph3 in epsilonforGraph3],1)\nyears\n\n%matplotlib inline\n\ndataset = VforInteractiveGraphA_33\n\ncontinents = []\nfor continent in dataset['continent']:\n    if continent not in continents:\n        continents.append(continent)\n# make figure\nfigure = {\n    'data': [],\n    'layout': {},\n    'frames': []\n}\n\n# fill in most of layout\nfigure['layout']['title'] = \"Parameter Optimisation using Interactive Animation <br> PhilipOsborneData.com\"\nfigure['layout']['xaxis'] = {'title': 'Episode'}\nfigure['layout']['yaxis'] = {'title': 'Sum of V', 'type': 'linear'}\nfigure['layout']['hovermode'] = 'closest'\nfigure['layout']['sliders'] = {\n    'args': [\n        'transition', {\n            'duration': 400,\n            'easing': 'cubic-in-out'\n        }\n    ],\n    'initialValue': '1952',\n    'plotlycommand': 'animate',\n    'values': years,\n    'visible': True\n}\nfigure['layout']['updatemenus'] = [\n    {\n        'buttons': [\n            {\n                'args': [None, {'frame': {'duration': 500, 'redraw': False},\n                         'fromcurrent': True, 'transition': {'duration': 300, 'easing': 'quadratic-in-out'}}],\n                'label': 'Play',\n                'method': 'animate'\n            },\n            {\n                'args': [[None], {'frame': {'duration': 0, 'redraw': False}, 'mode': 'immediate',\n                'transition': {'duration': 0}}],\n                'label': 'Pause',\n                'method': 'animate'\n            }\n        ],\n        'direction': 'left',\n        'pad': {'r': 10, 't': 87},\n        'showactive': False,\n        'type': 'buttons',\n        'x': 0.1,\n        'xanchor': 'right',\n        'y': 0,\n        'yanchor': 'top'\n    }\n]\n\nsliders_dict = {\n    'active': 0,\n    'yanchor': 'top',\n    'xanchor': 'left',\n    'currentvalue': {\n        'font': {'size': 20},\n        'prefix': 'Epsilon: ',\n        'visible': True,\n        'xanchor': 'right'\n    },\n    'transition': {'duration': 300, 'easing': 'cubic-in-out'},\n    'pad': {'b': 10, 't': 50},\n    'len': 0.9,\n    'x': 0.1,\n    'y': 0,\n    'steps': []\n}\n\n# make data\nyear = 1.0\nfor continent in continents:\n    dataset_by_year = dataset[np.round(dataset['year'],1) == np.round(year,1)]\n    dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n    data_dict = {\n        'x': list(dataset_by_year_and_cont['lifeExp']),\n        'y': list(dataset_by_year_and_cont['gdpPercap']),\n        'mode': 'markers',\n        'text': list(dataset_by_year_and_cont['country']),\n        'marker': {\n            'sizemode': 'area',\n            'sizeref': 200000,\n            'size': list(dataset_by_year_and_cont['pop'])\n        },\n        'name': continent\n    }\n    figure['data'].append(data_dict)\n\n# make frames\nfor year in years:\n    frame = {'data': [], 'name': str(year)}\n    for continent in continents:\n        dataset_by_year = dataset[np.round(dataset['year'],1) == np.round(year,1)]\n        dataset_by_year_and_cont = dataset_by_year[dataset_by_year['continent'] == continent]\n\n        data_dict = {\n            'x': list(dataset_by_year_and_cont['lifeExp']),\n            'y': list(dataset_by_year_and_cont['gdpPercap']),\n            'mode': 'markers',\n            'text': list(dataset_by_year_and_cont['country']),\n            'marker': {\n                'sizemode': 'area',\n                'sizeref': 200000,\n                'size': list(dataset_by_year_and_cont['pop']),\n                'color': 'rgba(255, 182, 193, .9)'\n            },\n            'name': continent\n        }\n        frame['data'].append(data_dict)\n\n    figure['frames'].append(frame)\n    slider_step = {'args': [\n        [year],\n        {'frame': {'duration': 300, 'redraw': False},\n         'mode': 'immediate',\n       'transition': {'duration': 300}}\n     ],\n     'label': year,\n     'method': 'animate'}\n    sliders_dict['steps'].append(slider_step)\n\n\nfigure['layout']['sliders'] = [sliders_dict]\n\niplot(figure)\n","execution_count":86,"outputs":[]},{"metadata":{"_uuid":"3f600031feecf7c9c976870c375dcb9cfa55f447"},"cell_type":"markdown","source":"## Increasing the Number of Episodes"},{"metadata":{"_uuid":"fe50592bb354de519639d0870570d39af953b84a"},"cell_type":"markdown","source":"Lastly, we can increase the number of episodes. I refrained from doing this sooner because we were running 10 models in a loop to output our animated graphs and this would have caused the time taken to run the model to explode. \n\nWe noted that a low alpha would require more episodes to learn so we can run our model for 1000 episodes.\n\nHowever, we still notice that the output is oscillating, but, as mentioned before, this is due to our aim being simply to recommend a combination that is below budget. What this shows is that the model can't find the single best combination when there are many that fit below our budget.\n\nTherefore, what happens if we change our aim slightly so that we can use the model to find the cheapest combination of products?"},{"metadata":{"trusted":false,"_uuid":"b79a593156743b15f1cd4a27434e5eff1b822d9b"},"cell_type":"code","source":"# Increase the number of episodes\n\nbudget8 = 23\nalpha8 = 0.05\nnum_episodes8 = 1000\nepsilon8 = 0.2\n\n# Currently not using a reward\nreward8 = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl8 = MCModelv1(data=data, alpha = alpha8, e = num_episodes8,epsilon = epsilon8, budget = budget8, reward = reward8)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\n\nplt.plot(range(0,num_episodes8), Mdl8[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n\n\nplt.plot(range(0,num_episodes8), Mdl8[1],range(0,num_episodes8), Mdl8[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":84,"outputs":[]},{"metadata":{"collapsed":true,"_uuid":"8af8badd6c103b4f2a83e88ac4b102220f4b0306"},"cell_type":"markdown","source":"## Changing our Model's Aim to Find the Cheapest Combination of Products"},{"metadata":{"_uuid":"914c0fb8dd790d58228747f8051b9cdffc1b9355"},"cell_type":"markdown","source":"#### This will now more clearly seperate the cheapest products from the rest and nearly always provides us with the cheapest combination of products."},{"metadata":{"collapsed":true,"_uuid":"a3a66de37cba96621cdae1c522cb631f8f64860f"},"cell_type":"markdown","source":"To do this, all we need do is adapt our model slightly to provide a terminal reward that is relative to how far below or above budget this combination in the episode is. \n\nThis can done by changing the calculation for return to:\n\n\n\n        if(budget >= episode2['Real_Cost'].sum()):\n            Return = (budget - episode2['Real_Cost'].sum())\n        else:\n            Return = (budget - episode2['Real_Cost'].sum())\n            \nWe now see that the seperation between the cheapest products and the others is emphasised.\n\nThis really demonstrates the flexibility of reinforcement learning and how easy it can be to adapt the model based on your aims."},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"85871506245698835e92918b64318f72f41c2340"},"cell_type":"code","source":"def MCModelv2(data, alpha, e, epsilon, budget, reward):\n    # Define the States\n    Ingredients = list(set(data['Ingredient']))\n    # Initialise V_0\n    V0 = data['V_0']\n    data['V'] = V0\n    output = []\n    output1 = []\n    output2 = []\n    actioninfull = []\n    #Interate over the number of episodes specified\n    for e in range(0,e):\n        \n        episode_run = []\n        #Introduce epsilon-greedy selection, we randomly select the first episode as V_0(a) = 0 for all actions\n        epsilon = epsilon\n        if e == 0:\n            for i in range(0,len(Ingredients)):\n                episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n            episode_run = episode_run.astype(int)\n        \n        else:\n            for i in range(0,len(Ingredients)):\n                greedyselection = np.random.random_integers(low = 1, high =10)\n                if greedyselection <= (epsilon)*10:\n                    episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n                else:\n                    data_I = data[data['Ingredient'] == (i+1)] \n                    MaxofVforI = data_I[data_I['V'] == data_I['V'].max() ]['Product']\n                    #If multiple max values, take first\n                    MaxofVforI = MaxofVforI.values[0]\n                    episode_run = np.append(episode_run, MaxofVforI)\n                    \n                episode_run = episode_run.astype(int)\n                \n               \n           \n        episode = pd.DataFrame({'Ingredient' : Ingredients, 'Product': episode_run})    \n        episode['Merged_label'] =  (episode['Ingredient']*10 + episode['Product']).astype(float)\n        data['QMerged_label'] = (data['QMerged_label']).astype(float)\n        data['Reward'] = reward\n        episode2 =  episode.merge(data[['QMerged_label','Real_Cost','Reward']], left_on='Merged_label',right_on='QMerged_label', how = 'inner')\n        data = data.drop('Reward',1)\n        \n        # Calculate our terminal reward\n        if(budget >= episode2['Real_Cost'].sum()):\n            Return = (budget - episode2['Real_Cost'].sum())  \n        else:\n            Return = (budget - episode2['Real_Cost'].sum())\n        episode2 = episode2.drop('Reward',1)\n        episode2['Return'] = Return\n        \n        # Apply update rule to actions that were involved in obtaining terminal reward \n        data = data.merge(episode2[['Merged_label','Return']], left_on='QMerged_label',right_on='Merged_label', how = 'outer')\n        data['Return'] = data['Return'].fillna(0)\n        for v in range(0,len(data)):\n            if data.iloc[v,7] == 0:\n                data.iloc[v,5] = data.iloc[v,5] \n            else:\n                data.iloc[v,5]  = data.iloc[v,5]  + alpha*( (data.iloc[v,7]/len(Ingredients)) - data.iloc[v,5] )\n                \n        # Output table    \n        data = data.drop('Merged_label',1)\n        data = data.drop('Return',1)\n        \n        # Output is the Sum of V(a) for all episodes\n        output  = np.append(output, data.iloc[:,-1].sum())\n        \n        # Output 1 and 2 are the Sum of V(a) for for the cheapest actions and rest respectively\n        # I did this so we can copare how they converge whilst applying to such a small sample problem\n        output1 = np.append(output1, data.iloc[[1,2,4,8],-1].sum())\n        output2 = np.append(output2, data.iloc[[0,3,5,6,7],-1].sum())\n        \n        # Ouput to optimal action from the model based on highest V(a)\n        action = pd.DataFrame(data.groupby('Ingredient')['V'].max())\n        action2 = action.merge(data, left_on = 'V',right_on = 'V', how = 'inner')\n        action3 = action2[['Ingredient','Product']]\n        action3 = action3.groupby('Ingredient')['Product'].apply(lambda x :x.iloc[np.random.randint(0, len(x))])\n        \n        # Output the optimal action at each episode so we can see how this changes over time\n        actioninfull = np.append(actioninfull, action3)\n        actioninfull = actioninfull.astype(int)\n        \n        # Rename for clarity\n        SumofV = output\n        SumofVForCheapest = output1\n        SumofVForExpensive = output2\n        OptimalActions = action3\n        ActionsSelectedinTime = actioninfull\n        \n    return(SumofV, SumofVForCheapest, SumofVForExpensive, OptimalActions, data, ActionsSelectedinTime)\n\n\n","execution_count":80,"outputs":[]},{"metadata":{"trusted":false,"_uuid":"cbdd0a1f3266aa0065504c7e46609298c2ee1e78"},"cell_type":"code","source":"##### Make budget reasonably small\nbudget9 = 23\n\n\nalpha9 = 0.05\nnum_episodes9 = 1000\nepsilon9 = 0.2\n\n\n# Currently not using a reward\nreward9 = [0,0,0,0,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl9 = MCModelv2(data=data, alpha = alpha9, e = num_episodes9,epsilon = epsilon9, budget = budget9, reward = reward9)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nprint(Mdl9[3])\n\n\nplt.plot(range(0,num_episodes9), Mdl9[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n\n\nplt.plot(range(0,num_episodes9), Mdl9[1],range(0,num_episodes9), Mdl9[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":85,"outputs":[]},{"metadata":{"_uuid":"75fc1ff86949bac4c039157db6182aa91ccf974b"},"cell_type":"markdown","source":"# Introducing Preferences"},{"metadata":{"_uuid":"9d310dfb4a4bd12c40c22c7e66d45a7d9063fdf7"},"cell_type":"markdown","source":"So far, we have not included any personal preferences towards products. If we wanted to include this, we can simply introduce rewards for each product whilst still having a terminal reward that encourages the model to be below budget. \n\n\nThis can done by changing the calculation for return to:\n\n\n\n        if(budget >= episode2['Real_Cost'].sum()):\n            Return = 1 + (episode2['Reward'].sum())/len(Ingredients)\n        else:\n            Return = -1 + (episode2['Reward'].sum())/len(Ingredients)\n            \nSo why is our return calculation now like this? \n\nWell firstly, we still want our combination to be below budget so we provide the positive and negative rewards for being above and below budget respectively.\n\nNext, we want to account for the reward of each product. For our purposes, we define the rewards to be a value between 0 and 1. MC return is formally calculated using the following:\n\n\n$$R =  \\sum_{k=0}^{Num Actions} \\gamma^{k-1} r_{k}  $$\n\n\n$\\gamma$ is the discount factor and this tells us how much we value later steps compared to earlier steps. In our case, all actions are equally as important to reaching the desired outcome of being below budget so we set $\\gamma = 1$. \n\nHowever, to ensure that we reach the primary goal of being below budget, we take the average of the sum of the rewards for each action so that this will always be less than 1 or -1 respectively.\n\n\n"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"348726e8f6a698ed6bc24854b21b771760af8513"},"cell_type":"code","source":"def MCModelv3(data, alpha, e, epsilon, budget, reward):\n    # Define the States\n    Ingredients = list(set(data['Ingredient']))\n    # Initialise V_0\n    V0 = data['V_0']\n    data['V'] = V0\n    output = []\n    output1 = []\n    output2 = []\n    actioninfull = []\n    #Interate over the number of episodes specified\n    for e in range(0,e):\n        \n        episode_run = []\n        #Introduce epsilon-greedy selection, we randomly select the first episode as V_0(a) = 0 for all actions\n        epsilon = epsilon\n        if e == 0:\n            for i in range(0,len(Ingredients)):\n                episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n            episode_run = episode_run.astype(int)\n        \n        else:\n            for i in range(0,len(Ingredients)):\n                greedyselection = np.random.random_integers(low = 1, high =10)\n                if greedyselection <= (epsilon)*10:\n                    episode_run = np.append(episode_run,np.random.random_integers(low = 1, high = sum(1 for p in data.iloc[:, 0] if p == i+1 ), size = None))\n                else:\n                    data_I = data[data['Ingredient'] == (i+1)] \n                    MaxofVforI = data_I[data_I['V'] == data_I['V'].max() ]['Product']\n                    #If multiple max values, take first\n                    MaxofVforI = MaxofVforI.values[0]\n                    episode_run = np.append(episode_run, MaxofVforI)\n                    \n                episode_run = episode_run.astype(int)\n                \n               \n           \n        episode = pd.DataFrame({'Ingredient' : Ingredients, 'Product': episode_run})    \n        episode['Merged_label'] =  (episode['Ingredient']*10 + episode['Product']).astype(float)\n        data['QMerged_label'] = (data['QMerged_label']).astype(float)\n        data['Reward'] = reward\n        episode2 =  episode.merge(data[['QMerged_label','Real_Cost','Reward']], left_on='Merged_label',right_on='QMerged_label', how = 'inner')\n        data = data.drop('Reward',1)\n        \n        # Calculate our terminal reward\n        if(budget >= episode2['Real_Cost'].sum()):\n            Return = 1 + (episode2['Reward'].sum())/(len(Ingredients))\n        else:\n            Return = -1 + (episode2['Reward'].sum())/(len(Ingredients))\n        episode2 = episode2.drop('Reward',1)\n        episode2['Return'] = Return\n        \n        # Apply update rule to actions that were involved in obtaining terminal reward \n        data = data.merge(episode2[['Merged_label','Return']], left_on='QMerged_label',right_on='Merged_label', how = 'outer')\n        data['Return'] = data['Return'].fillna(0)\n        for v in range(0,len(data)):\n            if data.iloc[v,7] == 0:\n                data.iloc[v,5] = data.iloc[v,5] \n            else:\n                data.iloc[v,5]  = data.iloc[v,5]  + alpha*( (data.iloc[v,7]/len(Ingredients)) - data.iloc[v,5] )\n                \n        # Output table    \n        data = data.drop('Merged_label',1)\n        data = data.drop('Return',1)\n        \n        # Output is the Sum of V(a) for all episodes\n        output  = np.append(output, data.iloc[:,-1].sum())\n        \n        # Output 1 and 2 are the Sum of V(a) for for the cheapest actions and rest respectively\n        # I did this so we can copare how they converge whilst applying to such a small sample problem\n        output1 = np.append(output1, data.iloc[[1,2,4,8],-1].sum())\n        output2 = np.append(output2, data.iloc[[0,3,5,6,7],-1].sum())\n        \n        # Ouput to optimal action from the model based on highest V(a)\n        action = pd.DataFrame(data.groupby('Ingredient')['V'].max())\n        action2 = action.merge(data, left_on = 'V',right_on = 'V', how = 'inner')\n        action3 = action2[['Ingredient','Product']]\n        action3 = action3.groupby('Ingredient')['Product'].apply(lambda x :x.iloc[np.random.randint(0, len(x))])\n        \n        # Output the optimal action at each episode so we can see how this changes over time\n        actioninfull = np.append(actioninfull, action3)\n        actioninfull = actioninfull.astype(int)\n        \n        # Rename for clarity\n        SumofV = output\n        SumofVForCheapest = output1\n        SumofVForExpensive = output2\n        OptimalActions = action3\n        ActionsSelectedinTime = actioninfull\n        \n    return(SumofV, SumofVForCheapest, SumofVForExpensive, OptimalActions, data, ActionsSelectedinTime)\n\n\n","execution_count":109,"outputs":[]},{"metadata":{"_uuid":"412f89ca9de4f2dff99b890a0999c4cd75bd54fc"},"cell_type":"markdown","source":"## Introducing Preferences using Rewards"},{"metadata":{"_uuid":"7f1c816ec1938b6e870954bf46ad69568aa17f3e"},"cell_type":"markdown","source":"Say we decided we wanted product a1 and b2, we could add a reward to each. Let us see what happens if we do this in the output and graphs below. We have changed out budget slightly as a1 and b2 add up to £21 which means there is no way to select two more products that would put it below a budget of £23.\n\nApplying a very high reward forces the modle to pick a1 and b2 then work around to find products that will put it under our budget.\n\nI have kept in the comparison between the cheapest products and the rest to show that the model now is not valuing the cheapest once more. Instead we get the output a1, b2, c1 and d3 which has a total cost of £25. This is both below our budget and includes our preferred products.\n"},{"metadata":{"trusted":false,"_uuid":"abe0a2a481ba9151b3fbf18af2fff46e67ae69e8"},"cell_type":"code","source":"# Introduce simple rewards\nbudget10 = 30\n\n\nalpha10 = 0.05\nnum_episodes10 = 1000\nepsilon10 = 0.2\n\n\n# Currently not using a reward\nreward10 = [0.8,0,0,0.8,0,0,0,0,0]\n\nstart_time = time.time()\n\nMdl10 = MCModelv3(data=data, alpha = alpha10, e = num_episodes10,epsilon = epsilon10, budget = budget10, reward = reward10)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nprint(Mdl10[3])\n\n\nplt.plot(range(0,num_episodes10), Mdl10[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n\n\nplt.plot(range(0,num_episodes10), Mdl10[1],range(0,num_episodes10), Mdl10[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":111,"outputs":[]},{"metadata":{"_uuid":"a2cc7e023d5e10d38e7964daa9147a3afd866dbe"},"cell_type":"markdown","source":"Let's try one more reward signal. This time, I give some reward to each but want it to provide the best combination from my rewards that still keeps us below budget.\n\nWe have the following rewards:\n\n|Action|Reward|\n|:----:|:----:|\n|a1    | 0.8  |\n|a2    | 0.4  |\n|b1    | 0.5  |\n|b2    | 0.6  |\n|c1    | 0.4  |\n|c2    | 0.4  |\n|d1    | 0.6  |\n|d2    | 0.2  |\n|d3    | 0.4  |\n\nRunning this model a few times shows that it would:\n- Often select a1 as this has a much higher reward\n- Would always pick c1 as the rewards are the same but is cheaper\n- Had a hard time selecting between b1 and b2 as the rewards are 0.5 and 0.6 but the costs are £8 and £11 respectively\n- Would typically select d3 as being significantly cheaper than d1 even though reward is slightly less"},{"metadata":{"trusted":false,"_uuid":"51d29bddacea2de315caf309fc05b621f322446a"},"cell_type":"code","source":"# Add rewards for more actions\nbudget11 = 30\n\n\nalpha11 = 0.05\nnum_episodes11 = 1000\nepsilon11 = 0.2\n\n\n# Currently not using a reward\nreward11 = [0.8,0.4,0.5,0.6,0.4,0.4,0.6,0.2,0.4]\n\nstart_time = time.time()\n\nMdl11 = MCModelv3(data=data, alpha = alpha11, e = num_episodes11,epsilon = epsilon11, budget = budget11, reward = reward11)\n\nprint(\"--- %s seconds ---\" % (time.time() - start_time))\n\nprint(Mdl11[3])\n\n\nplt.plot(range(0,num_episodes11), Mdl11[0])\nplt.title('Sum of V for all Actions at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()\n\n\nplt.plot(range(0,num_episodes11), Mdl10[1],range(0,num_episodes11), Mdl11[2])\nplt.title('Sum of V for the cheapest actions and others seperated at each Episode')\nplt.xlabel('Episode')\nplt.ylabel('Sum of V')\nplt.show()","execution_count":112,"outputs":[]},{"metadata":{"_uuid":"bafbf50265c6e718b600251e4d19fcf4dd886944"},"cell_type":"markdown","source":"# Conclusion\n\nWe have managed to build a Monte Carlo Reinforcement Learning model to: 1) recommend products below a budget, 2) recommend the cheapest products and 3) recommened the best products based on a preference that is still below a budget.\n\n\n\nAlong the way, we have demonstrated the effect of changing parameters in reinforcement learning and how understanding these enables us to reach a desired result. \n\n\nThere is much more that we could do, in my mind, the end goal would be to apply to a real recipe and products from a supermarket where the increased number of ingredients and products need to be accounted for.\n\n\n\nI created this sample data and problem to better my understanding of Reinforcement Learning and hope that you find it useful.\n\nThanks\n\nPhilip Osborne\n\n"},{"metadata":{"collapsed":true,"trusted":false,"_uuid":"97cc458359de02ca6475817f0c9397debe81d458"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"anaconda-cloud":{},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}