ex1_MSLE.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3"},"colab":{"provenance":[{"file_id":"1wNhyHG0FgShQtAV3BjCiC_H9PeEMrTC5","timestamp":1666031878650}],"collapsed_sections":[]}},"cells":[{"cell_type":"markdown","source":["<center><img src='https://drive.google.com/uc?id=1_utx_ZGclmCwNttSe40kYA6VHzNocdET' height=\"60\"></center>\n","\n","AI TECH - Akademia Innowacyjnych Zastosowań Technologii Cyfrowych. Program Operacyjny Polska Cyfrowa na lata 2014-2020\n","<hr>\n","\n","<center><img src='https://drive.google.com/uc?id=1BXZ0u3562N_MqCLcekI-Ens77Kk4LpPm'></center>\n","\n","<center>\n","Projekt współfinansowany ze środków Unii Europejskiej w ramach Europejskiego Funduszu Rozwoju Regionalnego \n","Program Operacyjny Polska Cyfrowa na lata 2014-2020,\n","Oś Priorytetowa nr 3 \"Cyfrowe kompetencje społeczeństwa\" Działanie  nr 3.2 \"Innowacyjne rozwiązania na rzecz aktywizacji cyfrowej\" \n","Tytuł projektu:  „Akademia Innowacyjnych Zastosowań Technologii Cyfrowych (AI Tech)”\n","    </center>"],"metadata":{"id":"op6nRS5M-oXX"}},{"cell_type":"markdown","metadata":{"id":"VqdiMxZx-EoD"},"source":["# Linear regression \n","\n","In this exercise you will use linear regression to predict flat prices. Training will be handled via gradient descent and we will:\n","* have multiple features (i.e. variables used to make the prediction),\n","* employ some basic feature engineering,\n","* work with a non-standard loss function.\n","\n","Let's start with getting the data."]},{"cell_type":"code","metadata":{"id":"heeKjQ2r-EoH","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120521744,"user_tz":-120,"elapsed":2282,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"d33d014e-5071-4024-e3cd-4276a56b74a6"},"source":["%matplotlib inline\n","\n","!wget -O mieszkania.csv https://www.dropbox.com/s/zey0gx91pna8irj/mieszkania.csv?dl=1\n","!wget -O mieszkania_test.csv https://www.dropbox.com/s/dbrj6sbxb4ayqjz/mieszkania_test.csv?dl=1"],"execution_count":2,"outputs":[{"output_type":"stream","name":"stdout","text":["--2022-10-18 19:15:21--  https://www.dropbox.com/s/zey0gx91pna8irj/mieszkania.csv?dl=1\n","Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6021:18::a27d:4112\n","Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n","Location: /s/dl/zey0gx91pna8irj/mieszkania.csv [following]\n","--2022-10-18 19:15:21--  https://www.dropbox.com/s/dl/zey0gx91pna8irj/mieszkania.csv\n","Reusing existing connection to www.dropbox.com:443.\n","HTTP request sent, awaiting response... 302 Found\n","Location: https://uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com/cd/0/get/BvE2Baq4RdD8U_a6TaDsGH8rm5ihQ9LK9eJ2YXNurvxsYBXIADDKxZHFCtp-uozZuTBhIKzjvNPMEcI6ZhYjtJxVOm_oBUl-uAm7ctvctpQrBc52E_q6aR_6ouNvwmQFLFnYBAiUpWCqQyGT36q0hZtPWKCI13fblKHlsPUkBAsvjA/file?dl=1# [following]\n","--2022-10-18 19:15:22--  https://uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com/cd/0/get/BvE2Baq4RdD8U_a6TaDsGH8rm5ihQ9LK9eJ2YXNurvxsYBXIADDKxZHFCtp-uozZuTBhIKzjvNPMEcI6ZhYjtJxVOm_oBUl-uAm7ctvctpQrBc52E_q6aR_6ouNvwmQFLFnYBAiUpWCqQyGT36q0hZtPWKCI13fblKHlsPUkBAsvjA/file?dl=1\n","Resolving uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com (uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6021:15::a27d:410f\n","Connecting to uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com (uca08ee05cbe6ef58bc79fecdf7c.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 6211 (6.1K) [application/binary]\n","Saving to: ‘mieszkania.csv’\n","\n","mieszkania.csv      100%[===================>]   6.07K  --.-KB/s    in 0s      \n","\n","2022-10-18 19:15:22 (645 MB/s) - ‘mieszkania.csv’ saved [6211/6211]\n","\n","--2022-10-18 19:15:22--  https://www.dropbox.com/s/dbrj6sbxb4ayqjz/mieszkania_test.csv?dl=1\n","Resolving www.dropbox.com (www.dropbox.com)... 162.125.2.18, 2620:100:6021:18::a27d:4112\n","Connecting to www.dropbox.com (www.dropbox.com)|162.125.2.18|:443... connected.\n","HTTP request sent, awaiting response... 302 Found\n","Location: /s/dl/dbrj6sbxb4ayqjz/mieszkania_test.csv [following]\n","--2022-10-18 19:15:22--  https://www.dropbox.com/s/dl/dbrj6sbxb4ayqjz/mieszkania_test.csv\n","Reusing existing connection to www.dropbox.com:443.\n","HTTP request sent, awaiting response... 302 Found\n","Location: https://ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com/cd/0/get/BvFD80p7qBrNQBEwEEICAQCyUAEuG8v9iIolDYgY0ovszWoR6s48BB21kVjax6loZYyhHs1ydSmFDo2MyaTDCAGzChjnn1GLnk3tOwkA4kC2O9tT1EgOrfOXTs6QxIDq_hqM0TdQ3cxz49xHR0cXK2ZSlzB0ZSrpehDr4Fm5oukQKA/file?dl=1# [following]\n","--2022-10-18 19:15:23--  https://ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com/cd/0/get/BvFD80p7qBrNQBEwEEICAQCyUAEuG8v9iIolDYgY0ovszWoR6s48BB21kVjax6loZYyhHs1ydSmFDo2MyaTDCAGzChjnn1GLnk3tOwkA4kC2O9tT1EgOrfOXTs6QxIDq_hqM0TdQ3cxz49xHR0cXK2ZSlzB0ZSrpehDr4Fm5oukQKA/file?dl=1\n","Resolving ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com (ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com)... 162.125.2.15, 2620:100:6021:15::a27d:410f\n","Connecting to ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com (ucd5eb5f739b6e0d847b04233fa5.dl.dropboxusercontent.com)|162.125.2.15|:443... connected.\n","HTTP request sent, awaiting response... 200 OK\n","Length: 6247 (6.1K) [application/binary]\n","Saving to: ‘mieszkania_test.csv’\n","\n","mieszkania_test.csv 100%[===================>]   6.10K  --.-KB/s    in 0s      \n","\n","2022-10-18 19:15:23 (677 MB/s) - ‘mieszkania_test.csv’ saved [6247/6247]\n","\n"]}]},{"cell_type":"code","metadata":{"id":"yI9wFXv5-EoP","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120521744,"user_tz":-120,"elapsed":7,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"d3f0622d-d4f1-43be-ecc5-22add8e80214"},"source":["!head mieszkania.csv mieszkania_test.csv"],"execution_count":3,"outputs":[{"output_type":"stream","name":"stdout","text":["==> mieszkania.csv <==\n","m2,dzielnica,ilość_sypialni,ilość_łazienek,rok_budowy,parking_podziemny,cena\n","104,mokotowo,2,2,1940,1,780094\n","43,ochotowo,1,1,1970,1,346912\n","128,grodziskowo,3,2,1916,1,523466\n","112,mokotowo,3,2,1920,1,830965\n","149,mokotowo,3,3,1977,0,1090479\n","80,ochotowo,2,2,1937,0,599060\n","58,ochotowo,2,1,1922,0,463639\n","23,ochotowo,1,1,1929,0,166785\n","40,mokotowo,1,1,1973,0,318849\n","\n","==> mieszkania_test.csv <==\n","m2,dzielnica,ilość_sypialni,ilość_łazienek,rok_budowy,parking_podziemny,cena\n","71,wolowo,2,2,1912,1,322227\n","45,mokotowo,1,1,1938,0,295878\n","38,mokotowo,1,1,1999,1,306530\n","70,ochotowo,2,2,1980,1,553641\n","136,mokotowo,3,2,1939,1,985348\n","128,wolowo,3,2,1983,1,695726\n","23,grodziskowo,1,1,1975,0,99751\n","117,mokotowo,3,2,1942,0,891261\n","65,ochotowo,2,1,2002,1,536499\n"]}]},{"cell_type":"markdown","metadata":{"id":"iH8Ut02G-EoU"},"source":["Each row in the data represents a separate property. Our goal is to use the data from `mieszkania.csv` to create a model that can predict a property's price (i.e. `cena`) given its features (i.e. `m2,dzielnica,ilosc_sypialni,ilosc_lazienek,rok_budowy,parking_podziemny`). \n","\n","From now on, we should interfere only with `mieszkania.csv` (dubbed the training dataset) to make our decisions and create the model. The (only) purpose of `mieszkania_test.csv` is to test our model on **unseen** data.\n","\n","Our predictions should minimize the so-called mean squared logarithmic error:\n","$$\n","MSLE = \\frac{1}{n} \\sum_{i=1}^n (\\log(1+y_i) - \\log(1+p_i))^2,\n","$$\n","where $y_i$ is the ground truth, and $p_i$ is our prediction."]},{"cell_type":"markdown","metadata":{"id":"y8QX6Ncr-EoW"},"source":["Let's start with implementing the loss function."]},{"cell_type":"code","metadata":{"id":"JAiB8S9A-EoX","executionInfo":{"status":"ok","timestamp":1666120522291,"user_tz":-120,"elapsed":549,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}}},"source":["import numpy as np\n","import pandas as pd\n","\n","def msle(ys, ps):\n","    assert len(ys) == len(ps)\n","    #################################\n","    # TODO: Implement this function #\n","    #################################\n","    return np.mean(np.square(np.log(1+ys) - np.log(1+ps)))"],"execution_count":4,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zAmd2qLR-Eob"},"source":["The simplest model is predicting the same constant for each instance. Test your implementation of msle against outputing the mean price."]},{"cell_type":"code","source":["data_train = pd.read_csv('mieszkania.csv')\n","X_train = data_train.drop(columns=['cena'])\n","y_train = data_train['cena'].to_numpy().reshape(-1,1)\n","data_test = pd.read_csv('mieszkania_test.csv')\n","X_test = data_test.drop(columns=['cena'])\n","y_test = data_test['cena'].to_numpy().reshape(-1,1)\n","X_train.head()"],"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":206},"id":"0-Yi_YAue2ft","executionInfo":{"status":"ok","timestamp":1666120522291,"user_tz":-120,"elapsed":10,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"2a794733-6ebd-49a5-f1b3-cb5c7695dcd5"},"execution_count":5,"outputs":[{"output_type":"execute_result","data":{"text/plain":["    m2    dzielnica  ilość_sypialni  ilość_łazienek  rok_budowy  \\\n","0  104     mokotowo               2               2        1940   \n","1   43     ochotowo               1               1        1970   \n","2  128  grodziskowo               3               2        1916   \n","3  112     mokotowo               3               2        1920   \n","4  149     mokotowo               3               3        1977   \n","\n","   parking_podziemny  \n","0                  1  \n","1                  1  \n","2                  1  \n","3                  1  \n","4                  0  "],"text/html":["\n","  <div id=\"df-e06f5af2-1d9c-48e5-affb-e4177433e5a0\">\n","    <div class=\"colab-df-container\">\n","      <div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>m2</th>\n","      <th>dzielnica</th>\n","      <th>ilość_sypialni</th>\n","      <th>ilość_łazienek</th>\n","      <th>rok_budowy</th>\n","      <th>parking_podziemny</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>104</td>\n","      <td>mokotowo</td>\n","      <td>2</td>\n","      <td>2</td>\n","      <td>1940</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>43</td>\n","      <td>ochotowo</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>1970</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>128</td>\n","      <td>grodziskowo</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>1916</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>112</td>\n","      <td>mokotowo</td>\n","      <td>3</td>\n","      <td>2</td>\n","      <td>1920</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>149</td>\n","      <td>mokotowo</td>\n","      <td>3</td>\n","      <td>3</td>\n","      <td>1977</td>\n","      <td>0</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>\n","      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e06f5af2-1d9c-48e5-affb-e4177433e5a0')\"\n","              title=\"Convert this dataframe to an interactive table.\"\n","              style=\"display:none;\">\n","        \n","  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n","       width=\"24px\">\n","    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n","    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n","  </svg>\n","      </button>\n","      \n","  <style>\n","    .colab-df-container {\n","      display:flex;\n","      flex-wrap:wrap;\n","      gap: 12px;\n","    }\n","\n","    .colab-df-convert {\n","      background-color: #E8F0FE;\n","      border: none;\n","      border-radius: 50%;\n","      cursor: pointer;\n","      display: none;\n","      fill: #1967D2;\n","      height: 32px;\n","      padding: 0 0 0 0;\n","      width: 32px;\n","    }\n","\n","    .colab-df-convert:hover {\n","      background-color: #E2EBFA;\n","      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n","      fill: #174EA6;\n","    }\n","\n","    [theme=dark] .colab-df-convert {\n","      background-color: #3B4455;\n","      fill: #D2E3FC;\n","    }\n","\n","    [theme=dark] .colab-df-convert:hover {\n","      background-color: #434B5C;\n","      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n","      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n","      fill: #FFFFFF;\n","    }\n","  </style>\n","\n","      <script>\n","        const buttonEl =\n","          document.querySelector('#df-e06f5af2-1d9c-48e5-affb-e4177433e5a0 button.colab-df-convert');\n","        buttonEl.style.display =\n","          google.colab.kernel.accessAllowed ? 'block' : 'none';\n","\n","        async function convertToInteractive(key) {\n","          const element = document.querySelector('#df-e06f5af2-1d9c-48e5-affb-e4177433e5a0');\n","          const dataTable =\n","            await google.colab.kernel.invokeFunction('convertToInteractive',\n","                                                     [key], {});\n","          if (!dataTable) return;\n","\n","          const docLinkHtml = 'Like what you see? Visit the ' +\n","            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n","            + ' to learn more about interactive tables.';\n","          element.innerHTML = '';\n","          dataTable['output_type'] = 'display_data';\n","          await google.colab.output.renderOutput(dataTable, element);\n","          const docLink = document.createElement('div');\n","          docLink.innerHTML = docLinkHtml;\n","          element.appendChild(docLink);\n","        }\n","      </script>\n","    </div>\n","  </div>\n","  "]},"metadata":{},"execution_count":5}]},{"cell_type":"code","metadata":{"id":"b0RT7VW7-Eoc","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120522292,"user_tz":-120,"elapsed":9,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"75ee6d63-a109-4c2f-a281-559b6411c7a1"},"source":["###################################################\n","# TODO: Compute msle for outputing the mean price #\n","###################################################\n","\n","mean_price_train = np.mean(y_train)\n","mean_prediction = np.full_like(y_train, mean_price_train)\n","\n","msle_test = msle(y_train, mean_prediction)\n","print(msle_test)"],"execution_count":6,"outputs":[{"output_type":"stream","name":"stdout","text":["0.3915250389336738\n"]}]},{"cell_type":"markdown","metadata":{"id":"a0zMmTiv-Eof"},"source":["Recall that outputing the mean minimzes $MSE$. However, we're now dealing with $MSLE$.\n","\n","Think of a constant that should result in the lowest $MSLE$."]},{"cell_type":"code","metadata":{"id":"Q0Z4drNd-Eog","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120522292,"user_tz":-120,"elapsed":7,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"15a9553f-ca7f-43ba-8383-80c777bfbe87"},"source":["#############################################\n","# TODO: Find this constant and compute msle #\n","#############################################\n","\n","c = np.exp(np.sum(np.log(1+y_train))/len(y_train)) - 1\n","\n","mean_prediction = np.full_like(y_train, c)\n","msle_constant = msle(y_train, mean_prediction)\n","\n","print(msle_constant)"],"execution_count":7,"outputs":[{"output_type":"stream","name":"stdout","text":["0.36488961221465716\n"]}]},{"cell_type":"markdown","metadata":{"id":"2RdHlBw8-Eoi"},"source":["Now, let's implement a standard linear regression model. "]},{"cell_type":"code","metadata":{"id":"Vn97D3vd-Eoj","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120522692,"user_tz":-120,"elapsed":405,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"f2d60ec7-6641-42c5-a3c0-fa87d180a305"},"source":["##########################################################\n","# TODO: Implement linear regression and compute its msle #\n","##########################################################\n","\n","def mse_loss(ys, ps):\n","    assert len(ys) == len(ps)\n","    return np.mean(np.square(ys-ps))\n","\n","lr = 2e-9 # step size\n","n_epochs = 20000 # number of passes over the training data\n","\n","def predict(X, w):\n","    return X @ w\n","\n","def evaluate(X, y, w):\n","    return mse_loss(y, predict(X, w))\n","\n","\n","X_train_vec = X_train.drop(columns=['dzielnica']).to_numpy()\n","X_train_vec = np.concatenate([X_train_vec, np.ones((X_train_vec.shape[0], 1))], axis=-1)\n","\n","X_test_vec = X_test.drop(columns=['dzielnica']).to_numpy()\n","X_test_vec = np.concatenate([X_test_vec, np.ones((X_test_vec.shape[0], 1))], axis=-1)\n","\n","n_features = X_train_vec.shape[1]\n","w = np.ones((n_features, 1))\n","\n","for i in range(n_epochs):\n","    y_hat = predict(X_train_vec, w)\n","    error = y_hat - y_train\n","    grad_w = X_train_vec.T @ error\n","\n","    w = w - lr * grad_w\n","    \n","    # loss = evaluate(X_train_vec, y_train, w)\n","    # print(f'Iter: {i:>3} Loss: {loss:8.8f}')\n","\n","prediction = predict(X_train_vec, w)\n","msle_lr = msle(y_train, prediction)\n","print(msle_lr)\n","# prediction = predict(X_train_vec, w)\n","# msle_lr = msle(y_train, prediction)\n","# print(msle_lr)"],"execution_count":8,"outputs":[{"output_type":"stream","name":"stdout","text":["0.0544979826154857\n"]}]},{"cell_type":"markdown","metadata":{"id":"nzrrOte5-Eol"},"source":["Note that the loss function that the algorithms optimizes (i.e $MSE$) differs from $MSLE$. We've already seen that this may result in a suboptimal solution.\n","\n","How can you change the setting so that we optimze $MSLE$ instead?"]},{"cell_type":"markdown","metadata":{"id":"qVD8kWEJ-Eom"},"source":["Hint: \n","<sub><sup><sub><sup><sub><sup>\n","Be lazy. We don't want to change the algorithm.\n","</sup></sub></sup></sub></sup></sub>"]},{"cell_type":"code","metadata":{"id":"MniIPMg8-Eom","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666120682406,"user_tz":-120,"elapsed":1026,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"e6a9289c-e359-435d-a3ee-b62f72012e48"},"source":["#############################################\n","# TODO: Optimize msle and compare the error #\n","#############################################\n","\n","lr = 8e-4\n","n_epochs = 30000\n","\n","X_train_vec = X_train.drop(columns=['dzielnica']).to_numpy()\n","X_train_vec = np.concatenate([X_train_vec, np.ones((X_train_vec.shape[0], 1))], axis=-1)\n","\n","X_test_vec = X_test.drop(columns=['dzielnica']).to_numpy()\n","X_test_vec = np.concatenate([X_test_vec, np.ones((X_test_vec.shape[0], 1))], axis=-1)\n","\n","\n","def fit(X, y, epochs, lr, debug=False):\n","  n_features = X.shape[1]\n","  w = np.ones((n_features, 1))\n","  for i in range(n_epochs):\n","      y_hat = predict(X, w)\n","      error = (np.log1p(y_hat) - np.log1p(y))\n","      grad_w = X.T @ error\n","      w = w - lr * grad_w\n","\n","      if debug:\n","        loss = msle(y, y_hat)\n","        print(f'Iter: {i:>3} Loss: {loss:8.8f}')\n","  \n","  return w\n","    \n","def eval(X, y_true, w):\n","  prediction = predict(X, w)\n","  msle_loss = msle(y_true, prediction)\n","  return msle_loss\n","\n","\n","w = fit(X_train_vec, y_train, n_epochs, lr)\n","msle_train = eval(X_train_vec, y_train, w)\n","msle_test = eval(X_test_vec, y_test, w)\n","print(f'Train loss: {msle_train}')\n","print(f'Test  loss: {msle_test}')"],"execution_count":14,"outputs":[{"output_type":"stream","name":"stdout","text":["Train loss: 0.05302723978146017\n","Test  loss: 0.08054085102342762\n"]}]},{"cell_type":"markdown","source":["$$\n","MSLE = \\frac{1}{n} \\sum_{i=1}^n (\\log(1+y_i) - \\log(1+p_i))^2,\n","$$"],"metadata":{"id":"Hi99x6MNwX1D"}},{"cell_type":"markdown","metadata":{"id":"OvWrBkLu-Eoo"},"source":["Without any feature engineering our model approximates the price as a linear combination of original features:\n","$$\n","\\text{price} \\approx w_1 \\cdot \\text{area} + w_2 \\cdot \\text{district} + \\dots.\n","$$\n","Let's now introduce some interactions between the variables. For instance, let's consider a following formula:\n","$$\n","\\text{price} \\approx w_1 \\cdot \\text{area} \\cdot \\text{avg. price in the district per sq. meter} + w_2 \\cdot \\dots + \\dots.\n","$$\n","Here, we model the price with far greater granularity, and we may expect to see more acurate results."]},{"cell_type":"markdown","metadata":{"id":"zBM60E7t-Eop"},"source":["Add some feature engineering to your model. Be sure to play with the data and not with the algorithm's code. \n","\n","Think how to make sure that your model is capable of capturing the $w_1 \\cdot \\text{area} \\cdot \\text{avg. price...}$ part, without actually computing the averages."]},{"cell_type":"markdown","metadata":{"id":"XT5DziNC-Eoq"},"source":["Hint: \n","<sub><sup><sub><sup><sub><sup>\n","Is having a binary encoding for each district and multiplying it by area enough?\n","</sup></sub></sup></sub></sup></sub>"]},{"cell_type":"markdown","metadata":{"id":"PKOVCHNz-Eor"},"source":["Hint 2: \n","<sub><sup><sub><sup><sub><sup>\n","Why not multiply everything together? I.e. (A,B,C) -> (AB,AC,BC).\n","</sup></sub></sup></sub></sup></sub>"]},{"cell_type":"code","metadata":{"id":"DPviz09O-Eor","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666121063751,"user_tz":-120,"elapsed":600,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"30bb6cb2-234f-4fc6-f197-9823c50aaf47"},"source":["###############################################\n","# TODO: Implement the feature engineering part #\n","###############################################\n","\n","from sklearn.preprocessing import StandardScaler\n","\n","\n","X_train_vec = X_train.drop(columns=['dzielnica', 'm2']).to_numpy()\n","districts_train = pd.get_dummies(X_train['dzielnica'], prefix='dzielnia')\n","districts_train = districts_train.apply(lambda col: col*X_train['m2'])\n","X_train_vec = np.concatenate([X_train_vec, districts_train], axis=-1)\n","# scaler = StandardScaler()\n","# X_train_vec = scaler.fit_transform(X_train_vec)\n","\n","X_train_vec = np.concatenate([X_train_vec, np.ones((X_train_vec.shape[0], 1))], axis=-1)\n","\n","\n","\n","X_test_vec = X_test.drop(columns=['dzielnica', 'm2']).to_numpy()\n","districts_test = pd.get_dummies(X_test['dzielnica'], prefix='dzielnia')\n","districts_test = districts_test.apply(lambda col: col*X_test['m2'])\n","districts_test.reindex(columns = districts_train.columns, fill_value=0)\n","X_test_vec = np.concatenate([X_test_vec, districts_test], axis=-1)\n","# X_test_vec = scaler.transform(X_test_vec)\n","\n","X_test_vec = np.concatenate([X_test_vec, np.ones((X_test_vec.shape[0], 1))], axis=-1)\n","\n","lr = 9e-4\n","n_epochs = 30000\n","\n","w = fit(X_train_vec, y_train, n_epochs, lr)\n","print(w)"],"execution_count":27,"outputs":[{"output_type":"stream","name":"stdout","text":["[[4.41244310e+02]\n"," [3.02078316e+02]\n"," [1.18755844e+01]\n"," [4.58685643e+01]\n"," [4.05557995e+03]\n"," [7.14906191e+03]\n"," [7.30043370e+03]\n"," [5.20985148e+03]\n"," [2.48108490e-02]]\n"]}]},{"cell_type":"code","metadata":{"id":"v2Lj1TPr-Eot","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1666121064058,"user_tz":-120,"elapsed":2,"user":{"displayName":"Michał Janik","userId":"10483929333441523386"}},"outputId":"f7133d75-c89e-43df-e82f-40900b0458f8"},"source":["##############################################################\n","# TODO: Test your solution on the training and test datasets #\n","##############################################################\n","\n","msle_train = eval(X_train_vec, y_train, w)\n","msle_test = eval(X_test_vec, y_test, w)\n","print(f'Train MSLE loss: {msle_train}')\n","print(f'Test MSLE  loss: {msle_test}')"],"execution_count":28,"outputs":[{"output_type":"stream","name":"stdout","text":["Train MSLE loss: 0.007410406876304113\n","Test MSLE  loss: 0.01704737964822043\n"]}]},{"cell_type":"code","source":[],"metadata":{"id":"9zS5OMHwiw7Q"},"execution_count":null,"outputs":[]}]}