fastai2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "fastai2.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "[View in Colaboratory](https://colab.research.google.com/gist/naviarh/e9ff9583d6587d619c845f9518b949e3/fastai2.ipynb)"
      ]
    },
    {
      "metadata": {
        "id": "9Ca68P70uDhm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Обучение на структурированных данных\n",
        "Типы структурированных данных (признаков):\n",
        "- категориальные\n",
        "- Количественные\n",
        "\n",
        "Для категориальных признаков применяется метод эмбеддинга\n",
        "\n",
        "3 способ передачи данных (структурированных) нейросети - датафрейм (таблица)\n",
        "\n",
        "(Для неструтурированных данных - размещение по каталогам или по файлу разметки csv )"
      ]
    },
    {
      "metadata": {
        "id": "pRjLc4Z8NJK_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Алгоритм:\n",
        "1. Находим все возможные полезные признаки;\n",
        "2. Разделяем признаки на количественные и категориальные;\n",
        "3. Формируем датафрейм признаков;\n",
        "4. Создаём валидационную выборку;\n",
        "5. Объект данных (ColumnarModelData.from_data_frame())\n",
        "6. Создаём массив размеров матриц эмбеддинга\n",
        "7. Создаём алгоритм обучения (моделль нейросети)\n",
        "8. Обучаем модель\n"
      ]
    },
    {
      "metadata": {
        "id": "HrPDR7_0uEWN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#!/usr/bin/env python3\n",
        "# -*- coding: utf-8 -*-"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5uPsJWdCXWjh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install --upgrade pip\n",
        "!pip3 install fastai"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TKT9cBb6XaSS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "31f823a8-c4b4-43a4-ac44-dc48b904eefc"
      },
      "cell_type": "code",
      "source": [
        "# Put these at the top of every notebook, to get automatic reloading and inline plotting\n",
        "%reload_ext autoreload\n",
        "%autoreload 2\n",
        "%matplotlib inline\n",
        "!pwd"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\r\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "rpY3XcKXXg_t",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 59
        },
        "outputId": "1d53b416-8f22-43f3-a715-a34ffa94fb69"
      },
      "cell_type": "code",
      "source": [
        "import subprocess, os\n",
        "os.uname()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "posix.uname_result(sysname='Linux', nodename='54c7ac56ac31', release='4.14.33+', version='#1 SMP Wed Jun 20 01:36:48 PDT 2018', machine='x86_64')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "metadata": {
        "id": "2BjQb7-MXjow",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "5786ef6e-e7a8-4ca6-bda2-70de6e9bfdf7"
      },
      "cell_type": "code",
      "source": [
        "from fastai.structured import *\n",
        "from fastai.column_data import *\n",
        "!pip3 show fastai torch | grep Name -A 1"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Name: fastai\r\n",
            "Version: 0.7.0\n",
            "--\n",
            "Name: torch\n",
            "Version: 0.3.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hByGgSB2XxXj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#from fastai.imports import *\n",
        "#from fastai.transforms import *\n",
        "#from fastai.conv_learner import *\n",
        "#from fastai.model import *\n",
        "#from fastai.dataset import *\n",
        "#from fastai.sgdr import *\n",
        "#from fastai.plots import *\n",
        "#!pip3 show fastai torch | grep Name -A 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mmGfSBAAYR6q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b2e0d22e-c862-4934-af0a-b95b55ac76e6"
      },
      "cell_type": "code",
      "source": [
        "torch.cuda.set_device(0)\n",
        "torch.cuda.get_device_name(0)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Tesla K80'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "metadata": {
        "id": "N8A6-IJlZhwm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Устанавливаем формат показа массивов numpy\n",
        "np.set_printoptions(threshold=50, edgeitems=20)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EZZkvdu-YCXO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Исходные данные"
      ]
    },
    {
      "metadata": {
        "id": "JOyE75SeYC1D",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Создадим путь к каталогу PATH\n",
        "PATH='data/rossmann/'\n",
        "os.makedirs(PATH, exist_ok=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NQIjwvD2aY_3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "c53f3910-e722-40f3-e5bc-0b776b3abdac"
      },
      "cell_type": "code",
      "source": [
        "# Скачаем нужные данные\n",
        "!wget http://files.fast.ai/part2/lesson14/rossmann.tgz"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2018-07-28 10:23:00--  http://files.fast.ai/part2/lesson14/rossmann.tgz\n",
            "Resolving files.fast.ai (files.fast.ai)... 67.205.15.147\n",
            "Connecting to files.fast.ai (files.fast.ai)|67.205.15.147|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 7730448 (7.4M) [text/plain]\n",
            "Saving to: ‘rossmann.tgz’\n",
            "\n",
            "rossmann.tgz        100%[===================>]   7.37M  9.86MB/s    in 0.7s    \n",
            "\n",
            "2018-07-28 10:23:01 (9.86 MB/s) - ‘rossmann.tgz’ saved [7730448/7730448]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4glnYHfWbLJs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "aa01ea09-fa63-475d-e05a-a5fab241878f"
      },
      "cell_type": "code",
      "source": [
        "# Распакуем данные\n",
        "!tar zxvf rossmann.tgz -C {PATH}"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "googletrend.csv\r\n",
            "sample_submission.csv\r\n",
            "state_names.csv\r\n",
            "store.csv\r\n",
            "store_states.csv\r\n",
            "test.csv\r\n",
            "train.csv\n",
            "weather.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bu9fxcc0hb5J",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Feature Space:\n",
        "- googletrend: trend of certain google keywords over time, found by users to correlate well given data\n",
        "- List of German state names\n",
        "- store: List of stores\n",
        "- store_states: mapping of store to the German state they are in\n",
        "- test: testing set\n",
        "- train: Training set provided by competition\n",
        "- weather: weather"
      ]
    },
    {
      "metadata": {
        "id": "CpzVNjUWYD20",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Функция объединения данных\n",
        "def concat_csvs(dirname):\n",
        "    path = f'{PATH}{dirname}'\n",
        "    filenames=glob(f\"{PATH}/*.csv\")\n",
        "\n",
        "    wrote_header = False\n",
        "    with open(f\"{path}.csv\",\"w\") as outputfile:\n",
        "        for filename in filenames:\n",
        "            name = filename.split(\".\")[0]\n",
        "            with open(filename) as f:\n",
        "                line = f.readline()\n",
        "                if not wrote_header:\n",
        "                    wrote_header = True\n",
        "                    outputfile.write(\"file,\"+line)\n",
        "                for line in f:\n",
        "                     outputfile.write(name + \",\" + line)\n",
        "                outputfile.write(\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H8EgKD3QiQQp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# concat_csvs('googletrend')\n",
        "# concat_csvs('weather')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2LU8sKHXiVbG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "table_names = ['train', 'store', 'store_states', 'state_names', 'googletrend', 'weather', 'test']\n",
        "tables = [pd.read_csv(f'{PATH}{fname}.csv', low_memory=False) for fname in table_names]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QwjStv4VizSM",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from IPython.display import HTML, display"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "j_QYussYmx_1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1465
        },
        "outputId": "19c375fe-c929-4e34-9faf-df0c77f81a19"
      },
      "cell_type": "code",
      "source": [
        "# Посмотрим как выглядят данные в таблицах\n",
        "for t in tables: display(t.head())"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Date</th>\n",
              "      <th>Sales</th>\n",
              "      <th>Customers</th>\n",
              "      <th>Open</th>\n",
              "      <th>Promo</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-31</td>\n",
              "      <td>5263</td>\n",
              "      <td>555</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-31</td>\n",
              "      <td>6064</td>\n",
              "      <td>625</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-31</td>\n",
              "      <td>8314</td>\n",
              "      <td>821</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-31</td>\n",
              "      <td>13995</td>\n",
              "      <td>1498</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>2015-07-31</td>\n",
              "      <td>4822</td>\n",
              "      <td>559</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Store  DayOfWeek        Date  Sales  Customers  Open  Promo StateHoliday  \\\n",
              "0      1          5  2015-07-31   5263        555     1      1            0   \n",
              "1      2          5  2015-07-31   6064        625     1      1            0   \n",
              "2      3          5  2015-07-31   8314        821     1      1            0   \n",
              "3      4          5  2015-07-31  13995       1498     1      1            0   \n",
              "4      5          5  2015-07-31   4822        559     1      1            0   \n",
              "\n",
              "   SchoolHoliday  \n",
              "0              1  \n",
              "1              1  \n",
              "2              1  \n",
              "3              1  \n",
              "4              1  "
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>StoreType</th>\n",
              "      <th>Assortment</th>\n",
              "      <th>CompetitionDistance</th>\n",
              "      <th>CompetitionOpenSinceMonth</th>\n",
              "      <th>CompetitionOpenSinceYear</th>\n",
              "      <th>Promo2</th>\n",
              "      <th>Promo2SinceWeek</th>\n",
              "      <th>Promo2SinceYear</th>\n",
              "      <th>PromoInterval</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>c</td>\n",
              "      <td>a</td>\n",
              "      <td>1270.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>2008.0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>570.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>2007.0</td>\n",
              "      <td>1</td>\n",
              "      <td>13.0</td>\n",
              "      <td>2010.0</td>\n",
              "      <td>Jan,Apr,Jul,Oct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>14130.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>2006.0</td>\n",
              "      <td>1</td>\n",
              "      <td>14.0</td>\n",
              "      <td>2011.0</td>\n",
              "      <td>Jan,Apr,Jul,Oct</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>c</td>\n",
              "      <td>c</td>\n",
              "      <td>620.0</td>\n",
              "      <td>9.0</td>\n",
              "      <td>2009.0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>29910.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2015.0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Store StoreType Assortment  CompetitionDistance  CompetitionOpenSinceMonth  \\\n",
              "0      1         c          a               1270.0                        9.0   \n",
              "1      2         a          a                570.0                       11.0   \n",
              "2      3         a          a              14130.0                       12.0   \n",
              "3      4         c          c                620.0                        9.0   \n",
              "4      5         a          a              29910.0                        4.0   \n",
              "\n",
              "   CompetitionOpenSinceYear  Promo2  Promo2SinceWeek  Promo2SinceYear  \\\n",
              "0                    2008.0       0              NaN              NaN   \n",
              "1                    2007.0       1             13.0           2010.0   \n",
              "2                    2006.0       1             14.0           2011.0   \n",
              "3                    2009.0       0              NaN              NaN   \n",
              "4                    2015.0       0              NaN              NaN   \n",
              "\n",
              "     PromoInterval  \n",
              "0              NaN  \n",
              "1  Jan,Apr,Jul,Oct  \n",
              "2  Jan,Apr,Jul,Oct  \n",
              "3              NaN  \n",
              "4              NaN  "
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>State</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>HE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>TH</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NW</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>BE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>SN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Store State\n",
              "0      1    HE\n",
              "1      2    TH\n",
              "2      3    NW\n",
              "3      4    BE\n",
              "4      5    SN"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>StateName</th>\n",
              "      <th>State</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>BadenWuerttemberg</td>\n",
              "      <td>BW</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Bayern</td>\n",
              "      <td>BY</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Berlin</td>\n",
              "      <td>BE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Brandenburg</td>\n",
              "      <td>BB</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Bremen</td>\n",
              "      <td>HB</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           StateName State\n",
              "0  BadenWuerttemberg    BW\n",
              "1             Bayern    BY\n",
              "2             Berlin    BE\n",
              "3        Brandenburg    BB\n",
              "4             Bremen    HB"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file</th>\n",
              "      <th>week</th>\n",
              "      <th>trend</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "      <td>2012-12-02 - 2012-12-08</td>\n",
              "      <td>96</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "      <td>2012-12-09 - 2012-12-15</td>\n",
              "      <td>95</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "      <td>2012-12-16 - 2012-12-22</td>\n",
              "      <td>91</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "      <td>2012-12-23 - 2012-12-29</td>\n",
              "      <td>48</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "      <td>2012-12-30 - 2013-01-05</td>\n",
              "      <td>67</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             file                     week  trend\n",
              "0  Rossmann_DE_SN  2012-12-02 - 2012-12-08     96\n",
              "1  Rossmann_DE_SN  2012-12-09 - 2012-12-15     95\n",
              "2  Rossmann_DE_SN  2012-12-16 - 2012-12-22     91\n",
              "3  Rossmann_DE_SN  2012-12-23 - 2012-12-29     48\n",
              "4  Rossmann_DE_SN  2012-12-30 - 2013-01-05     67"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file</th>\n",
              "      <th>Date</th>\n",
              "      <th>Max_TemperatureC</th>\n",
              "      <th>Mean_TemperatureC</th>\n",
              "      <th>Min_TemperatureC</th>\n",
              "      <th>Dew_PointC</th>\n",
              "      <th>MeanDew_PointC</th>\n",
              "      <th>Min_DewpointC</th>\n",
              "      <th>Max_Humidity</th>\n",
              "      <th>Mean_Humidity</th>\n",
              "      <th>...</th>\n",
              "      <th>Max_VisibilityKm</th>\n",
              "      <th>Mean_VisibilityKm</th>\n",
              "      <th>Min_VisibilitykM</th>\n",
              "      <th>Max_Wind_SpeedKm_h</th>\n",
              "      <th>Mean_Wind_SpeedKm_h</th>\n",
              "      <th>Max_Gust_SpeedKm_h</th>\n",
              "      <th>Precipitationmm</th>\n",
              "      <th>CloudCover</th>\n",
              "      <th>Events</th>\n",
              "      <th>WindDirDegrees</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NordrheinWestfalen</td>\n",
              "      <td>2013-01-01</td>\n",
              "      <td>8</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>7</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "      <td>94</td>\n",
              "      <td>87</td>\n",
              "      <td>...</td>\n",
              "      <td>31.0</td>\n",
              "      <td>12.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>39</td>\n",
              "      <td>26</td>\n",
              "      <td>58.0</td>\n",
              "      <td>5.08</td>\n",
              "      <td>6.0</td>\n",
              "      <td>Rain</td>\n",
              "      <td>215</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>NordrheinWestfalen</td>\n",
              "      <td>2013-01-02</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>93</td>\n",
              "      <td>85</td>\n",
              "      <td>...</td>\n",
              "      <td>31.0</td>\n",
              "      <td>14.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>24</td>\n",
              "      <td>16</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.00</td>\n",
              "      <td>6.0</td>\n",
              "      <td>Rain</td>\n",
              "      <td>225</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NordrheinWestfalen</td>\n",
              "      <td>2013-01-03</td>\n",
              "      <td>11</td>\n",
              "      <td>8</td>\n",
              "      <td>6</td>\n",
              "      <td>10</td>\n",
              "      <td>8</td>\n",
              "      <td>4</td>\n",
              "      <td>100</td>\n",
              "      <td>93</td>\n",
              "      <td>...</td>\n",
              "      <td>31.0</td>\n",
              "      <td>8.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>26</td>\n",
              "      <td>21</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.02</td>\n",
              "      <td>7.0</td>\n",
              "      <td>Rain</td>\n",
              "      <td>240</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>NordrheinWestfalen</td>\n",
              "      <td>2013-01-04</td>\n",
              "      <td>9</td>\n",
              "      <td>9</td>\n",
              "      <td>8</td>\n",
              "      <td>9</td>\n",
              "      <td>9</td>\n",
              "      <td>8</td>\n",
              "      <td>100</td>\n",
              "      <td>94</td>\n",
              "      <td>...</td>\n",
              "      <td>11.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>23</td>\n",
              "      <td>14</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.25</td>\n",
              "      <td>7.0</td>\n",
              "      <td>Rain</td>\n",
              "      <td>263</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>NordrheinWestfalen</td>\n",
              "      <td>2013-01-05</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>7</td>\n",
              "      <td>8</td>\n",
              "      <td>7</td>\n",
              "      <td>6</td>\n",
              "      <td>100</td>\n",
              "      <td>94</td>\n",
              "      <td>...</td>\n",
              "      <td>10.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>16</td>\n",
              "      <td>10</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.00</td>\n",
              "      <td>7.0</td>\n",
              "      <td>Rain</td>\n",
              "      <td>268</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 24 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                 file        Date  Max_TemperatureC  Mean_TemperatureC  \\\n",
              "0  NordrheinWestfalen  2013-01-01                 8                  4   \n",
              "1  NordrheinWestfalen  2013-01-02                 7                  4   \n",
              "2  NordrheinWestfalen  2013-01-03                11                  8   \n",
              "3  NordrheinWestfalen  2013-01-04                 9                  9   \n",
              "4  NordrheinWestfalen  2013-01-05                 8                  8   \n",
              "\n",
              "   Min_TemperatureC  Dew_PointC  MeanDew_PointC  Min_DewpointC  Max_Humidity  \\\n",
              "0                 2           7               5              1            94   \n",
              "1                 1           5               3              2            93   \n",
              "2                 6          10               8              4           100   \n",
              "3                 8           9               9              8           100   \n",
              "4                 7           8               7              6           100   \n",
              "\n",
              "   Mean_Humidity       ...        Max_VisibilityKm  Mean_VisibilityKm  \\\n",
              "0             87       ...                    31.0               12.0   \n",
              "1             85       ...                    31.0               14.0   \n",
              "2             93       ...                    31.0                8.0   \n",
              "3             94       ...                    11.0                5.0   \n",
              "4             94       ...                    10.0                6.0   \n",
              "\n",
              "   Min_VisibilitykM  Max_Wind_SpeedKm_h  Mean_Wind_SpeedKm_h  \\\n",
              "0               4.0                  39                   26   \n",
              "1              10.0                  24                   16   \n",
              "2               2.0                  26                   21   \n",
              "3               2.0                  23                   14   \n",
              "4               3.0                  16                   10   \n",
              "\n",
              "   Max_Gust_SpeedKm_h  Precipitationmm  CloudCover  Events  WindDirDegrees  \n",
              "0                58.0             5.08         6.0    Rain             215  \n",
              "1                 NaN             0.00         6.0    Rain             225  \n",
              "2                 NaN             1.02         7.0    Rain             240  \n",
              "3                 NaN             0.25         7.0    Rain             263  \n",
              "4                 NaN             0.00         7.0    Rain             268  \n",
              "\n",
              "[5 rows x 24 columns]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Date</th>\n",
              "      <th>Open</th>\n",
              "      <th>Promo</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>8</td>\n",
              "      <td>4</td>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>9</td>\n",
              "      <td>4</td>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Id  Store  DayOfWeek        Date  Open  Promo StateHoliday  SchoolHoliday\n",
              "0   1      1          4  2015-09-17   1.0      1            0              0\n",
              "1   2      3          4  2015-09-17   1.0      1            0              0\n",
              "2   3      7          4  2015-09-17   1.0      1            0              0\n",
              "3   4      8          4  2015-09-17   1.0      1            0              0\n",
              "4   5      9          4  2015-09-17   1.0      1            0              0"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "KcUT4wUKm0bl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 2987
        },
        "outputId": "767bcc59-e9f3-4c27-8a9e-d5b51df55914"
      },
      "cell_type": "code",
      "source": [
        "# Сводная информация по таблицам\n",
        "for t in tables: display(DataFrameSummary(t).summary())"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Date</th>\n",
              "      <th>Sales</th>\n",
              "      <th>Customers</th>\n",
              "      <th>Open</th>\n",
              "      <th>Promo</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>1.01721e+06</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1.01721e+06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>558.43</td>\n",
              "      <td>3.99834</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5773.82</td>\n",
              "      <td>633.146</td>\n",
              "      <td>0.830107</td>\n",
              "      <td>0.381515</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.178647</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>321.909</td>\n",
              "      <td>1.99739</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3849.93</td>\n",
              "      <td>464.412</td>\n",
              "      <td>0.375539</td>\n",
              "      <td>0.485759</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.383056</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>280</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3727</td>\n",
              "      <td>405</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>558</td>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5744</td>\n",
              "      <td>609</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>838</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7856</td>\n",
              "      <td>837</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1115</td>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>41551</td>\n",
              "      <td>7388</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "      <td>1017209</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>1115</td>\n",
              "      <td>7</td>\n",
              "      <td>942</td>\n",
              "      <td>21734</td>\n",
              "      <td>4086</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>bool</td>\n",
              "      <td>bool</td>\n",
              "      <td>categorical</td>\n",
              "      <td>bool</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                    Store    DayOfWeek         Date        Sales    Customers  \\\n",
              "count         1.01721e+06  1.01721e+06          NaN  1.01721e+06  1.01721e+06   \n",
              "mean               558.43      3.99834          NaN      5773.82      633.146   \n",
              "std               321.909      1.99739          NaN      3849.93      464.412   \n",
              "min                     1            1          NaN            0            0   \n",
              "25%                   280            2          NaN         3727          405   \n",
              "50%                   558            4          NaN         5744          609   \n",
              "75%                   838            6          NaN         7856          837   \n",
              "max                  1115            7          NaN        41551         7388   \n",
              "counts            1017209      1017209      1017209      1017209      1017209   \n",
              "uniques              1115            7          942        21734         4086   \n",
              "missing                 0            0            0            0            0   \n",
              "missing_perc           0%           0%           0%           0%           0%   \n",
              "types             numeric      numeric  categorical      numeric      numeric   \n",
              "\n",
              "                     Open        Promo StateHoliday SchoolHoliday  \n",
              "count         1.01721e+06  1.01721e+06          NaN   1.01721e+06  \n",
              "mean             0.830107     0.381515          NaN      0.178647  \n",
              "std              0.375539     0.485759          NaN      0.383056  \n",
              "min                     0            0          NaN             0  \n",
              "25%                     1            0          NaN             0  \n",
              "50%                     1            0          NaN             0  \n",
              "75%                     1            1          NaN             0  \n",
              "max                     1            1          NaN             1  \n",
              "counts            1017209      1017209      1017209       1017209  \n",
              "uniques                 2            2            4             2  \n",
              "missing                 0            0            0             0  \n",
              "missing_perc           0%           0%           0%            0%  \n",
              "types                bool         bool  categorical          bool  "
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>StoreType</th>\n",
              "      <th>Assortment</th>\n",
              "      <th>CompetitionDistance</th>\n",
              "      <th>CompetitionOpenSinceMonth</th>\n",
              "      <th>CompetitionOpenSinceYear</th>\n",
              "      <th>Promo2</th>\n",
              "      <th>Promo2SinceWeek</th>\n",
              "      <th>Promo2SinceYear</th>\n",
              "      <th>PromoInterval</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1115</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1112</td>\n",
              "      <td>761</td>\n",
              "      <td>761</td>\n",
              "      <td>1115</td>\n",
              "      <td>571</td>\n",
              "      <td>571</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>558</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>5404.9</td>\n",
              "      <td>7.2247</td>\n",
              "      <td>2008.67</td>\n",
              "      <td>0.512108</td>\n",
              "      <td>23.5954</td>\n",
              "      <td>2011.76</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>322.017</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>7663.17</td>\n",
              "      <td>3.21235</td>\n",
              "      <td>6.19598</td>\n",
              "      <td>0.500078</td>\n",
              "      <td>14.142</td>\n",
              "      <td>1.67494</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>20</td>\n",
              "      <td>1</td>\n",
              "      <td>1900</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2009</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>279.5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>717.5</td>\n",
              "      <td>4</td>\n",
              "      <td>2006</td>\n",
              "      <td>0</td>\n",
              "      <td>13</td>\n",
              "      <td>2011</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>558</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2325</td>\n",
              "      <td>8</td>\n",
              "      <td>2010</td>\n",
              "      <td>1</td>\n",
              "      <td>22</td>\n",
              "      <td>2012</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>836.5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6882.5</td>\n",
              "      <td>10</td>\n",
              "      <td>2013</td>\n",
              "      <td>1</td>\n",
              "      <td>37</td>\n",
              "      <td>2013</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1115</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>75860</td>\n",
              "      <td>12</td>\n",
              "      <td>2015</td>\n",
              "      <td>1</td>\n",
              "      <td>50</td>\n",
              "      <td>2015</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>1115</td>\n",
              "      <td>1115</td>\n",
              "      <td>1115</td>\n",
              "      <td>1112</td>\n",
              "      <td>761</td>\n",
              "      <td>761</td>\n",
              "      <td>1115</td>\n",
              "      <td>571</td>\n",
              "      <td>571</td>\n",
              "      <td>571</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>1115</td>\n",
              "      <td>4</td>\n",
              "      <td>3</td>\n",
              "      <td>654</td>\n",
              "      <td>12</td>\n",
              "      <td>23</td>\n",
              "      <td>2</td>\n",
              "      <td>24</td>\n",
              "      <td>7</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>354</td>\n",
              "      <td>354</td>\n",
              "      <td>0</td>\n",
              "      <td>544</td>\n",
              "      <td>544</td>\n",
              "      <td>544</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0.27%</td>\n",
              "      <td>31.75%</td>\n",
              "      <td>31.75%</td>\n",
              "      <td>0%</td>\n",
              "      <td>48.79%</td>\n",
              "      <td>48.79%</td>\n",
              "      <td>48.79%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "      <td>categorical</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>bool</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                Store    StoreType   Assortment CompetitionDistance  \\\n",
              "count            1115          NaN          NaN                1112   \n",
              "mean              558          NaN          NaN              5404.9   \n",
              "std           322.017          NaN          NaN             7663.17   \n",
              "min                 1          NaN          NaN                  20   \n",
              "25%             279.5          NaN          NaN               717.5   \n",
              "50%               558          NaN          NaN                2325   \n",
              "75%             836.5          NaN          NaN              6882.5   \n",
              "max              1115          NaN          NaN               75860   \n",
              "counts           1115         1115         1115                1112   \n",
              "uniques          1115            4            3                 654   \n",
              "missing             0            0            0                   3   \n",
              "missing_perc       0%           0%           0%               0.27%   \n",
              "types         numeric  categorical  categorical             numeric   \n",
              "\n",
              "             CompetitionOpenSinceMonth CompetitionOpenSinceYear    Promo2  \\\n",
              "count                              761                      761      1115   \n",
              "mean                            7.2247                  2008.67  0.512108   \n",
              "std                            3.21235                  6.19598  0.500078   \n",
              "min                                  1                     1900         0   \n",
              "25%                                  4                     2006         0   \n",
              "50%                                  8                     2010         1   \n",
              "75%                                 10                     2013         1   \n",
              "max                                 12                     2015         1   \n",
              "counts                             761                      761      1115   \n",
              "uniques                             12                       23         2   \n",
              "missing                            354                      354         0   \n",
              "missing_perc                    31.75%                   31.75%        0%   \n",
              "types                          numeric                  numeric      bool   \n",
              "\n",
              "             Promo2SinceWeek Promo2SinceYear PromoInterval  \n",
              "count                    571             571           NaN  \n",
              "mean                 23.5954         2011.76           NaN  \n",
              "std                   14.142         1.67494           NaN  \n",
              "min                        1            2009           NaN  \n",
              "25%                       13            2011           NaN  \n",
              "50%                       22            2012           NaN  \n",
              "75%                       37            2013           NaN  \n",
              "max                       50            2015           NaN  \n",
              "counts                   571             571           571  \n",
              "uniques                   24               7             3  \n",
              "missing                  544             544           544  \n",
              "missing_perc          48.79%          48.79%        48.79%  \n",
              "types                numeric         numeric   categorical  "
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>State</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1115</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>558</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>322.017</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>279.5</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>558</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>836.5</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>1115</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>1115</td>\n",
              "      <td>1115</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>1115</td>\n",
              "      <td>12</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                Store        State\n",
              "count            1115          NaN\n",
              "mean              558          NaN\n",
              "std           322.017          NaN\n",
              "min                 1          NaN\n",
              "25%             279.5          NaN\n",
              "50%               558          NaN\n",
              "75%             836.5          NaN\n",
              "max              1115          NaN\n",
              "counts           1115         1115\n",
              "uniques          1115           12\n",
              "missing             0            0\n",
              "missing_perc       0%           0%\n",
              "types         numeric  categorical"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>StateName</th>\n",
              "      <th>State</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>16</td>\n",
              "      <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>unique</th>\n",
              "      <td>16</td>\n",
              "      <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>top</th>\n",
              "      <td>RheinlandPfalz</td>\n",
              "      <td>HB</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>freq</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>16</td>\n",
              "      <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>16</td>\n",
              "      <td>16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>unique</td>\n",
              "      <td>unique</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   StateName   State\n",
              "count                     16      16\n",
              "unique                    16      16\n",
              "top           RheinlandPfalz      HB\n",
              "freq                       1       1\n",
              "counts                    16      16\n",
              "uniques                   16      16\n",
              "missing                    0       0\n",
              "missing_perc              0%      0%\n",
              "types                 unique  unique"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file</th>\n",
              "      <th>week</th>\n",
              "      <th>trend</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2072</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>63.8142</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>12.6502</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>55</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>72</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>100</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>2072</td>\n",
              "      <td>2072</td>\n",
              "      <td>2072</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>14</td>\n",
              "      <td>148</td>\n",
              "      <td>68</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>categorical</td>\n",
              "      <td>categorical</td>\n",
              "      <td>numeric</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                     file         week    trend\n",
              "count                 NaN          NaN     2072\n",
              "mean                  NaN          NaN  63.8142\n",
              "std                   NaN          NaN  12.6502\n",
              "min                   NaN          NaN        0\n",
              "25%                   NaN          NaN       55\n",
              "50%                   NaN          NaN       64\n",
              "75%                   NaN          NaN       72\n",
              "max                   NaN          NaN      100\n",
              "counts               2072         2072     2072\n",
              "uniques                14          148       68\n",
              "missing                 0            0        0\n",
              "missing_perc           0%           0%       0%\n",
              "types         categorical  categorical  numeric"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file</th>\n",
              "      <th>Date</th>\n",
              "      <th>Max_TemperatureC</th>\n",
              "      <th>Mean_TemperatureC</th>\n",
              "      <th>Min_TemperatureC</th>\n",
              "      <th>Dew_PointC</th>\n",
              "      <th>MeanDew_PointC</th>\n",
              "      <th>Min_DewpointC</th>\n",
              "      <th>Max_Humidity</th>\n",
              "      <th>Mean_Humidity</th>\n",
              "      <th>...</th>\n",
              "      <th>Max_VisibilityKm</th>\n",
              "      <th>Mean_VisibilityKm</th>\n",
              "      <th>Min_VisibilitykM</th>\n",
              "      <th>Max_Wind_SpeedKm_h</th>\n",
              "      <th>Mean_Wind_SpeedKm_h</th>\n",
              "      <th>Max_Gust_SpeedKm_h</th>\n",
              "      <th>Precipitationmm</th>\n",
              "      <th>CloudCover</th>\n",
              "      <th>Events</th>\n",
              "      <th>WindDirDegrees</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>...</td>\n",
              "      <td>15459</td>\n",
              "      <td>15459</td>\n",
              "      <td>15459</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>3604</td>\n",
              "      <td>15840</td>\n",
              "      <td>14667</td>\n",
              "      <td>NaN</td>\n",
              "      <td>15840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>14.6441</td>\n",
              "      <td>10.389</td>\n",
              "      <td>6.19899</td>\n",
              "      <td>8.58782</td>\n",
              "      <td>6.20581</td>\n",
              "      <td>3.62614</td>\n",
              "      <td>93.6596</td>\n",
              "      <td>74.2829</td>\n",
              "      <td>...</td>\n",
              "      <td>24.0576</td>\n",
              "      <td>12.2398</td>\n",
              "      <td>7.02516</td>\n",
              "      <td>22.7666</td>\n",
              "      <td>11.9722</td>\n",
              "      <td>48.8643</td>\n",
              "      <td>0.831718</td>\n",
              "      <td>5.55131</td>\n",
              "      <td>NaN</td>\n",
              "      <td>175.897</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8.64601</td>\n",
              "      <td>7.37926</td>\n",
              "      <td>6.52639</td>\n",
              "      <td>6.24478</td>\n",
              "      <td>6.08677</td>\n",
              "      <td>6.12839</td>\n",
              "      <td>7.67853</td>\n",
              "      <td>13.4866</td>\n",
              "      <td>...</td>\n",
              "      <td>8.9768</td>\n",
              "      <td>5.06794</td>\n",
              "      <td>4.9806</td>\n",
              "      <td>8.98862</td>\n",
              "      <td>5.87284</td>\n",
              "      <td>13.027</td>\n",
              "      <td>2.51351</td>\n",
              "      <td>1.68771</td>\n",
              "      <td>NaN</td>\n",
              "      <td>101.589</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-11</td>\n",
              "      <td>-13</td>\n",
              "      <td>-15</td>\n",
              "      <td>-14</td>\n",
              "      <td>-15</td>\n",
              "      <td>-73</td>\n",
              "      <td>44</td>\n",
              "      <td>30</td>\n",
              "      <td>...</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>3</td>\n",
              "      <td>2</td>\n",
              "      <td>21</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>-1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>2</td>\n",
              "      <td>-1</td>\n",
              "      <td>90.75</td>\n",
              "      <td>65</td>\n",
              "      <td>...</td>\n",
              "      <td>14</td>\n",
              "      <td>10</td>\n",
              "      <td>3</td>\n",
              "      <td>16</td>\n",
              "      <td>8</td>\n",
              "      <td>39</td>\n",
              "      <td>0</td>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>80</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>15</td>\n",
              "      <td>11</td>\n",
              "      <td>7</td>\n",
              "      <td>9</td>\n",
              "      <td>7</td>\n",
              "      <td>4</td>\n",
              "      <td>94</td>\n",
              "      <td>76</td>\n",
              "      <td>...</td>\n",
              "      <td>31</td>\n",
              "      <td>11</td>\n",
              "      <td>7</td>\n",
              "      <td>21</td>\n",
              "      <td>11</td>\n",
              "      <td>48</td>\n",
              "      <td>0</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>202</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>21</td>\n",
              "      <td>16</td>\n",
              "      <td>11</td>\n",
              "      <td>13</td>\n",
              "      <td>11</td>\n",
              "      <td>8</td>\n",
              "      <td>100</td>\n",
              "      <td>85</td>\n",
              "      <td>...</td>\n",
              "      <td>31</td>\n",
              "      <td>14</td>\n",
              "      <td>10</td>\n",
              "      <td>27</td>\n",
              "      <td>14</td>\n",
              "      <td>55</td>\n",
              "      <td>0.25</td>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>256</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>39</td>\n",
              "      <td>31</td>\n",
              "      <td>24</td>\n",
              "      <td>25</td>\n",
              "      <td>20</td>\n",
              "      <td>19</td>\n",
              "      <td>100</td>\n",
              "      <td>100</td>\n",
              "      <td>...</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>101</td>\n",
              "      <td>53</td>\n",
              "      <td>111</td>\n",
              "      <td>58.93</td>\n",
              "      <td>8</td>\n",
              "      <td>NaN</td>\n",
              "      <td>360</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>...</td>\n",
              "      <td>15459</td>\n",
              "      <td>15459</td>\n",
              "      <td>15459</td>\n",
              "      <td>15840</td>\n",
              "      <td>15840</td>\n",
              "      <td>3604</td>\n",
              "      <td>15840</td>\n",
              "      <td>14667</td>\n",
              "      <td>11889</td>\n",
              "      <td>15840</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>16</td>\n",
              "      <td>990</td>\n",
              "      <td>51</td>\n",
              "      <td>45</td>\n",
              "      <td>40</td>\n",
              "      <td>40</td>\n",
              "      <td>36</td>\n",
              "      <td>40</td>\n",
              "      <td>53</td>\n",
              "      <td>71</td>\n",
              "      <td>...</td>\n",
              "      <td>24</td>\n",
              "      <td>32</td>\n",
              "      <td>24</td>\n",
              "      <td>44</td>\n",
              "      <td>29</td>\n",
              "      <td>47</td>\n",
              "      <td>41</td>\n",
              "      <td>9</td>\n",
              "      <td>21</td>\n",
              "      <td>362</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>...</td>\n",
              "      <td>381</td>\n",
              "      <td>381</td>\n",
              "      <td>381</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>12236</td>\n",
              "      <td>0</td>\n",
              "      <td>1173</td>\n",
              "      <td>3951</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>...</td>\n",
              "      <td>2.41%</td>\n",
              "      <td>2.41%</td>\n",
              "      <td>2.41%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>77.25%</td>\n",
              "      <td>0%</td>\n",
              "      <td>7.41%</td>\n",
              "      <td>24.94%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>categorical</td>\n",
              "      <td>categorical</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>...</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "      <td>numeric</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>13 rows × 24 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                     file         Date Max_TemperatureC Mean_TemperatureC  \\\n",
              "count                 NaN          NaN            15840             15840   \n",
              "mean                  NaN          NaN          14.6441            10.389   \n",
              "std                   NaN          NaN          8.64601           7.37926   \n",
              "min                   NaN          NaN              -11               -13   \n",
              "25%                   NaN          NaN                8                 4   \n",
              "50%                   NaN          NaN               15                11   \n",
              "75%                   NaN          NaN               21                16   \n",
              "max                   NaN          NaN               39                31   \n",
              "counts              15840        15840            15840             15840   \n",
              "uniques                16          990               51                45   \n",
              "missing                 0            0                0                 0   \n",
              "missing_perc           0%           0%               0%                0%   \n",
              "types         categorical  categorical          numeric           numeric   \n",
              "\n",
              "             Min_TemperatureC Dew_PointC MeanDew_PointC Min_DewpointC  \\\n",
              "count                   15840      15840          15840         15840   \n",
              "mean                  6.19899    8.58782        6.20581       3.62614   \n",
              "std                   6.52639    6.24478        6.08677       6.12839   \n",
              "min                       -15        -14            -15           -73   \n",
              "25%                         1          4              2            -1   \n",
              "50%                         7          9              7             4   \n",
              "75%                        11         13             11             8   \n",
              "max                        24         25             20            19   \n",
              "counts                  15840      15840          15840         15840   \n",
              "uniques                    40         40             36            40   \n",
              "missing                     0          0              0             0   \n",
              "missing_perc               0%         0%             0%            0%   \n",
              "types                 numeric    numeric        numeric       numeric   \n",
              "\n",
              "             Max_Humidity Mean_Humidity      ...       Max_VisibilityKm  \\\n",
              "count               15840         15840      ...                  15459   \n",
              "mean              93.6596       74.2829      ...                24.0576   \n",
              "std               7.67853       13.4866      ...                 8.9768   \n",
              "min                    44            30      ...                      0   \n",
              "25%                 90.75            65      ...                     14   \n",
              "50%                    94            76      ...                     31   \n",
              "75%                   100            85      ...                     31   \n",
              "max                   100           100      ...                     31   \n",
              "counts              15840         15840      ...                  15459   \n",
              "uniques                53            71      ...                     24   \n",
              "missing                 0             0      ...                    381   \n",
              "missing_perc           0%            0%      ...                  2.41%   \n",
              "types             numeric       numeric      ...                numeric   \n",
              "\n",
              "             Mean_VisibilityKm Min_VisibilitykM Max_Wind_SpeedKm_h  \\\n",
              "count                    15459            15459              15840   \n",
              "mean                   12.2398          7.02516            22.7666   \n",
              "std                    5.06794           4.9806            8.98862   \n",
              "min                          0                0                  3   \n",
              "25%                         10                3                 16   \n",
              "50%                         11                7                 21   \n",
              "75%                         14               10                 27   \n",
              "max                         31               31                101   \n",
              "counts                   15459            15459              15840   \n",
              "uniques                     32               24                 44   \n",
              "missing                    381              381                  0   \n",
              "missing_perc             2.41%            2.41%                 0%   \n",
              "types                  numeric          numeric            numeric   \n",
              "\n",
              "             Mean_Wind_SpeedKm_h Max_Gust_SpeedKm_h Precipitationmm  \\\n",
              "count                      15840               3604           15840   \n",
              "mean                     11.9722            48.8643        0.831718   \n",
              "std                      5.87284             13.027         2.51351   \n",
              "min                            2                 21               0   \n",
              "25%                            8                 39               0   \n",
              "50%                           11                 48               0   \n",
              "75%                           14                 55            0.25   \n",
              "max                           53                111           58.93   \n",
              "counts                     15840               3604           15840   \n",
              "uniques                       29                 47              41   \n",
              "missing                        0              12236               0   \n",
              "missing_perc                  0%             77.25%              0%   \n",
              "types                    numeric            numeric         numeric   \n",
              "\n",
              "             CloudCover       Events WindDirDegrees  \n",
              "count             14667          NaN          15840  \n",
              "mean            5.55131          NaN        175.897  \n",
              "std             1.68771          NaN        101.589  \n",
              "min                   0          NaN             -1  \n",
              "25%                   5          NaN             80  \n",
              "50%                   6          NaN            202  \n",
              "75%                   7          NaN            256  \n",
              "max                   8          NaN            360  \n",
              "counts            14667        11889          15840  \n",
              "uniques               9           21            362  \n",
              "missing            1173         3951              0  \n",
              "missing_perc      7.41%       24.94%             0%  \n",
              "types           numeric  categorical        numeric  \n",
              "\n",
              "[13 rows x 24 columns]"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Date</th>\n",
              "      <th>Open</th>\n",
              "      <th>Promo</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>NaN</td>\n",
              "      <td>41077</td>\n",
              "      <td>41088</td>\n",
              "      <td>NaN</td>\n",
              "      <td>41088</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>20544.5</td>\n",
              "      <td>555.9</td>\n",
              "      <td>3.97917</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.854322</td>\n",
              "      <td>0.395833</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.443487</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>11861.2</td>\n",
              "      <td>320.274</td>\n",
              "      <td>2.01548</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.352787</td>\n",
              "      <td>0.489035</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.496802</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>10272.8</td>\n",
              "      <td>279.75</td>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>20544.5</td>\n",
              "      <td>553.5</td>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>30816.2</td>\n",
              "      <td>832.25</td>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>41088</td>\n",
              "      <td>1115</td>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>41077</td>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "      <td>41088</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>uniques</th>\n",
              "      <td>41088</td>\n",
              "      <td>856</td>\n",
              "      <td>7</td>\n",
              "      <td>48</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>missing_perc</th>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0.03%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "      <td>0%</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>types</th>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>numeric</td>\n",
              "      <td>categorical</td>\n",
              "      <td>bool</td>\n",
              "      <td>bool</td>\n",
              "      <td>bool</td>\n",
              "      <td>bool</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   Id    Store DayOfWeek         Date      Open     Promo  \\\n",
              "count           41088    41088     41088          NaN     41077     41088   \n",
              "mean          20544.5    555.9   3.97917          NaN  0.854322  0.395833   \n",
              "std           11861.2  320.274   2.01548          NaN  0.352787  0.489035   \n",
              "min                 1        1         1          NaN         0         0   \n",
              "25%           10272.8   279.75         2          NaN         1         0   \n",
              "50%           20544.5    553.5         4          NaN         1         0   \n",
              "75%           30816.2   832.25         6          NaN         1         1   \n",
              "max             41088     1115         7          NaN         1         1   \n",
              "counts          41088    41088     41088        41088     41077     41088   \n",
              "uniques         41088      856         7           48         2         2   \n",
              "missing             0        0         0            0        11         0   \n",
              "missing_perc       0%       0%        0%           0%     0.03%        0%   \n",
              "types         numeric  numeric   numeric  categorical      bool      bool   \n",
              "\n",
              "             StateHoliday SchoolHoliday  \n",
              "count                 NaN         41088  \n",
              "mean                  NaN      0.443487  \n",
              "std                   NaN      0.496802  \n",
              "min                   NaN             0  \n",
              "25%                   NaN             0  \n",
              "50%                   NaN             0  \n",
              "75%                   NaN             1  \n",
              "max                   NaN             1  \n",
              "counts              41088         41088  \n",
              "uniques                 2             2  \n",
              "missing                 0             0  \n",
              "missing_perc           0%            0%  \n",
              "types                bool          bool  "
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "z6-SaD59orCM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Очистка данных"
      ]
    },
    {
      "metadata": {
        "id": "Hh-GiCbonNyf",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6bb73f45-26a3-47c3-9a7f-bd07e414831e"
      },
      "cell_type": "code",
      "source": [
        "train, store, store_states, state_names, googletrend, weather, test = tables\n",
        "len(train),len(test)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(1017209, 41088)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "metadata": {
        "id": "DtnMvAtio1NR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e57fa495-4954-4de6-a9e9-0db8d63818a1"
      },
      "cell_type": "code",
      "source": [
        "# Делаем государственные праздники булевыми значениями\n",
        "train.StateHoliday = train.StateHoliday!='0'\n",
        "test.StateHoliday = test.StateHoliday!='0'\n",
        "test.StateHoliday[0]"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "metadata": {
        "id": "Anry4MwLpNfQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Функция для объединения таблиц по определенным полям\n",
        "def join_df(left, right, left_on, right_on=None, suffix='_y'):\n",
        "    if right_on is None: right_on = left_on\n",
        "    return left.merge(right, how='left', left_on=left_on, right_on=right_on, suffixes=(\"\", suffix))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OW3NPKSipyz1",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Присоединяем state_names к weather\n",
        "weather = join_df(weather, state_names, \"file\", \"StateName\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NlexbT50p75F",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Преобразуем googletrend\n",
        "googletrend['Date'] = googletrend.week.str.split(' - ', expand=True)[0]\n",
        "googletrend['State'] = googletrend.file.str.split('_', expand=True)[2]\n",
        "googletrend.loc[googletrend.State=='NI', \"State\"] = 'HB,NI'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "sTv4wLwprDp3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Расширяем информацию о датах (всё что знает pandas о днях)\n",
        "# Это функция библиотеки fastai (drop=False - не удалять исходную дату)\n",
        "add_datepart(weather, \"Date\", drop=False)\n",
        "add_datepart(googletrend, \"Date\", drop=False)\n",
        "add_datepart(train, \"Date\", drop=False)\n",
        "add_datepart(test, \"Date\", drop=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EG2bWex2rnr4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Вытащим \"Rossmann_DE\" из googletrend\n",
        "trend_de = googletrend[googletrend.file == 'Rossmann_DE']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "v9tr-hrBtPRs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Объединим данные в итоговые обучающую и тестовую таблицы"
      ]
    },
    {
      "metadata": {
        "id": "oMWINc8psdPg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1b3fd9e3-1f83-4c83-ddf7-0db384d8193e"
      },
      "cell_type": "code",
      "source": [
        "\n",
        "store = join_df(store, store_states, \"Store\")\n",
        "len(store[store.State.isnull()])"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "metadata": {
        "id": "ijShlO7ds6aM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "82403076-3be9-45e8-8aaa-bb550de63aeb"
      },
      "cell_type": "code",
      "source": [
        "joined = join_df(train, store, \"Store\")\n",
        "joined_test = join_df(test, store, \"Store\")\n",
        "len(joined[joined.StoreType.isnull()]),len(joined_test[joined_test.StoreType.isnull()])"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "metadata": {
        "id": "9A24ONButUON",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "4feae5f4-bcc3-43d5-b60c-1d4865c3de9c"
      },
      "cell_type": "code",
      "source": [
        "joined = join_df(joined, googletrend, [\"State\",\"Year\", \"Week\"])\n",
        "joined_test = join_df(joined_test, googletrend, [\"State\",\"Year\", \"Week\"])\n",
        "len(joined[joined.trend.isnull()]),len(joined_test[joined_test.trend.isnull()])"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "metadata": {
        "id": "fYUjoNMitXnw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "bb055417-3322-445c-8a54-d8d903e1f8fa"
      },
      "cell_type": "code",
      "source": [
        "joined = joined.merge(trend_de, 'left', [\"Year\", \"Week\"], suffixes=('', '_DE'))\n",
        "joined_test = joined_test.merge(trend_de, 'left', [\"Year\", \"Week\"], suffixes=('', '_DE'))\n",
        "len(joined[joined.trend_DE.isnull()]),len(joined_test[joined_test.trend_DE.isnull()])"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "metadata": {
        "id": "gchAkrGitbA1",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d458935f-2baf-4008-b204-e8b926998b91"
      },
      "cell_type": "code",
      "source": [
        "joined = join_df(joined, weather, [\"State\",\"Date\"])\n",
        "joined_test = join_df(joined_test, weather, [\"State\",\"Date\"])\n",
        "len(joined[joined.Mean_TemperatureC.isnull()]),len(joined_test[joined_test.Mean_TemperatureC.isnull()])"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0, 0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "metadata": {
        "id": "2Oxs0vtNte_T",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined, joined_test):\n",
        "    for c in df.columns:\n",
        "        if c.endswith('_y'):\n",
        "            if c in df.columns: df.drop(c, inplace=True, axis=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dkFKrGXOt0Qc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df['CompetitionOpenSinceYear'] = df.CompetitionOpenSinceYear.fillna(1900).astype(np.int32)\n",
        "    df['CompetitionOpenSinceMonth'] = df.CompetitionOpenSinceMonth.fillna(1).astype(np.int32)\n",
        "    df['Promo2SinceYear'] = df.Promo2SinceYear.fillna(1900).astype(np.int32)\n",
        "    df['Promo2SinceWeek'] = df.Promo2SinceWeek.fillna(1).astype(np.int32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cS5MM2ARt6BF",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df[\"CompetitionOpenSince\"] = pd.to_datetime(dict(year=df.CompetitionOpenSinceYear, month=df.CompetitionOpenSinceMonth, day=15))\n",
        "    df[\"CompetitionDaysOpen\"] = df.Date.subtract(df.CompetitionOpenSince).dt.days"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HxeFcNzduFA8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df.loc[df.CompetitionDaysOpen<0, \"CompetitionDaysOpen\"] = 0\n",
        "    df.loc[df.CompetitionOpenSinceYear<1990, \"CompetitionDaysOpen\"] = 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CNCRsadHuM_S",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "11edc076-8264-453a-c894-ca435271ee9a"
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df[\"CompetitionMonthsOpen\"] = df[\"CompetitionDaysOpen\"]//30\n",
        "    df.loc[df.CompetitionMonthsOpen>24, \"CompetitionMonthsOpen\"] = 24\n",
        "joined.CompetitionMonthsOpen.unique()"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([24,  3, 19,  9,  0, 16, 17,  7, 15, 22, 11, 13,  2, 23, 12,  4, 10,  1, 14, 20,  8, 18,  6, 21,  5])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "metadata": {
        "id": "3WbVlKH3uT0V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df[\"Promo2Since\"] = pd.to_datetime(df.apply(lambda x: Week(x.Promo2SinceYear, x.Promo2SinceWeek).monday(), axis=1).astype(pd.datetime))\n",
        "    df[\"Promo2Days\"] = df.Date.subtract(df[\"Promo2Since\"]).dt.days"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6ikPFF6CuajJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for df in (joined,joined_test):\n",
        "    df.loc[df.Promo2Days<0, \"Promo2Days\"] = 0\n",
        "    df.loc[df.Promo2SinceYear<1990, \"Promo2Days\"] = 0\n",
        "    df[\"Promo2Weeks\"] = df[\"Promo2Days\"]//7\n",
        "    df.loc[df.Promo2Weeks<0, \"Promo2Weeks\"] = 0\n",
        "    df.loc[df.Promo2Weeks>25, \"Promo2Weeks\"] = 25\n",
        "    df.Promo2Weeks.unique()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ss9QT5E1vDLv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined.to_feather(f'{PATH}joined')\n",
        "joined_test.to_feather(f'{PATH}joined_test')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "poUoE-32vGxS",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def get_elapsed(fld, pre):\n",
        "    day1 = np.timedelta64(1, 'D')\n",
        "    last_date = np.datetime64()\n",
        "    last_store = 0\n",
        "    res = []\n",
        "\n",
        "    for s,v,d in zip(df.Store.values,df[fld].values, df.Date.values):\n",
        "        if s != last_store:\n",
        "            last_date = np.datetime64()\n",
        "            last_store = s\n",
        "        if v: last_date = d\n",
        "        res.append(((d-last_date).astype('timedelta64[D]') / day1))\n",
        "    df[pre+fld] = res"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YWKbSuU-vplH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "columns = [\"Date\", \"Store\", \"Promo\", \"StateHoliday\", \"SchoolHoliday\"]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YmzupWBKvw2N",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#df = train[columns]\n",
        "df = train[columns].append(test[columns])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jNfgTfOcvzxU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "fld = 'SchoolHoliday'\n",
        "df = df.sort_values(['Store', 'Date'])\n",
        "get_elapsed(fld, 'After')\n",
        "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
        "get_elapsed(fld, 'Before')\n",
        "\n",
        "fld = 'StateHoliday'\n",
        "df = df.sort_values(['Store', 'Date'])\n",
        "get_elapsed(fld, 'After')\n",
        "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
        "get_elapsed(fld, 'Before')\n",
        "\n",
        "fld = 'Promo'\n",
        "df = df.sort_values(['Store', 'Date'])\n",
        "get_elapsed(fld, 'After')\n",
        "df = df.sort_values(['Store', 'Date'], ascending=[True, False])\n",
        "get_elapsed(fld, 'Before')\n",
        "\n",
        "df = df.set_index(\"Date\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bYFFAzPCwNQJ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "columns = ['SchoolHoliday', 'StateHoliday', 'Promo']"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UxtA5xHpwksd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for o in ['Before', 'After']:\n",
        "    for p in columns:\n",
        "        a = o+p\n",
        "        df[a] = df[a].fillna(0).astype(int)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-KEfbXzjwnp8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "bwd = df[['Store']+columns].sort_index().groupby(\"Store\").rolling(7, min_periods=1).sum()\n",
        "fwd = df[['Store']+columns].sort_index(ascending=False).groupby(\"Store\").rolling(7, min_periods=1).sum()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Q5vFOgbjw1yu",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "bwd.drop('Store',1,inplace=True)\n",
        "bwd.reset_index(inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SscZJ1eLxBoV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "fwd.drop('Store',1,inplace=True)\n",
        "fwd.reset_index(inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MV43viuSxFmi",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "df.reset_index(inplace=True)\n",
        "\n",
        "df = df.merge(bwd, 'left', ['Date', 'Store'], suffixes=['', '_bw'])\n",
        "df = df.merge(fwd, 'left', ['Date', 'Store'], suffixes=['', '_fw'])\n",
        "\n",
        "df.drop(columns,1,inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "vs-lMmlaxN5d",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 314
        },
        "outputId": "ca8d1826-1fdd-4b57-8c78-6bc87005cbb4"
      },
      "cell_type": "code",
      "source": [
        "# Посмотрим, что получилось\n",
        "df.head()"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date</th>\n",
              "      <th>Store</th>\n",
              "      <th>AfterSchoolHoliday</th>\n",
              "      <th>BeforeSchoolHoliday</th>\n",
              "      <th>AfterStateHoliday</th>\n",
              "      <th>BeforeStateHoliday</th>\n",
              "      <th>AfterPromo</th>\n",
              "      <th>BeforePromo</th>\n",
              "      <th>SchoolHoliday_bw</th>\n",
              "      <th>StateHoliday_bw</th>\n",
              "      <th>Promo_bw</th>\n",
              "      <th>SchoolHoliday_fw</th>\n",
              "      <th>StateHoliday_fw</th>\n",
              "      <th>Promo_fw</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2015-09-17</td>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>0</td>\n",
              "      <td>105</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2015-09-16</td>\n",
              "      <td>1</td>\n",
              "      <td>12</td>\n",
              "      <td>0</td>\n",
              "      <td>104</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2015-09-15</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "      <td>0</td>\n",
              "      <td>103</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2015-09-14</td>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "      <td>0</td>\n",
              "      <td>102</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2015-09-13</td>\n",
              "      <td>1</td>\n",
              "      <td>9</td>\n",
              "      <td>0</td>\n",
              "      <td>101</td>\n",
              "      <td>0</td>\n",
              "      <td>9</td>\n",
              "      <td>-1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        Date  Store  AfterSchoolHoliday  BeforeSchoolHoliday  \\\n",
              "0 2015-09-17      1                  13                    0   \n",
              "1 2015-09-16      1                  12                    0   \n",
              "2 2015-09-15      1                  11                    0   \n",
              "3 2015-09-14      1                  10                    0   \n",
              "4 2015-09-13      1                   9                    0   \n",
              "\n",
              "   AfterStateHoliday  BeforeStateHoliday  AfterPromo  BeforePromo  \\\n",
              "0                105                   0           0            0   \n",
              "1                104                   0           0            0   \n",
              "2                103                   0           0            0   \n",
              "3                102                   0           0            0   \n",
              "4                101                   0           9           -1   \n",
              "\n",
              "   SchoolHoliday_bw  StateHoliday_bw  Promo_bw  SchoolHoliday_fw  \\\n",
              "0               0.0              0.0       4.0               0.0   \n",
              "1               0.0              0.0       3.0               0.0   \n",
              "2               0.0              0.0       2.0               0.0   \n",
              "3               0.0              0.0       1.0               0.0   \n",
              "4               0.0              0.0       0.0               0.0   \n",
              "\n",
              "   StateHoliday_fw  Promo_fw  \n",
              "0              0.0       1.0  \n",
              "1              0.0       2.0  \n",
              "2              0.0       3.0  \n",
              "3              0.0       4.0  \n",
              "4              0.0       4.0  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "metadata": {
        "id": "_sT1gQ9KxUiP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Сохраним результат\n",
        "df.to_feather(f'{PATH}df')\n",
        "df = pd.read_feather(f'{PATH}df')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cZ_CBM4Nx7py",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "47d417e1-578b-4884-e86f-19de1f8033cb"
      },
      "cell_type": "code",
      "source": [
        "df.columns"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['Date', 'Store', 'AfterSchoolHoliday', 'BeforeSchoolHoliday',\n",
              "       'AfterStateHoliday', 'BeforeStateHoliday', 'AfterPromo', 'BeforePromo',\n",
              "       'SchoolHoliday_bw', 'StateHoliday_bw', 'Promo_bw', 'SchoolHoliday_fw',\n",
              "       'StateHoliday_fw', 'Promo_fw'],\n",
              "      dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "metadata": {
        "id": "z6sJYstWx-ms",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined = join_df(joined, df, ['Store', 'Date'])\n",
        "\n",
        "joined_test = join_df(joined_test, df, ['Store', 'Date'])\n",
        "\n",
        "joined = joined[joined.Sales!=0]\n",
        "\n",
        "joined.reset_index(inplace=True)\n",
        "joined_test.reset_index(inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8HAeO_OqyQHg",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined.to_feather(f'{PATH}joined')\n",
        "joined_test.to_feather(f'{PATH}joined_test')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PKeRuESAzOi7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Преобразование данных для нейросети"
      ]
    },
    {
      "metadata": {
        "id": "53WX6DMFy9pn",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Загрузим таблицы данных\n",
        "joined = pd.read_feather(f'{PATH}joined')\n",
        "joined_test = pd.read_feather(f'{PATH}joined_test')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oTu6T9PIzx7G",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "eb13469f-d9d8-42ad-ba4a-39d83d22a421"
      },
      "cell_type": "code",
      "source": [
        "joined.shape"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(844338, 93)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 74
        }
      ]
    },
    {
      "metadata": {
        "id": "5AEPD1Fez8aa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7f366154-65a1-4ac6-c9e2-44294d5550c5"
      },
      "cell_type": "code",
      "source": [
        "joined_test.shape"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(41088, 92)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "metadata": {
        "id": "7_3tVxslzQhO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1289
        },
        "outputId": "6017f9d3-91c1-4619-b7bd-383df5f68e22"
      },
      "cell_type": "code",
      "source": [
        "# Часть обучающей таблицы\n",
        "joined.head().T.head(40)"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "      <th>2</th>\n",
              "      <th>3</th>\n",
              "      <th>4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>index</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Store</th>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>3</td>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DayOfWeek</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <td>2015-07-31 00:00:00</td>\n",
              "      <td>2015-07-31 00:00:00</td>\n",
              "      <td>2015-07-31 00:00:00</td>\n",
              "      <td>2015-07-31 00:00:00</td>\n",
              "      <td>2015-07-31 00:00:00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Sales</th>\n",
              "      <td>5263</td>\n",
              "      <td>6064</td>\n",
              "      <td>8314</td>\n",
              "      <td>13995</td>\n",
              "      <td>4822</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Customers</th>\n",
              "      <td>555</td>\n",
              "      <td>625</td>\n",
              "      <td>821</td>\n",
              "      <td>1498</td>\n",
              "      <td>559</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Open</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Promo</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>StateHoliday</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>SchoolHoliday</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Year</th>\n",
              "      <td>2015</td>\n",
              "      <td>2015</td>\n",
              "      <td>2015</td>\n",
              "      <td>2015</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Month</th>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "      <td>7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Week</th>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Day</th>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "      <td>31</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Dayofweek</th>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Dayofyear</th>\n",
              "      <td>212</td>\n",
              "      <td>212</td>\n",
              "      <td>212</td>\n",
              "      <td>212</td>\n",
              "      <td>212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_month_end</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_month_start</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_quarter_end</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_quarter_start</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_year_end</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Is_year_start</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Elapsed</th>\n",
              "      <td>1438300800</td>\n",
              "      <td>1438300800</td>\n",
              "      <td>1438300800</td>\n",
              "      <td>1438300800</td>\n",
              "      <td>1438300800</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>StoreType</th>\n",
              "      <td>c</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>c</td>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Assortment</th>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>c</td>\n",
              "      <td>a</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CompetitionDistance</th>\n",
              "      <td>1270</td>\n",
              "      <td>570</td>\n",
              "      <td>14130</td>\n",
              "      <td>620</td>\n",
              "      <td>29910</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CompetitionOpenSinceMonth</th>\n",
              "      <td>9</td>\n",
              "      <td>11</td>\n",
              "      <td>12</td>\n",
              "      <td>9</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>CompetitionOpenSinceYear</th>\n",
              "      <td>2008</td>\n",
              "      <td>2007</td>\n",
              "      <td>2006</td>\n",
              "      <td>2009</td>\n",
              "      <td>2015</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Promo2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Promo2SinceWeek</th>\n",
              "      <td>1</td>\n",
              "      <td>13</td>\n",
              "      <td>14</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Promo2SinceYear</th>\n",
              "      <td>1900</td>\n",
              "      <td>2010</td>\n",
              "      <td>2011</td>\n",
              "      <td>1900</td>\n",
              "      <td>1900</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>PromoInterval</th>\n",
              "      <td>None</td>\n",
              "      <td>Jan,Apr,Jul,Oct</td>\n",
              "      <td>Jan,Apr,Jul,Oct</td>\n",
              "      <td>None</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>State</th>\n",
              "      <td>HE</td>\n",
              "      <td>TH</td>\n",
              "      <td>NW</td>\n",
              "      <td>BE</td>\n",
              "      <td>SN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>file</th>\n",
              "      <td>Rossmann_DE_HE</td>\n",
              "      <td>Rossmann_DE_TH</td>\n",
              "      <td>Rossmann_DE_NW</td>\n",
              "      <td>Rossmann_DE_BE</td>\n",
              "      <td>Rossmann_DE_SN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>week</th>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>trend</th>\n",
              "      <td>85</td>\n",
              "      <td>80</td>\n",
              "      <td>86</td>\n",
              "      <td>74</td>\n",
              "      <td>82</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>file_DE</th>\n",
              "      <td>Rossmann_DE</td>\n",
              "      <td>Rossmann_DE</td>\n",
              "      <td>Rossmann_DE</td>\n",
              "      <td>Rossmann_DE</td>\n",
              "      <td>Rossmann_DE</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>week_DE</th>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "      <td>2015-08-02 - 2015-08-08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>trend_DE</th>\n",
              "      <td>83</td>\n",
              "      <td>83</td>\n",
              "      <td>83</td>\n",
              "      <td>83</td>\n",
              "      <td>83</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date_DE</th>\n",
              "      <td>2015-08-02 00:00:00</td>\n",
              "      <td>2015-08-02 00:00:00</td>\n",
              "      <td>2015-08-02 00:00:00</td>\n",
              "      <td>2015-08-02 00:00:00</td>\n",
              "      <td>2015-08-02 00:00:00</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 0                        1  \\\n",
              "index                                            0                        1   \n",
              "Store                                            1                        2   \n",
              "DayOfWeek                                        5                        5   \n",
              "Date                           2015-07-31 00:00:00      2015-07-31 00:00:00   \n",
              "Sales                                         5263                     6064   \n",
              "Customers                                      555                      625   \n",
              "Open                                             1                        1   \n",
              "Promo                                            1                        1   \n",
              "StateHoliday                                 False                    False   \n",
              "SchoolHoliday                                    1                        1   \n",
              "Year                                          2015                     2015   \n",
              "Month                                            7                        7   \n",
              "Week                                            31                       31   \n",
              "Day                                             31                       31   \n",
              "Dayofweek                                        4                        4   \n",
              "Dayofyear                                      212                      212   \n",
              "Is_month_end                                  True                     True   \n",
              "Is_month_start                               False                    False   \n",
              "Is_quarter_end                               False                    False   \n",
              "Is_quarter_start                             False                    False   \n",
              "Is_year_end                                  False                    False   \n",
              "Is_year_start                                False                    False   \n",
              "Elapsed                                 1438300800               1438300800   \n",
              "StoreType                                        c                        a   \n",
              "Assortment                                       a                        a   \n",
              "CompetitionDistance                           1270                      570   \n",
              "CompetitionOpenSinceMonth                        9                       11   \n",
              "CompetitionOpenSinceYear                      2008                     2007   \n",
              "Promo2                                           0                        1   \n",
              "Promo2SinceWeek                                  1                       13   \n",
              "Promo2SinceYear                               1900                     2010   \n",
              "PromoInterval                                 None          Jan,Apr,Jul,Oct   \n",
              "State                                           HE                       TH   \n",
              "file                                Rossmann_DE_HE           Rossmann_DE_TH   \n",
              "week                       2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
              "trend                                           85                       80   \n",
              "file_DE                                Rossmann_DE              Rossmann_DE   \n",
              "week_DE                    2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
              "trend_DE                                        83                       83   \n",
              "Date_DE                        2015-08-02 00:00:00      2015-08-02 00:00:00   \n",
              "\n",
              "                                                 2                        3  \\\n",
              "index                                            2                        3   \n",
              "Store                                            3                        4   \n",
              "DayOfWeek                                        5                        5   \n",
              "Date                           2015-07-31 00:00:00      2015-07-31 00:00:00   \n",
              "Sales                                         8314                    13995   \n",
              "Customers                                      821                     1498   \n",
              "Open                                             1                        1   \n",
              "Promo                                            1                        1   \n",
              "StateHoliday                                 False                    False   \n",
              "SchoolHoliday                                    1                        1   \n",
              "Year                                          2015                     2015   \n",
              "Month                                            7                        7   \n",
              "Week                                            31                       31   \n",
              "Day                                             31                       31   \n",
              "Dayofweek                                        4                        4   \n",
              "Dayofyear                                      212                      212   \n",
              "Is_month_end                                  True                     True   \n",
              "Is_month_start                               False                    False   \n",
              "Is_quarter_end                               False                    False   \n",
              "Is_quarter_start                             False                    False   \n",
              "Is_year_end                                  False                    False   \n",
              "Is_year_start                                False                    False   \n",
              "Elapsed                                 1438300800               1438300800   \n",
              "StoreType                                        a                        c   \n",
              "Assortment                                       a                        c   \n",
              "CompetitionDistance                          14130                      620   \n",
              "CompetitionOpenSinceMonth                       12                        9   \n",
              "CompetitionOpenSinceYear                      2006                     2009   \n",
              "Promo2                                           1                        0   \n",
              "Promo2SinceWeek                                 14                        1   \n",
              "Promo2SinceYear                               2011                     1900   \n",
              "PromoInterval                      Jan,Apr,Jul,Oct                     None   \n",
              "State                                           NW                       BE   \n",
              "file                                Rossmann_DE_NW           Rossmann_DE_BE   \n",
              "week                       2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
              "trend                                           86                       74   \n",
              "file_DE                                Rossmann_DE              Rossmann_DE   \n",
              "week_DE                    2015-08-02 - 2015-08-08  2015-08-02 - 2015-08-08   \n",
              "trend_DE                                        83                       83   \n",
              "Date_DE                        2015-08-02 00:00:00      2015-08-02 00:00:00   \n",
              "\n",
              "                                                 4  \n",
              "index                                            4  \n",
              "Store                                            5  \n",
              "DayOfWeek                                        5  \n",
              "Date                           2015-07-31 00:00:00  \n",
              "Sales                                         4822  \n",
              "Customers                                      559  \n",
              "Open                                             1  \n",
              "Promo                                            1  \n",
              "StateHoliday                                 False  \n",
              "SchoolHoliday                                    1  \n",
              "Year                                          2015  \n",
              "Month                                            7  \n",
              "Week                                            31  \n",
              "Day                                             31  \n",
              "Dayofweek                                        4  \n",
              "Dayofyear                                      212  \n",
              "Is_month_end                                  True  \n",
              "Is_month_start                               False  \n",
              "Is_quarter_end                               False  \n",
              "Is_quarter_start                             False  \n",
              "Is_year_end                                  False  \n",
              "Is_year_start                                False  \n",
              "Elapsed                                 1438300800  \n",
              "StoreType                                        a  \n",
              "Assortment                                       a  \n",
              "CompetitionDistance                          29910  \n",
              "CompetitionOpenSinceMonth                        4  \n",
              "CompetitionOpenSinceYear                      2015  \n",
              "Promo2                                           0  \n",
              "Promo2SinceWeek                                  1  \n",
              "Promo2SinceYear                               1900  \n",
              "PromoInterval                                 None  \n",
              "State                                           SN  \n",
              "file                                Rossmann_DE_SN  \n",
              "week                       2015-08-02 - 2015-08-08  \n",
              "trend                                           82  \n",
              "file_DE                                Rossmann_DE  \n",
              "week_DE                    2015-08-02 - 2015-08-08  \n",
              "trend_DE                                        83  \n",
              "Date_DE                        2015-08-02 00:00:00  "
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "metadata": {
        "id": "fflTLYTXzbF-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e99f7496-34b5-4f6d-afa9-cb69be501cbf"
      },
      "cell_type": "code",
      "source": [
        "# Категориальные данные\n",
        "cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',\n",
        "    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',\n",
        "    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',\n",
        "    'SchoolHoliday_fw', 'SchoolHoliday_bw']\n",
        "\n",
        "# Количественные данные\n",
        "contin_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',\n",
        "   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', \n",
        "   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',\n",
        "   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']\n",
        "\n",
        "n = len(joined)\n",
        "n"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "844338"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "metadata": {
        "id": "OE-ntcN81L0b",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "dep = 'Sales'\n",
        "joined = joined[cat_vars+contin_vars+[dep, 'Date']].copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YeoCxNOM1Uwt",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined_test[dep] = 0\n",
        "joined_test = joined_test[cat_vars+contin_vars+[dep, 'Date', 'Id']].copy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fshtIIF91YJd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for v in cat_vars: joined[v] = joined[v].astype('category').cat.as_ordered()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5VIJw8VE1a8V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "apply_cats(joined_test, joined)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "PCVhgjOy1eWd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "for v in contin_vars:\n",
        "    joined[v] = joined[v].fillna(0).astype('float32')\n",
        "    joined_test[v] = joined_test[v].fillna(0).astype('float32')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TWJ8zf-_1h1A",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "0260423e-dfe3-448e-8c8d-113b71b1edd6"
      },
      "cell_type": "code",
      "source": [
        "idxs = get_cv_idxs(n, val_pct=150000/n)\n",
        "joined_samp = joined.iloc[idxs].set_index(\"Date\")\n",
        "samp_size = len(joined_samp)\n",
        "samp_size"
      ],
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "150000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "metadata": {
        "id": "azmSR-No1pFz",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "samp_size = n\n",
        "joined_samp = joined.set_index(\"Date\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "jBQmhz171vjn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "ec92342c-ae8b-4db1-9f26-216a86735bae"
      },
      "cell_type": "code",
      "source": [
        "joined_samp.head(5)"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Year</th>\n",
              "      <th>Month</th>\n",
              "      <th>Day</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>CompetitionMonthsOpen</th>\n",
              "      <th>Promo2Weeks</th>\n",
              "      <th>StoreType</th>\n",
              "      <th>Assortment</th>\n",
              "      <th>...</th>\n",
              "      <th>Max_Wind_SpeedKm_h</th>\n",
              "      <th>Mean_Wind_SpeedKm_h</th>\n",
              "      <th>CloudCover</th>\n",
              "      <th>trend</th>\n",
              "      <th>trend_DE</th>\n",
              "      <th>AfterStateHoliday</th>\n",
              "      <th>BeforeStateHoliday</th>\n",
              "      <th>Promo</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "      <th>Sales</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>2015</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>False</td>\n",
              "      <td>24</td>\n",
              "      <td>0</td>\n",
              "      <td>c</td>\n",
              "      <td>a</td>\n",
              "      <td>...</td>\n",
              "      <td>24.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>85.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>57.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>5263</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>2015</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>False</td>\n",
              "      <td>24</td>\n",
              "      <td>25</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>...</td>\n",
              "      <td>14.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>67.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6064</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>2015</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>False</td>\n",
              "      <td>24</td>\n",
              "      <td>25</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>...</td>\n",
              "      <td>14.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>86.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>57.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>8314</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>2015</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>False</td>\n",
              "      <td>24</td>\n",
              "      <td>0</td>\n",
              "      <td>c</td>\n",
              "      <td>c</td>\n",
              "      <td>...</td>\n",
              "      <td>23.0</td>\n",
              "      <td>16.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>67.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>13995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>2015</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>False</td>\n",
              "      <td>3</td>\n",
              "      <td>0</td>\n",
              "      <td>a</td>\n",
              "      <td>a</td>\n",
              "      <td>...</td>\n",
              "      <td>14.0</td>\n",
              "      <td>11.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>82.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>57.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>4822</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 39 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "           Store DayOfWeek  Year Month Day StateHoliday CompetitionMonthsOpen  \\\n",
              "Date                                                                            \n",
              "2015-07-31     1         5  2015     7  31        False                    24   \n",
              "2015-07-31     2         5  2015     7  31        False                    24   \n",
              "2015-07-31     3         5  2015     7  31        False                    24   \n",
              "2015-07-31     4         5  2015     7  31        False                    24   \n",
              "2015-07-31     5         5  2015     7  31        False                     3   \n",
              "\n",
              "           Promo2Weeks StoreType Assortment  ...   Max_Wind_SpeedKm_h  \\\n",
              "Date                                         ...                        \n",
              "2015-07-31           0         c          a  ...                 24.0   \n",
              "2015-07-31          25         a          a  ...                 14.0   \n",
              "2015-07-31          25         a          a  ...                 14.0   \n",
              "2015-07-31           0         c          c  ...                 23.0   \n",
              "2015-07-31           0         a          a  ...                 14.0   \n",
              "\n",
              "           Mean_Wind_SpeedKm_h CloudCover trend trend_DE AfterStateHoliday  \\\n",
              "Date                                                                         \n",
              "2015-07-31                11.0        1.0  85.0     83.0              57.0   \n",
              "2015-07-31                11.0        4.0  80.0     83.0              67.0   \n",
              "2015-07-31                 5.0        2.0  86.0     83.0              57.0   \n",
              "2015-07-31                16.0        6.0  74.0     83.0              67.0   \n",
              "2015-07-31                11.0        4.0  82.0     83.0              57.0   \n",
              "\n",
              "           BeforeStateHoliday Promo SchoolHoliday  Sales  \n",
              "Date                                                      \n",
              "2015-07-31                0.0   1.0           1.0   5263  \n",
              "2015-07-31                0.0   1.0           1.0   6064  \n",
              "2015-07-31                0.0   1.0           1.0   8314  \n",
              "2015-07-31                0.0   1.0           1.0  13995  \n",
              "2015-07-31                0.0   1.0           1.0   4822  \n",
              "\n",
              "[5 rows x 39 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "metadata": {
        "id": "oq-4Zrrc1054",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Разделение на датафрейм и целевую переменную y\n",
        "df, y, nas, mapper = proc_df(joined_samp, 'Sales', do_scale=True)\n",
        "yl = np.log(y) # Логарифмируем целевую переменную (зависит от требуемой задачи)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ads2eIte18JI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined_test = joined_test.set_index(\"Date\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Q06GlO731_WN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "df_test, _, nas, mapper = proc_df(joined_test, 'Sales', do_scale=True, skip_flds=['Id'], mapper=mapper, na_dict=nas)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_UamRdsT2Db8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "9a7965b3-e583-44e9-a3d5-8b446c945f5f"
      },
      "cell_type": "code",
      "source": [
        "# Датафрейм преобразованный для нейронной сети\n",
        "df.head(5)"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Store</th>\n",
              "      <th>DayOfWeek</th>\n",
              "      <th>Year</th>\n",
              "      <th>Month</th>\n",
              "      <th>Day</th>\n",
              "      <th>StateHoliday</th>\n",
              "      <th>CompetitionMonthsOpen</th>\n",
              "      <th>Promo2Weeks</th>\n",
              "      <th>StoreType</th>\n",
              "      <th>Assortment</th>\n",
              "      <th>...</th>\n",
              "      <th>Min_Humidity</th>\n",
              "      <th>Max_Wind_SpeedKm_h</th>\n",
              "      <th>Mean_Wind_SpeedKm_h</th>\n",
              "      <th>CloudCover</th>\n",
              "      <th>trend</th>\n",
              "      <th>trend_DE</th>\n",
              "      <th>AfterStateHoliday</th>\n",
              "      <th>BeforeStateHoliday</th>\n",
              "      <th>Promo</th>\n",
              "      <th>SchoolHoliday</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Date</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>1</td>\n",
              "      <td>25</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.620066</td>\n",
              "      <td>0.149027</td>\n",
              "      <td>-0.142774</td>\n",
              "      <td>-1.844823</td>\n",
              "      <td>1.732492</td>\n",
              "      <td>1.724334</td>\n",
              "      <td>0.604461</td>\n",
              "      <td>1.13112</td>\n",
              "      <td>1.113717</td>\n",
              "      <td>2.04105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>2</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>1</td>\n",
              "      <td>25</td>\n",
              "      <td>26</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.264031</td>\n",
              "      <td>-0.960613</td>\n",
              "      <td>-0.142774</td>\n",
              "      <td>-0.488722</td>\n",
              "      <td>1.294578</td>\n",
              "      <td>1.724334</td>\n",
              "      <td>0.926957</td>\n",
              "      <td>1.13112</td>\n",
              "      <td>1.113717</td>\n",
              "      <td>2.04105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>3</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>1</td>\n",
              "      <td>25</td>\n",
              "      <td>26</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.314893</td>\n",
              "      <td>-0.960613</td>\n",
              "      <td>-1.154031</td>\n",
              "      <td>-1.392789</td>\n",
              "      <td>1.820074</td>\n",
              "      <td>1.724334</td>\n",
              "      <td>0.604461</td>\n",
              "      <td>1.13112</td>\n",
              "      <td>1.113717</td>\n",
              "      <td>2.04105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>4</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>1</td>\n",
              "      <td>25</td>\n",
              "      <td>1</td>\n",
              "      <td>3</td>\n",
              "      <td>3</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.009721</td>\n",
              "      <td>0.038063</td>\n",
              "      <td>0.699941</td>\n",
              "      <td>0.415345</td>\n",
              "      <td>0.769081</td>\n",
              "      <td>1.724334</td>\n",
              "      <td>0.926957</td>\n",
              "      <td>1.13112</td>\n",
              "      <td>1.113717</td>\n",
              "      <td>2.04105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2015-07-31</th>\n",
              "      <td>5</td>\n",
              "      <td>5</td>\n",
              "      <td>3</td>\n",
              "      <td>7</td>\n",
              "      <td>31</td>\n",
              "      <td>1</td>\n",
              "      <td>4</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.213169</td>\n",
              "      <td>-0.960613</td>\n",
              "      <td>-0.142774</td>\n",
              "      <td>-0.488722</td>\n",
              "      <td>1.469743</td>\n",
              "      <td>1.724334</td>\n",
              "      <td>0.604461</td>\n",
              "      <td>1.13112</td>\n",
              "      <td>1.113717</td>\n",
              "      <td>2.04105</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 38 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            Store  DayOfWeek  Year  Month  Day  StateHoliday  \\\n",
              "Date                                                           \n",
              "2015-07-31      1          5     3      7   31             1   \n",
              "2015-07-31      2          5     3      7   31             1   \n",
              "2015-07-31      3          5     3      7   31             1   \n",
              "2015-07-31      4          5     3      7   31             1   \n",
              "2015-07-31      5          5     3      7   31             1   \n",
              "\n",
              "            CompetitionMonthsOpen  Promo2Weeks  StoreType  Assortment  \\\n",
              "Date                                                                    \n",
              "2015-07-31                     25            1          3           1   \n",
              "2015-07-31                     25           26          1           1   \n",
              "2015-07-31                     25           26          1           1   \n",
              "2015-07-31                     25            1          3           3   \n",
              "2015-07-31                      4            1          1           1   \n",
              "\n",
              "                ...        Min_Humidity  Max_Wind_SpeedKm_h  \\\n",
              "Date            ...                                           \n",
              "2015-07-31      ...           -1.620066            0.149027   \n",
              "2015-07-31      ...           -1.264031           -0.960613   \n",
              "2015-07-31      ...           -1.314893           -0.960613   \n",
              "2015-07-31      ...           -1.009721            0.038063   \n",
              "2015-07-31      ...           -1.213169           -0.960613   \n",
              "\n",
              "            Mean_Wind_SpeedKm_h  CloudCover     trend  trend_DE  \\\n",
              "Date                                                              \n",
              "2015-07-31            -0.142774   -1.844823  1.732492  1.724334   \n",
              "2015-07-31            -0.142774   -0.488722  1.294578  1.724334   \n",
              "2015-07-31            -1.154031   -1.392789  1.820074  1.724334   \n",
              "2015-07-31             0.699941    0.415345  0.769081  1.724334   \n",
              "2015-07-31            -0.142774   -0.488722  1.469743  1.724334   \n",
              "\n",
              "            AfterStateHoliday  BeforeStateHoliday     Promo  SchoolHoliday  \n",
              "Date                                                                        \n",
              "2015-07-31           0.604461             1.13112  1.113717        2.04105  \n",
              "2015-07-31           0.926957             1.13112  1.113717        2.04105  \n",
              "2015-07-31           0.604461             1.13112  1.113717        2.04105  \n",
              "2015-07-31           0.926957             1.13112  1.113717        2.04105  \n",
              "2015-07-31           0.604461             1.13112  1.113717        2.04105  \n",
              "\n",
              "[5 rows x 38 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 88
        }
      ]
    },
    {
      "metadata": {
        "id": "kGrPnfnV2G6M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "bcf37b57-f829-4273-f058-b617a03ac9ed"
      },
      "cell_type": "code",
      "source": [
        "# Возможный вариант валидационной выборки\n",
        "train_ratio = 0.75\n",
        "# train_ratio = 0.9\n",
        "train_size = int(samp_size * train_ratio); train_size\n",
        "val_idx = list(range(train_size, len(df)))\n",
        "val_idx[:10]"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[633253,\n",
              " 633254,\n",
              " 633255,\n",
              " 633256,\n",
              " 633257,\n",
              " 633258,\n",
              " 633259,\n",
              " 633260,\n",
              " 633261,\n",
              " 633262]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 94
        }
      ]
    },
    {
      "metadata": {
        "id": "uyi-_GWe2g_z",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "88e3c180-7f2c-417a-c3d2-5165c1b8e4ce"
      },
      "cell_type": "code",
      "source": [
        "# Валидационная выборка из 2 последних недель датафрейма\n",
        "val_idx = np.flatnonzero((df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014,8,1)))\n",
        "val_idx"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([277141, 277142, 277143, 277144, 277145, 277146, 277147, 277148, 277149, 277150, 277151, 277152,\n",
              "       277153, 277154, 277155, 277156, 277157, 277158, 277159, 277160, ..., 315520, 315521, 315522, 315523,\n",
              "       315524, 315525, 315526, 315527, 315528, 315529, 315530, 315531, 315532, 315533, 315534, 315535,\n",
              "       315536, 315537, 315538, 315539])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 92
        }
      ]
    },
    {
      "metadata": {
        "id": "ODJkjWwy2vyZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "val_idx=[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4WUSg9-33Wd7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Нейронная сеть"
      ]
    },
    {
      "metadata": {
        "id": "qpGQfqN93Jrq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Создаём метрику оценки качества модели\n",
        "\n",
        "def inv_y(a): return np.exp(a)\n",
        "\n",
        "def exp_rmspe(y_pred, targ):\n",
        "    targ = inv_y(targ)\n",
        "    pct_var = (targ - inv_y(y_pred))/targ\n",
        "    return math.sqrt((pct_var**2).mean())\n",
        "\n",
        "max_log_y = np.max(yl)\n",
        "y_range = (0, max_log_y*1.2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "A7uiJdta3hDQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Данные модели для обучения\n",
        "md = ColumnarModelData.from_data_frame(PATH,                  # каталог модели\n",
        "                                       val_idx,               # валидационная выборка\n",
        "                                       df,                    # обучающий датафрейм\n",
        "                                       yl.astype(np.float32), # целевая переменная\n",
        "                                       cat_flds=cat_vars,     # категориальные признаки\n",
        "                                       bs=128,                # размер минибатча\n",
        "                                       test_df=df_test)       # тестоввая выбока"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "YDrFRQ0i3xzx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "3e393ef9-d27e-47fa-eb7d-717438fb9199"
      },
      "cell_type": "code",
      "source": [
        "# Размеры категорий (+1 для неизвестного нейронке значения признака)\n",
        "cat_sz = [(c, len(joined_samp[c].cat.categories)+1) for c in cat_vars]\n",
        "cat_sz # мощностя категорий"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('Store', 1116),\n",
              " ('DayOfWeek', 8),\n",
              " ('Year', 4),\n",
              " ('Month', 13),\n",
              " ('Day', 32),\n",
              " ('StateHoliday', 3),\n",
              " ('CompetitionMonthsOpen', 26),\n",
              " ('Promo2Weeks', 27),\n",
              " ('StoreType', 5),\n",
              " ('Assortment', 4),\n",
              " ('PromoInterval', 4),\n",
              " ('CompetitionOpenSinceYear', 24),\n",
              " ('Promo2SinceYear', 9),\n",
              " ('State', 13),\n",
              " ('Week', 53),\n",
              " ('Events', 22),\n",
              " ('Promo_fw', 7),\n",
              " ('Promo_bw', 7),\n",
              " ('StateHoliday_fw', 4),\n",
              " ('StateHoliday_bw', 4),\n",
              " ('SchoolHoliday_fw', 9),\n",
              " ('SchoolHoliday_bw', 9)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "metadata": {
        "id": "WywTdPuv31MB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "772e655f-136c-4524-9680-2982982eeb52"
      },
      "cell_type": "code",
      "source": [
        "# Размеры матриц эмбэддинга\n",
        "emb_szs = [(c, min(50, (c+1)//2)) for _,c in cat_sz]\n",
        "emb_szs"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(1116, 50),\n",
              " (8, 4),\n",
              " (4, 2),\n",
              " (13, 7),\n",
              " (32, 16),\n",
              " (3, 2),\n",
              " (26, 13),\n",
              " (27, 14),\n",
              " (5, 3),\n",
              " (4, 2),\n",
              " (4, 2),\n",
              " (24, 12),\n",
              " (9, 5),\n",
              " (13, 7),\n",
              " (53, 27),\n",
              " (22, 11),\n",
              " (7, 4),\n",
              " (7, 4),\n",
              " (4, 2),\n",
              " (4, 2),\n",
              " (9, 5),\n",
              " (9, 5)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 100
        }
      ]
    },
    {
      "metadata": {
        "id": "oSmER8MC40_2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 714
        },
        "outputId": "26177ff9-011c-4894-a73e-a8622ad4946d"
      },
      "cell_type": "code",
      "source": [
        "# Модель нейронной сети\n",
        "m = md.get_learner(emb_szs,            # маттрица эмбеддинга для категориальных признаков\n",
        "                   len(df.columns)-len(cat_vars), # количество количественных признаков\n",
        "                   0.04,                          # дропаут для эмбеддинга\n",
        "                   1,                  # Количество чисел на выходе (мы предсказываем продажи)\n",
        "                   [1000,500],         # список количества активаций каждого слоя\n",
        "                   [0.001,0.01],       # список дропаутов для каждого слоя\n",
        "                   y_range=y_range)    # целевая переменная\n",
        "m"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MixedInputModel(\n",
              "  (embs): ModuleList(\n",
              "    (0): Embedding(1116, 50)\n",
              "    (1): Embedding(8, 4)\n",
              "    (2): Embedding(4, 2)\n",
              "    (3): Embedding(13, 7)\n",
              "    (4): Embedding(32, 16)\n",
              "    (5): Embedding(3, 2)\n",
              "    (6): Embedding(26, 13)\n",
              "    (7): Embedding(27, 14)\n",
              "    (8): Embedding(5, 3)\n",
              "    (9): Embedding(4, 2)\n",
              "    (10): Embedding(4, 2)\n",
              "    (11): Embedding(24, 12)\n",
              "    (12): Embedding(9, 5)\n",
              "    (13): Embedding(13, 7)\n",
              "    (14): Embedding(53, 27)\n",
              "    (15): Embedding(22, 11)\n",
              "    (16): Embedding(7, 4)\n",
              "    (17): Embedding(7, 4)\n",
              "    (18): Embedding(4, 2)\n",
              "    (19): Embedding(4, 2)\n",
              "    (20): Embedding(9, 5)\n",
              "    (21): Embedding(9, 5)\n",
              "  )\n",
              "  (lins): ModuleList(\n",
              "    (0): Linear(in_features=215, out_features=1000, bias=True)\n",
              "    (1): Linear(in_features=1000, out_features=500, bias=True)\n",
              "  )\n",
              "  (bns): ModuleList(\n",
              "    (0): BatchNorm1d(1000, eps=1e-05, momentum=0.1, affine=True)\n",
              "    (1): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True)\n",
              "  )\n",
              "  (outp): Linear(in_features=500, out_features=1, bias=True)\n",
              "  (emb_drop): Dropout(p=0.04)\n",
              "  (drops): ModuleList(\n",
              "    (0): Dropout(p=0.001)\n",
              "    (1): Dropout(p=0.01)\n",
              "  )\n",
              "  (bn): BatchNorm1d(16, eps=1e-05, momentum=0.1, affine=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 104
        }
      ]
    },
    {
      "metadata": {
        "id": "VIUWmZlX5AkW",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "2df76485-1d7a-47c9-cea0-d38a3facd063"
      },
      "cell_type": "code",
      "source": [
        "m.lr_find()"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5cd29caf36ce4380abf494bd1765d771",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            " 61%|██████    | 3993/6597 [01:54<01:14, 34.90it/s, loss=0.131] "
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TQoRN_1E5uNn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "a5ca2ff9-5d70-462f-dafc-6fad4291b51e"
      },
      "cell_type": "code",
      "source": [
        "m.sched.plot(100)"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEKCAYAAAAb7IIBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmcnWV99/HPWWefyUxmspKYAMlP\nFhcWQQwIEaQ8ahcVl8fHWhV3fKHWp2q1VGvr0sciFa0+WmyttS6tFSqICGKRQEAgKMjiL2wJCdkm\nk9m3s/aP+8zkZPZkcs+ZzP19v168Muec+5zzOxfzOt+5ruu+rytWLBYREZHoiVe6ABERqQwFgIhI\nRCkAREQiSgEgIhJRCgARkYhSAIiIRFSy0gXMVHt775TnqzY319LZOTBX5USG2jUcatdwqF3Ha2tr\niE322ILpASSTiUqXsCCpXcOhdg2H2vXwLJgAEBGRw6MAEBGJKAWAiEhEKQBERCJKASAiElEKABGR\niFrwATA4nOP+3+2jUNCy1yIi5RZ8ANz9yB6+ev3D+DOdlS5FRGReWfABkC/95d8/lKtwJSIi88uC\nD4BUIviIuXyhwpWIiMwvCz4AkqUAyCoAREQOsfADIBmsg5TLaxJYRKTcgg+AVCJYHCqbUw9ARKRc\nqMtBm9mpwH8BV7v7V8Y8thH4HJAHHHiHux/1b+nUaA9AASAiUi60HoCZ1QFfBm6b5JBvAJe6+wag\nAbgkjDpG5gBy6gGIiBwizCGgYeAVwK5JHj/D3XeWfm4HFodRhCaBRUQmFtoQkLvngJyZTfZ4D4CZ\nLQcuBq6c6vWam2un3eyhra1h3H2dg8H5/6l0csLHZXpqt3CoXcOhdp25im4JaWZLgBuA97l7x1TH\nTrfNW1tbA+3tvePu7+8dAqCnd2jCx2Vqk7WrzI7aNRxq1/GmCsSKBYCZNQI/BT7h7reE9T7JpC4E\nExGZSCVPA72K4Oygm8N8k2QiOAsom9N1ACIi5ULrAZjZGQRf8muArJldCvwYeBr4GfAWYJ2ZvaP0\nlO+6+zeOdh0pTQKLiEwozEngLcAFUxxSFdZ7l0sldRqoiMhEFvyVwEktBiciMqGFHwCaBBYRmdCC\nD4B4LEYiHtNaQCIiYyz4AICgF6BJYBGRQ0UiAFKJuJaDFhEZIxIBkEzEdBaQiMgYEQkADQGJiIwV\niQBIJeOaBBYRGSMaAZCI6zRQEZExIhEAyaQCQERkrGgEQOksoGJRZwKJiIyIRACkdDWwiMg40QiA\nkRVBtSS0iMioSATAyJ4A6gGIiBwUjQDQEJCIyDiRCICDQ0AKABGREZEIgJEegK4GFhE5KBIBkNKm\nMCIi40QiAEZ3BdNZQCIioyISAMFZQNlcvsKViIjMH5EIgNToHIB6ACIiI6IRAJoDEBEZJxIBoOsA\nRETGS4b54mZ2KvBfwNXu/pUxj10EfBbIAze5+1+HVUdS1wGIiIwTWg/AzOqALwO3TXLINcBrgQ3A\nxWZ2cli1jM4BKABEREaFOQQ0DLwC2DX2ATM7Hjjg7jvcvQDcBFwYViHpZAKAjAJARGRUaAHg7jl3\nH5zk4WVAe9ntfcDysGpJp0Z6ADoNVERkRKhzAIchNt0Bzc21JEt/yU+mra1h4vt7hgFIppKTHiOT\nU5uFQ+0aDrXrzFUqAHYR9AJGrGSCoaJynZ0DU75gW1sD7e29Ez422B8EQFfP4KTHyMSmalc5cmrX\ncKhdx5sqECtyGqi7bwMazWyNmSWBVwG3hPV+I5PAmgMQETkotB6AmZ0BXAWsAbJmdinwY+Bpd78O\neC/wvdLhP3D3rWHVkh4JgKzmAERERoQWAO6+BbhgisfvAM4J6/3LpVPB3IFOAxUROSgSVwIf7AEo\nAERERkQjAFIj1wFoCEhEZEQkAiARjxGLaRJYRKRcJAIgFouRTiY0CSwiUiYSAQDB1cCaBBYROSg6\nAZCMaxJYRKRMdAIgldAksIhImcgEQCoZ1ySwiEiZyATAyCRwsah9gUVEIEoBkIpTLEK+oAAQEYEo\nBcDIpjCaCBYRAaIUAKmRFUE1ESwiApEKgKAHMKyLwUREgAgFQNVIAGQUACIiEKEAqE6rByAiUi4y\nAaAhIBGRQ0UmAA4OAeksIBERiFAAHBwCylW4EhGR+SEyATByGuiwrgMQEQEiFAA6C0hE5FCRCYBq\nTQKLiBwiMgGQ1mmgIiKHiEwAVGsISETkENEJgHQSgKGMzgISEQFIhvniZnY18GKgCHzA3e8re+xy\n4M1AHrjf3T8YZi3VVUEPYEg9ABERIMQegJmdD6xz93OAy4Bryh5rBP4MOM/dzwVONrMXh1ULHLwO\nQAEgIhIIcwjoQuB6AHd/DGguffEDZEr/1ZtZEqgFDoRYC4l4nFQyriEgEZGSMIeAlgFbym63l+7r\ncfchM/sr4ClgEPi+u2+d6sWam2tJljZ1mUxbW8OUj9dWJ8nkitMeJ4dSe4VD7RoOtevMhToHMEZs\n5IdST+DjwHqgB/iFmb3A3R+c7MmdnQNTvnhbWwPt7b1THpNOxukfzEx7nBw0k3aVw6d2DYfadbyp\nAjHMIaBdBH/xj1gB7C79fBLwlLvvd/cMsAk4I8RagOBMIM0BiIgEwgyAW4BLAczsdGCXu49E8zbg\nJDOrKd0+E3g8xFqAYCJ4OJOnWNTG8CIioQ0BuftmM9tiZpuBAnC5mb0V6Hb368zsC8B/m1kO2Ozu\nm8KqZUR1OkmR4GrgkesCRESiKtRvQXf/2Ji7Hix77OvA18N8/7HKTwVVAIhI1EXmSmDQtQAiIuUi\nFgBaDkJEZETEAqDUAxhWD0BEJFoBoPWARERGRSoAakpDQIMaAhIRiVgAVJUCYFgBICKiABARiahI\nBUBtKQAGFAAiItEKgJrSJPCgzgISEYlaAGgISERkRKQCoLa6NAQ0pAAQEYlUAFSlEsRjMfUAREQ4\nggAwsyozWxVGMWGLxWLUVCUUACIizHA1UDP7c6AP+CZwP9BrZre4+5VhFheGmqqkzgISEWHmPYDf\nB74CvA64wd3PBjaEVlWI6qpT9A9lK12GiEjFzTQAsu5eBP4XcH3pvql3aJ+n6muSZLIFsjmdCioi\n0TbTAOgys58AJ7n73Wb2KoJdvo45dTUpAPoGNQwkItE2022x3gS8HLirdHsI+JNQKgrZwQDI0txQ\nVeFqREQqZ6Y9gDag3d3bzeydwP8G6sIrKzz11UEA9A9qHkBEom2mAfDPQMbMTgPeAfwncE1oVYWo\nvqwHICISZTMNgKK73we8GviKu98ExMIrKzz1tUEA9CoARCTiZjoHUG9mLwIuBc43syqgObyywtNY\nlwagpz9T4UpERCprpj2Aq4B/BL7u7u3Ap4DvhlVUmJpqFQAiIjDDHoC7/wD4gZm1mFkz8PHSdQFT\nMrOrgRcDReADpWGkkcdWAd8D0sAD7v6eI/kAh2u0BzCgABCRaJtRD8DMNpjZk8DvgMeBx8zszGme\ncz6wzt3PAS5j/KTxVcBV7n4WkDez1Ydd/RGor0kRi6kHICIy0yGgzwF/6O5L3L2V4DTQL07znAsp\nXTXs7o8BzWbWCGBmceA84Melxy9392eOoP7DFo/HaKhJKQBEJPJmOgmcd/eHR264+6/NbLpLaZcB\nW8put5fu6yG4rqAXuNrMTgc2ufufT/Vizc21JJNTrz7R1tYwTUmBlqYa9h4YmPHxUad2CofaNRxq\n15mbaQAUzOy1wK2l25cAh7uYTmzMzyuBLwHbgJ+Y2Svd/SeTPbmzc2DKF29ra6C9vXdGhdRXJ9k2\nnGPns11UpY/JJY3mzOG0q8yc2jUcatfxpgrEmQ4BvQd4J8GX9dMEy0C8e5rn7CL4i3/ECmB36ef9\nwHZ3f9Ld88BtwCkzrGXWmuqDieCu/uG5eksRkXlnygAws01mdgfBPgB1wCPAo0Aj8K1pXvsWgusG\nKA3z7HL3XgB3zwFPmdm60rFnAH6En+GwLaoP1gDq7tM8gIhE13RDQH9xpC/s7pvNbIuZbSZYOfRy\nM3sr0O3u1wEfBL5VmhD+LXDDkb7X4RoJgK4+9QBEJLqmDAB3/+VsXtzdPzbmrgfLHnsCOHc2r3+k\nRgKgs1cBICLRFalN4UcsaijNAagHICIRFskAaB4dAtIcgIhEVyQDYGQ5iC4NAYlIhEUyAJKJOI21\nKQ0BiUikRTIAAJobqunsHaZYnHZNOxGRBSmyAdC2qJpMrqB5ABGJrMgGwJLmWgDauwYrXImISGVE\nOABqANg7zRpDIiILVXQDYFEQAOoBiEhURTcASj2AfZ0KABGJpsgGwKKGKpKJOHsVACISUZENgHgs\nxpLmGvZ1DuhUUBGJpMgGAMDyxbUMDud1KqiIRFLEA6AOgN0d/RWuRERk7kU6AFYsDq4F2N2hU0FF\nJHoiHQDqAYhIlEU6AJa11BIDnm1XAIhI9EQ6AKrSCZa31rF9by+Fgs4EEpFoiXQAAKxd3sBQJs/u\nA5oHEJFoiXwAHL+8EYCndnVXuBIRkbmlAFjRBMCTzyoARCRaIh8Axy2pI52M8/Tu3kqXIiIypyIf\nAIl4nFVL69m1v59sLl/pckRE5kyoAWBmV5vZ3Wa22cxeNMkxnzOz28OsYzprljWSLxTZtke9ABGJ\njtACwMzOB9a5+znAZcA1ExxzMvDSsGqYqfWrFgGwdUdXhSsREZk7YfYALgSuB3D3x4BmM2scc8xV\nwCdCrGFG1h8XTAT7MwoAEYmOZIivvQzYUna7vXRfD4CZvRX4JbBtJi/W3FxLMpmY8pi2toYjKDN4\n3qql9fiOLhY115Ka5n2i5kjbVaamdg2H2nXmwgyAsWIjP5hZC/A24CJg5Uye3DnN3r1tbQ20tx/5\nGP76lYvYsbePex/aNTokJLNvV5mY2jUcatfxpgrEMIeAdhH8xT9iBbC79PPLgDZgE3AdcLqZXR1i\nLdM6ZW0LAA8+sb+SZYiIzJkwA+AW4FIAMzsd2OXuvQDu/kN3P9ndXwy8GnjA3T8UYi3TOnlNM+lU\nnC1b27VDmIhEQmgB4O6bgS1mtpngDKDLzeytZvbqsN5zNtKpBKevb2Nf5yCPbe+sdDkiIqELdQ7A\n3T825q4HJzhmG3BBmHXM1IVnHMc9j+zl5/fv5OQ1LZUuR0QkVJG/ErjcCSuaWLu8kQef2M++rsFK\nlyMiEioFwBgvP/M4isDtv3620qWIiIRKATDGGdYGwM2/eoZcvlDhakREwqMAGCOVTHBi6crg+363\nr8LViIiERwEwgT+55LkA/OMNj9I3mK1wNSIi4VAATGBla93oz1d8aRNDmVwFqxERCYcCYBJf/dOD\ni5S+74t3aK8AEVlwFACTqE4nueYD543e/vdfPFnBakREjj4FwBTqa1Jc+SdnAnDbAzu57o6nKlyR\niMjRowCYxtrljbzponUA3LB5G/u7dYGYiCwMCoAZuOjMVZx98lIAPvK1u3V9gIgsCAqAGXrX75/M\n2uXBhmbv+sLtPPlsd4UrEhGZHQXADMViMd77R6eM3v7HGx6tYDUiIrOnADgMrU01XPuRjaxeWs++\nrkFuvX9HpUsSETliCoDDFI/HeOerTgbgez9/nE9/6z66+zMVrkpE5PApAI7AyrZ63nKJAbBtTy8f\n+vKdPLNX+5CKyLFFAXCELnjhSt74shNHb19742MMDmvJCBE5digAZuHis1Zz7Uc2suHUZexs7+Oz\n/7qF7r7hSpclIjIjCoBZisdjvO0VJ3Hq2hae3d/Px75xD/c8uqfSZYmITEsBcBTE4zE++PoXcNGZ\nxzGcyfONHz/K2z//C3a291W6NBGRSSkAjpJ4LMabLlrP6zcenBf4y2/ey2PbOytYlYjI5BQAR9kl\nZ6/mi+/fQCoZNO0Xvvdrvnb9w9pYRkTmHQVACBbVV/H/P3w+b3jZicRiwdaSV3xpE5/51/vJ5rSO\nkIjMD7FisRjai5vZ1cCLgSLwAXe/r+yxjcDngDzgwDvcfdJvx/b23ikLbWtroL19/p2Lv/fAANfe\n+ChP7uoZve/E45q45KzVnL6+rYKVzcx8bddjndo1HGrX8draGmKTPRZaD8DMzgfWufs5wGXANWMO\n+QZwqbtvABqAS8KqpZKWttTyibecyf97zznU16QAeGJnN1/50W95++d/weaHd9M3mKUQYhCLiEwk\nGeJrXwhcD+Duj5lZs5k1uvvIn8JnlP3cDiwOsZaKa11Uw5euOJef3buDf//vJ0bvv/bGx0Z/fu7q\nRbz7D0+lqS5diRJFJGLCDIBlwJay2+2l+3oARr78zWw5cDFwZYi1zAuxWIxLzl7NJWevZmAoy833\n7mDrM51s3RksLf27Z7r40JfvpKWxitPXtfGS5y1jzbLGClctIgtVmAEw1rhxKDNbAtwAvM/dO6Z6\ncnNzLclkYso3aGtrmFWBc+3dq1pGf87lC3zzxw9z451Pc6BnmJ9v2cnPt+xkwwtWcMXrX0htdapi\ndR5r7XqsULuGQ+06c2EGwC6Cv/hHrAB2j9wws0bgp8An3P2W6V6ss3NgyscXwuTPa85dyx9tWMMj\nTx/gxs3beHxnN3c9uIu7HtxFKhknmyvwxgvXcfGLVs1ZTQuhXecjtWs41K7jTRWIYQbALcBfAV83\ns9OBXe5e/n/mKuBqd785xBqOOfFYjOcdv5jnHb+YvsEs37/tcR55+sDoktPfv+1xvn/b4zz/hMW8\n8cJ1NNdXUZWeumckIjKRsE8D/TzwUqAAXA6cBnQDPwM6gbvLDv+uu39jstc6Vk8DPVqKxSIPP32A\nm3/1zIRXF79+44mcc8pSmuqrjur7LvR2rRS1azjUruNNdRpoqAFwNEU9AMo9sbObW+57hvu9fcLH\nP/i65/P8E1qPyntFqV3nkto1HGrX8aYKgLmcBJaj5MTjmjjxuOcBUCgW+bdbt3LHb3bRUJuiqy/D\n3//HQ6SScU5b10pVKsHvnbWaFa11ZHOF0SUqRETUA1hgduzr48v/+RD7u4dG74vF4PnHL+ahpzoo\nFuGPzl3LxWetojo9ff6rXcOhdg2H2nU8DQFFTLFYZM+BAbr6Mmz+7W42P7yHiRqvpbGKP9iwlpaG\nKk5a00wiPr53oHYNh9o1HGrX8RQAEdc3mKV3IMOyllp6BrLcdPd2Nj+8m/6hg1tYrl3ewEfedDqp\nZJx47ODvi9o1HGrXcKhdx1MAyDiFYpFfPbKX7Xt7ueW+HaP3r15Sz5tevp71qxYBatewqF3DoXYd\nTwEgU8pk83zn1q3c+dDodXpcePpxXLrxBI5bsUjtGgL9voZD7TqeAkBmpFgs8sDWdr5/2+N09AxT\nX5PizJOX8vLTV7J8cV2ly1tQ9PsaDrXreAoAOSzDmTzXbXqK27bsJF8Imv11G0/g7JOW0tJYXeHq\nFgb9voZD7TqeAkCOSKFQ5Nu3buWOXz87ep+tWsS7/uAUmhuO7hXHUaPf13CoXceryIYwcuyLx2P8\n2ZvP5Or3bxhdgM53dPE3376fJ3d1V7g6EZktBYBMq6m+ijdeuI5vfnQjrz3/eLp6h/n8dx7gpnu2\nUygcGz1IERlPS0HIjMViMV55zhqes7SBr17/MD+8/UluvX8HtmoRG09bia1urnSJInIY1AOQw3bq\n8Yv59GVnseF5yygWitz72D7+9ru/5pofPsSBnqHpX0BE5gX1AOSItDbVcNkrT6ZYLLJ1Rxc/uuMp\nfvPEfh7ddoANz1vOqWtbeMGJrcRiQc9BROYfnQUkU5ppuxaLRe58aDc/2vQU3X3B5jU1VQkGh/Os\nWdbAc1c3s37VItavaqro9pbzhX5fw6F2HU/LQUvoYrEY571gBeecugzf0cV9j+3lrt/uAWDbnl62\n7enl5nufIQasWlrPitY6TlzZxMrWOvKFIk11aZY012q5apExtu/ppXVRNXUh/OGkAJCjKpmIc8qa\nFk5Z08L/efl69nYOsqi+iu17e9n6TBdbd3Tx5K5untnbxz2P7D3kuTGgtjpJc0MV1ekkrU3VLG6q\npm1RDSsW19G6qJpFR3nHM5H5rH8oy998+37Oe8EK3vJ7dtRfXwEgoUklExzXVg8wGgoAuXyBju4h\nHt3eyY59fSTjMQYzOfZ3DdHRM8S+rkGy2QJPPDv+WoNF9UFPoat3mGQyzq79/SQTMZa21NJYm6Yq\nlWA4mwcgny+QLxTJF4qkUwlaGqpoaaymrjpJIhEnlYhRJNiHeVFDFclEjL7BLPXVKdoW1bC4qZpk\n4vB6JIVikWyuQDoZJxaLkc0VGM7mKRSL1FYlD/v1JNq6eodHr8YPgwJA5lwyEWdpSy1LW2onPaZY\nLLKrY4C+gQy7OwbY1zXIno4Btu3pYeuOLmqrkgxmguWsc/ki+7uHeLa9/5DXiBFczBaPB1/Ehyse\ni7G4qYolzbVUpxL0D2UpFmFf1yDxWIyqdILegQzZXIFEPLjdN5AlU7pdnU4csuR2MhGnoTbF6iX1\nrFzaQDaTZ2AoS01VkpqqJFXpBIvq06xoraO1qYb6Gs2VRF3fYBaA+ppwvqoVADIvxWIxVrbWAXWH\nXF9QHPkLO5Ugm8uTiMeJx2MUi8Ff+j39mdGx0nQqPnoGUr5QoLN3mAM9wwwM58jni+TyBbK5Arl8\noXRfgep0kv6hLPu7g57Ivs5BHnn6wCG1NdalKcSKDPTmaKxLQ7FIPB6jpz9DW3MNTXVphjJ5hjJ5\nVi2pp6YqSbbU69ndMUBn7zAPPtkxbRtUpxO0NlVTX5Mimy9QKBTp6BmmfzBLQ22KJc21LGupAWLU\n16SoqwmCpK46RVUqTqEYrPQ6nMnTM5ChtamGFa11LG6sIpsv8ti2Awxl86RKvZJlLbWsaK2jpmry\nr4WO7iHSqTgNtenD/D9aeQ892cEDW/dRKEBTfZpT17Zwwsqmcb2yYrFILBb8Tj29u5euvmBhxIHh\nHMVCkZbGalYtqScen/3ZbX2DWZ7a1UNtVZJli2vJ5wvs6xpkOJsnky3w8/uDpdrDGP8HBYAcY2Kx\nGOlUAgiGmMrvTyZiky5Wl4jHaW2qobWp5rDfc3A4x+Bwjoba1Lj3PVy9Axly+SKJdJL2/X2lICsw\nnMmRyRV4dn8/Hd3BUNjeziCAdrb3E4sFPZKm+jSrl9bT05/l8Z3BnMrR1lCbYmVrHUuaa1nSXEMm\nm+fR7Z08sfPgkFxzQxXxGBSK0FCTYtWSehY3VbO09JwlzTUMZvIc6B4inUpQlU5Qkw7+rUolDvnS\nfXZ/P0/t6uY3j+/nQO8wFIOQbaxNBf/WpWmsTVNfm6K+Jhiee2JnNw891UFfqT2r04ngi3l5I7FC\ngZbGajLZPNXpJNXpBD+9Zzt3PbznkM/5k7u3k0zEWNJcy9rlDaxqq+fhbQf43fYulrXUMpTJHbK1\narmaqgTPXd3MqiX1ZPMF0sng8xGLkUrESCWDocjegQzFYhDmXX0ZUsk4S1tqaO8a4vEdXWzd2cVM\nTsRcvbThyP5nTkOngcqU1K7hOJzTa/OFIonSX5vl11Rksnn2dg5SKBQZzuYZGM4xOJSjfyjLcDZP\nvBSWsRikEnGGMnl2dfSzv3uI9q5BTl/XxnAuTwxY2lLLngMDtHcOsufAAB3dQ+O2Ea2rTtI/lKOp\nLk0yEQNixOPQ2Zshlz+8IbZkIkZVKfwyY4bn0qk4mezhD9lNp7mhije87ESWL66jo2eIR546wKPb\nD7C/e+iQIcKlLbV09gxRKMLzT1jM2uUNtHcNUVeTpDqdZHdHP0/s7J40HGYqBqxaUs9Ja4Ie7p6O\nAZLJOIvqq6ivSZHLFzhhRRNLW2pmtRy7TgMVOUaN9Gwmkk4lWLWkPpT3zeby7OscZF/XYPA+bfU0\n1qVHh0fK5fIF9ncP0dE9xL7OgdGeSzoVZ0lzLblcgaFsnuFMbnRobLg0NAWwpLmG41c0ctq6NpYv\nriUWizGUydHTn6FnIBv825+hdyDD/u4hOvuGWdpcy4ueu4RlLbUkEzEGhnMc6BmmmIizfWcX7aWh\nqmIhOJOmrjrFK1/ynNGhlFVL6nnhia1AsOrts/v72d3Rz9LmWp6zrIFCoUihWJxy0r6je4hn9/dT\nW5Ukm8szmMkHJx3kg4n/mqokDTUpKH2eRfVVZLJ5dncM0NJYxYkrK39NjHoAMiW1azjUruFQu45X\nsR6AmV0NvBgoAh9w9/vKHrsI+CyQB25y978OsxYRETlUaCclm9n5wDp3Pwe4DLhmzCHXAK8FNgAX\nm9nJYdUiIiLjhXlVyoXA9QDu/hjQbGaNAGZ2PHDA3Xe4ewG4qXS8iIjMkTCHgJYBW8put5fu6yn9\n21722D7ghKlerLm5luQ0p9+1tYVzqlTUqV3DoXYNh9p15ubyLKCprpqY9oqKzs6BKR/X5E841K7h\nULuGQ+063lSBGOYQ0C6Cv/RHrAB2T/LYytJ9IiIyR8IMgFuASwHM7HRgl7v3Arj7NqDRzNaYWRJ4\nVel4ERGZI6ENAbn7ZjPbYmabgQJwuZm9Feh29+uA9wLfKx3+A3ffGlYtIiIy3jFzIZiIiBxdWpxc\nRCSiFAAiIhGlABARiSgFgIhIRCkAREQiSgEgIhJRCgARkYhSAIiIRNSC3hLSzD4FHAd0Ad9x999U\ntqKFw8yWAb8GVrl7rtL1LARmtgF4D5AGvuDu91e4pAXBzM4B3kHwfXeNu2+Z5imRcUwEgJmdCvwX\ncLW7f6V036S7jY0xCKTQYnPjzLJd/xT45ZwUeoyZRbv2AO8Eng9cACgAysyiXfuBy4HnErSrAqBk\n3geAmdUBXwZuK7tvdLcxMzsJ+CfgHDP7IHBu6bBHgK8DBwhWHv0g8PG5rH0+m2W7Pg78iOCvVSkz\nm3Z190+a2SuA/0sQBFJyFNq1EXgf8LE5Ln1em/cBAAwDrwA+WnbfIbuNmVmzmTW6+98Dfz9ykJld\nCNxOMARUNWcVHxtm065fAU4EXgi8EfjOnFU9/82mXc8GfgrcC3wKeP9cFX0MmE27NgF/C/y5ux+Y\nw5rnvXk/CezuOXcfHHP32B3FRnYbG6sG+BbwRYLegJTMpl3d/f3u/ingN8D3QyvyGDTL39dmgt/T\nLwE/CafCY9Ms2/WjQCNwpZm9NqQSj0nHQg9gJibcUczdbwRunONaFpIpd2pz97fOUR0LzWS/rzcD\nN89xLQvJZO2qod9JzPsewCQgzukBAAAFbklEQVSm2m1MjpzaNRxq13CoXWfpWA2ASXcbk1lRu4ZD\n7RoOtesszfsNYczsDOAqYA2QBZ4FXgN8BHgppd3G3P3BStV4LFK7hkPtGg61azjmfQCIiEg4jtUh\nIBERmSUFgIhIRCkAREQiSgEgIhJRCgARkYhSAIiIRJQCQEJlZheY2Z0hv8fHzOyVYb7HmPd7iZkd\nf5jPOcXM/tvMqsysaGbzYhmW6Woxsx+a2cVzWZPMHQWAHPPc/fPuPpeLp70NmHEAmFmcYMXU97n7\ncGhVhePdwFfNrL7ShcjRNy/+CpFoMLPVwFeBWqAe+Li7/9zMnkuwCmaOYNXGv3D3n5V2dFsLPAf4\nMMGVoD8HXgKsBz7p7v9mZt8C7iw99mPgZ8DZQAPwSnffZWZvJ9gToh3YBFzk7iNrxo/UdzvBCqen\nAS8D3gW8BcgAQ8AbgI3A64CzzOxDwBMTfaYxH/0PgZ3u/tiY96sDvgGsIti06Nvu/jUzqwb+heCq\n152ldrnV3a8te2498F2CFURTwA3u/hkzWwL8M9AE5Amujn3YzD5NsHwypdd8s7tny14vDfwDwTLf\nDcD33P0qd+8wsxsJdtQaXWJZFgb1AGQufQ24yt1fBvwBcG1p+GEZcKW7XwhcAXym7DlrgY1l2/jV\nu/srgMsIlgEY62TgW+7+UoIv8zeUNgP5AvDy0nusn6LGPnc/393zBMuJX+zu5wPbCL40ryu97ofd\n/RdTfKZylzDxKp9XAF2lWl8GfLQ0tPRmIOXuZxPsZDXREMzLS8ecRxCIfaWexueAm0rh9pfAH5fq\nGQDOc/cNwCLg98a83gcI1tLZSBCebzSz55ceu7X0GWSBUQ9A5tJGoMHMPlm6nQWWEKzg+AUz+wzB\nfritZc+5x93L1yu5vfTvdqBlgvfY7+6PjDlmPbDd3feW7v9P4EOT1Li57OcO4CYzKxD8NT7RSpOT\nfabyLUhXMfH6/mcT7FeBuw+a2f3A6QQb7dxeun/PJHModwGfNrN/B24CrnX3QmlTmS+WnvtLStt2\nmlke2GRmOYKtEVvHvN5G4LjSLlsA1QS9gYcI2nHNBDXIMU4BIHNpGHiNu+8vv9PMbiUYcvin0r6v\n5Xs4ZMa8RvkG9BOt/z52g/oYQU+3UHZffooaM6WajgP+DjjF3feZ2d9NcvyEn2mGxi7EFSvdN229\npZpeAJxDMMR0f2lFzJHnjyptNv924Ex37zezH07yOT7t7hM9JguUhoBkLt0JvB7AzFrNbGRMeSnB\nXsMQjLMf7e07nwROMLPm0u1Xz+A5Swh6E/vMrIVgGGakrgLBuDtM/pnK7SDoBYx1D6WhmNJ8wBkE\nG5b/jmBYh9KY/rljn1g6M+eV7n6Xu38E6CvVvJnScI2ZnWtm/0LQvttKX/7PIdhEfWwbl3+OuJl9\nsfS5IZiD2TZRI8mxTQEgc+kK4NVmtolg2OIXpfuvAr5tZj8j+CI6YGZXHa03dfcOgnmFu8zspwRf\nyGN7CmP9BnjczO4lmBz9JPA2MzuXYEz862b2mik+U7mbGT/mDsEm5w1mdkfpeZ92920Ew0KtZnY3\nwcTrpgnqdeDDZrapNHl9i7tvB64ELii95mcJ2vYWoLE0lPRxgv2GP2Fm5XMh/0Awj3A3QTB1le2f\nexHaqWxB0nLQEglm9sfAT9z9gJn9KWDu/u45eu84wV/2bxp7JtAkx68EXuLu/1F67gPAe9397pBL\nnaiWxcCvgNO02crCox6AREU98Asz+yXwCuBv5uqN3b0A/DHB+fQzGd7qIjgL517gbuCnlfjyL/k6\nwfUL+vJfgNQDEBGJKPUAREQiSgEgIhJRCgARkYhSAIiIRJQCQEQkohQAIiIR9T9nCeh/3+XujAAA\nAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f56d7390550>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "I_kETQD66Rti",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "lr = 1e-3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mUE06J6C6Yfq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "60ebf0ed-c8ec-47c3-f312-e5943d300ea1"
      },
      "cell_type": "code",
      "source": [
        "m.fit(lr, 3, metrics=[exp_rmspe]) # применяем свою метрику exp_rmspe"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f9436e915fbf4b1ca2d02922eea04dd7",
              "version_minor": 0,
              "version_major": 2
            },
            "text/plain": [
              "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "epoch      trn_loss   val_loss   exp_rmspe  \n",
            "    0      0.014144   8e-06      0.00288   \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "    1      0.010493   0.001549   0.040145  \n",
            "  0%|          | 0/6597 [00:00<?, ?it/s, loss=0.0104]"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "    2      0.009488   0.001056   0.031969  \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[array([0.00106]), 0.031968637946800375]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 108
        }
      ]
    },
    {
      "metadata": {
        "id": "A55857X76v64",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m.fit(lr, 5, metrics=[exp_rmspe], cycle_len=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "W2xLZH8e7QII",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m.fit(lr, 3, metrics=[exp_rmspe], cycle_len=4)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8GpXuAg077ML",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m.save('val0')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uZgvLxmx78Ir",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m.load('val0')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4KbTd3uY9CtQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x,y=m.predict_with_targs()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "4vwhEalT9FvB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "69020435-d026-4a44-9059-4ce7ca57c6e4"
      },
      "cell_type": "code",
      "source": [
        "exp_rmspe(x,y)"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.031968637946800375"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 112
        }
      ]
    },
    {
      "metadata": {
        "id": "d5cPIA8Y9IWr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "pred_test=m.predict(True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "SrER9xGv9Mr4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "pred_test = np.exp(pred_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eanTtB-k9RLd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined_test['Sales']=pred_test"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bsMotp4R9T9d",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "csv_fn=f'{PATH}tmp/sub.csv'"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H9pVAxB_9W7v",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "joined_test[['Id','Sales']].to_csv(csv_fn, index=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0-Cbr8Cp9Zej",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "eab160a4-d24d-488b-b05b-4eb09f92c0a3"
      },
      "cell_type": "code",
      "source": [
        "f'{PATH}tmp/sub.csv'"
      ],
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'data/rossmann/tmp/sub.csv'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 121
        }
      ]
    },
    {
      "metadata": {
        "id": "lGObKXIa-bRf",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Случайный лес**"
      ]
    },
    {
      "metadata": {
        "id": "AHh7nrBz9dEr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.ensemble import RandomForestRegressor"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "MZ-XMilV-B3m",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "((val,trn), (y_val,y_trn)) = split_by_idx(val_idx, df.values, yl)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "yon73EpR-FPN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "m = RandomForestRegressor(n_estimators=40, max_features=0.99, min_samples_leaf=2, n_jobs=-1, oob_score=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5pGiUmu4-L3H",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "3d340c20-1938-4f59-84fd-b24cea7c3436"
      },
      "cell_type": "code",
      "source": [
        "m.fit(trn, y_trn)"
      ],
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
              "           max_features=0.99, max_leaf_nodes=None,\n",
              "           min_impurity_decrease=0.0, min_impurity_split=None,\n",
              "           min_samples_leaf=2, min_samples_split=2,\n",
              "           min_weight_fraction_leaf=0.0, n_estimators=40, n_jobs=-1,\n",
              "           oob_score=True, random_state=None, verbose=0, warm_start=False)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "metadata": {
        "id": "6FNmP6xe-NEz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c1a98c25-cb93-4a0b-afb3-e385e9d6d18e"
      },
      "cell_type": "code",
      "source": [
        "preds = m.predict(val)\n",
        "m.score(trn, y_trn), m.score(val, y_val), m.oob_score_, exp_rmspe(preds, y_val)"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.9825751372036499, 0.0, 0.9260912509570574, 0.08346625483525645)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 126
        }
      ]
    },
    {
      "metadata": {
        "id": "sa_oaUuY__87",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VIQzi5OCAUZ6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}