3_data_exploration_and_cleaning.ipynb

{
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.6.6",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kaggle": {
      "accelerator": "none",
      "dataSources": [
        {
          "sourceId": 3136,
          "databundleVersionId": 26502,
          "sourceType": "competition"
        }
      ],
      "dockerImageVersionId": 24140,
      "isInternetEnabled": false,
      "language": "python",
      "sourceType": "notebook",
      "isGpuEnabled": false
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat_minor": 0,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Data Exploration and Cleaning"
      ],
      "metadata": {
        "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
        "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
        "trusted": true,
        "id": "0RfiPNpWpnof"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first part of any data analysis or predictive modeling task is an initial exploration of the data. Even if you collected the data yourself and you already have a list of questions in mind that you want to answer, it is important to explore the data before doing any serious analysis, since oddities in the data can cause bugs and muddle your results. Before exploring deeper questions, you have to answer many simpler ones about the form and quality of data. That said, it is important to go into your initial data exploration with a big picture question in mind since the goal of your analysis should inform how you prepare the data.\n",
        "\n",
        "This lesson aims to raise some of the questions you should consider when you look at a new data set for the first time and show how to perform various Python operations related to those questions. We are going to cover a lot of ground in this lesson, touching briefly on many topics from data cleaning to graphing to feature engineering. We will cover many of these topics in future lessons in greater detail.\n",
        "\n",
        "In this lesson, we will explore the Titanic disaster training set."
      ],
      "metadata": {
        "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
        "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
        "id": "gLz107AFpnoj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exploring The Variables"
      ],
      "metadata": {
        "id": "GLKNfnFgpnok"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The first step in exploratory analysis is reading in the data and then exploring the variables. It is important to get a sense of how many variables and cases there are, the data types of the variables and the range of values they take on.\n",
        "\n",
        "We'll start by reading in the data:"
      ],
      "metadata": {
        "id": "b-9QaKp5pnol"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Load in some packages\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import os"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:27:41.543379Z",
          "iopub.execute_input": "2024-08-22T08:27:41.543687Z",
          "iopub.status.idle": "2024-08-22T08:27:41.548671Z",
          "shell.execute_reply.started": "2024-08-22T08:27:41.543640Z",
          "shell.execute_reply": "2024-08-22T08:27:41.547890Z"
        },
        "trusted": true,
        "id": "00KEbFuVpnol"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train = pd.read_csv(\"../input/train.csv\")      # Read the data"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:27:47.386440Z",
          "iopub.execute_input": "2024-08-22T08:27:47.386757Z",
          "iopub.status.idle": "2024-08-22T08:27:47.426087Z",
          "shell.execute_reply.started": "2024-08-22T08:27:47.386698Z",
          "shell.execute_reply": "2024-08-22T08:27:47.424919Z"
        },
        "trusted": true,
        "id": "UIucp0F4pnon"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It's a good idea to start off by checking the dimensions of your data set with df.shape and the variable data types of df.dtypes."
      ],
      "metadata": {
        "id": "Mak4Pfpipnon"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.shape      # Check dimensions"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:27:50.290391Z",
          "iopub.execute_input": "2024-08-22T08:27:50.290686Z",
          "iopub.status.idle": "2024-08-22T08:27:50.297465Z",
          "shell.execute_reply.started": "2024-08-22T08:27:50.290640Z",
          "shell.execute_reply": "2024-08-22T08:27:50.296589Z"
        },
        "trusted": true,
        "id": "mZqXGIIOpnoo",
        "outputId": "0feef867-b0bc-4785-8fe1-4f8cfa927a66"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 4,
          "output_type": "execute_result",
          "data": {
            "text/plain": "(891, 12)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.dtypes"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:27:54.303339Z",
          "iopub.execute_input": "2024-08-22T08:27:54.303649Z",
          "iopub.status.idle": "2024-08-22T08:27:54.314171Z",
          "shell.execute_reply.started": "2024-08-22T08:27:54.303601Z",
          "shell.execute_reply": "2024-08-22T08:27:54.313153Z"
        },
        "trusted": true,
        "id": "6fY3sVb7pnop",
        "outputId": "4f9af7d8-66ea-40da-e9be-4f9e5c7f9fe4"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 5,
          "output_type": "execute_result",
          "data": {
            "text/plain": "PassengerId      int64\nSurvived         int64\nPclass           int64\nName            object\nSex             object\nAge            float64\nSibSp            int64\nParch            int64\nTicket          object\nFare           float64\nCabin           object\nEmbarked        object\ndtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The output shows us that we're working with a set of 891 records and 12 columns. Several of the column variables are encoded as numeric data types (ints and floats) but a few of them are encoded as \"object\". Let's check the head of the data to get a better sense of what the variables look like:"
      ],
      "metadata": {
        "id": "C29CIpRHpnop"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.head(5)  # Check the first 5 rows"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:28:03.211652Z",
          "iopub.execute_input": "2024-08-22T08:28:03.211946Z",
          "iopub.status.idle": "2024-08-22T08:28:03.250899Z",
          "shell.execute_reply.started": "2024-08-22T08:28:03.211901Z",
          "shell.execute_reply": "2024-08-22T08:28:03.250212Z"
        },
        "trusted": true,
        "id": "CIC9zAGEpnop",
        "outputId": "6afa3c39-a421-440f-9f7d-20f39be35f98"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 6,
          "output_type": "execute_result",
          "data": {
            "text/plain": "   PassengerId  Survived  Pclass    ...        Fare Cabin  Embarked\n0            1         0       3    ...      7.2500   NaN         S\n1            2         1       1    ...     71.2833   C85         C\n2            3         1       3    ...      7.9250   NaN         S\n3            4         1       1    ...     53.1000  C123         S\n4            5         0       3    ...      8.0500   NaN         S\n\n[5 rows x 12 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PassengerId</th>\n      <th>Survived</th>\n      <th>Pclass</th>\n      <th>Name</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Ticket</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0</td>\n      <td>3</td>\n      <td>Braund, Mr. Owen Harris</td>\n      <td>male</td>\n      <td>22.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>A/5 21171</td>\n      <td>7.2500</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n      <td>female</td>\n      <td>38.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>PC 17599</td>\n      <td>71.2833</td>\n      <td>C85</td>\n      <td>C</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>1</td>\n      <td>3</td>\n      <td>Heikkinen, Miss. Laina</td>\n      <td>female</td>\n      <td>26.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>STON/O2. 3101282</td>\n      <td>7.9250</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>4</td>\n      <td>1</td>\n      <td>1</td>\n      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n      <td>female</td>\n      <td>35.0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>113803</td>\n      <td>53.1000</td>\n      <td>C123</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>5</td>\n      <td>0</td>\n      <td>3</td>\n      <td>Allen, Mr. William Henry</td>\n      <td>male</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>373450</td>\n      <td>8.0500</td>\n      <td>NaN</td>\n      <td>S</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It appears we have a mixture of numeric columns and columns with text data. In data analysis, variables that split records into a fixed number of unique categories, such as Sex, are known as categorical variables. Pandas will attempt to interpret categorical variables as such when you load data, but you can specifically convert a variable to categorical if necessary, as we'll see later.\n",
        "\n",
        "Note that if you're using a local Python environment with Spyder as your Python editor, you will see the data in the variable explorer pane, showing its type (DataFrame) and size. If you double click on the data in the variable explorer, you can see the data in a spreadsheet-like view that lets you sort by columns and edit values directly. You shouldn't rely too much on the variable explorer, however, since it doesn't work well when dealing with large data sets.\n",
        "\n",
        "After getting a sense of the data's structure, it is a good idea to look at a statistical summary of the variables with df.describe():"
      ],
      "metadata": {
        "id": "dtuUgUH7pnoq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:28:06.338757Z",
          "iopub.execute_input": "2024-08-22T08:28:06.339101Z",
          "iopub.status.idle": "2024-08-22T08:28:06.388891Z",
          "shell.execute_reply.started": "2024-08-22T08:28:06.339040Z",
          "shell.execute_reply": "2024-08-22T08:28:06.387966Z"
        },
        "trusted": true,
        "id": "zoqn3ZKfpnoq",
        "outputId": "487fc08b-5e9d-4a00-889b-4573abfcab15"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 7,
          "output_type": "execute_result",
          "data": {
            "text/plain": "       PassengerId    Survived     ...           Parch        Fare\ncount   891.000000  891.000000     ...      891.000000  891.000000\nmean    446.000000    0.383838     ...        0.381594   32.204208\nstd     257.353842    0.486592     ...        0.806057   49.693429\nmin       1.000000    0.000000     ...        0.000000    0.000000\n25%     223.500000    0.000000     ...        0.000000    7.910400\n50%     446.000000    0.000000     ...        0.000000   14.454200\n75%     668.500000    1.000000     ...        0.000000   31.000000\nmax     891.000000    1.000000     ...        6.000000  512.329200\n\n[8 rows x 7 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>PassengerId</th>\n      <th>Survived</th>\n      <th>Pclass</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>891.000000</td>\n      <td>891.000000</td>\n      <td>891.000000</td>\n      <td>714.000000</td>\n      <td>891.000000</td>\n      <td>891.000000</td>\n      <td>891.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>446.000000</td>\n      <td>0.383838</td>\n      <td>2.308642</td>\n      <td>29.699118</td>\n      <td>0.523008</td>\n      <td>0.381594</td>\n      <td>32.204208</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>257.353842</td>\n      <td>0.486592</td>\n      <td>0.836071</td>\n      <td>14.526497</td>\n      <td>1.102743</td>\n      <td>0.806057</td>\n      <td>49.693429</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>1.000000</td>\n      <td>0.000000</td>\n      <td>1.000000</td>\n      <td>0.420000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>223.500000</td>\n      <td>0.000000</td>\n      <td>2.000000</td>\n      <td>20.125000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>7.910400</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>446.000000</td>\n      <td>0.000000</td>\n      <td>3.000000</td>\n      <td>28.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>14.454200</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>668.500000</td>\n      <td>1.000000</td>\n      <td>3.000000</td>\n      <td>38.000000</td>\n      <td>1.000000</td>\n      <td>0.000000</td>\n      <td>31.000000</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>891.000000</td>\n      <td>1.000000</td>\n      <td>3.000000</td>\n      <td>80.000000</td>\n      <td>8.000000</td>\n      <td>6.000000</td>\n      <td>512.329200</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "j5dQTedRpnoq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice that non-numeric columns are dropped from the statistical summary provided by df.describe().\n",
        "\n",
        "We can get a summary of the categorical variables by passing only those columns to describe():"
      ],
      "metadata": {
        "id": "6JxdkWldpnoq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "categorical = titanic_train.dtypes[titanic_train.dtypes == \"object\"].index\n",
        "print(categorical)\n",
        "\n",
        "titanic_train[categorical].describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:29:26.959536Z",
          "iopub.execute_input": "2024-08-22T08:29:26.959890Z",
          "iopub.status.idle": "2024-08-22T08:29:27.001455Z",
          "shell.execute_reply.started": "2024-08-22T08:29:26.959839Z",
          "shell.execute_reply": "2024-08-22T08:29:27.000359Z"
        },
        "trusted": true,
        "id": "c55_DKhrpnor",
        "outputId": "af3b31a1-6641-426c-d8aa-c6e1d0548638"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 9,
          "output_type": "execute_result",
          "data": {
            "text/plain": "                   Name   Sex    Ticket        Cabin Embarked\ncount               891   891       891          204      889\nunique              891     2       681          147        3\ntop     Elias, Mr. Dibo  male  CA. 2343  C23 C25 C27        S\nfreq                  1   577         7            4      644",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Name</th>\n      <th>Sex</th>\n      <th>Ticket</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>891</td>\n      <td>891</td>\n      <td>891</td>\n      <td>204</td>\n      <td>889</td>\n    </tr>\n    <tr>\n      <th>unique</th>\n      <td>891</td>\n      <td>2</td>\n      <td>681</td>\n      <td>147</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>top</th>\n      <td>Elias, Mr. Dibo</td>\n      <td>male</td>\n      <td>CA. 2343</td>\n      <td>C23 C25 C27</td>\n      <td>S</td>\n    </tr>\n    <tr>\n      <th>freq</th>\n      <td>1</td>\n      <td>577</td>\n      <td>7</td>\n      <td>4</td>\n      <td>644</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The categorical variable summary shows the count of non-NaN records, the number of unique categories, the most frequently occurring value and the number of occurrences of the most frequent value.\n",
        "\n",
        "Although describe() gives a concise overview of each variable, it does not necessarily give us enough information to determine what each variable means. Certain variables like \"Age\" and \"Fare\" are self-explanatory, while others like \"SibSp\" and \"Parch\" are not. Whoever collects or provides data for download should also provide a list of variable descriptions. In this case, Kaggle provides a list of descriptions on the data download page:"
      ],
      "metadata": {
        "id": "aWgQ7RPIpnor"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# VARIABLE DESCRIPTIONS:\n",
        "# survival        Survival\n",
        "#                 (0 = No; 1 = Yes)\n",
        "# pclass          Passenger Class\n",
        "#                 (1 = 1st; 2 = 2nd; 3 = 3rd)\n",
        "# name            Name\n",
        "# sex             Sex\n",
        "# age             Age\n",
        "# sibsp           Number of Siblings/Spouses Aboard\n",
        "# parch           Number of Parents/Children Aboard\n",
        "# ticket          Ticket Number\n",
        "# fare            Passenger Fare\n",
        "# cabin           Cabin\n",
        "# embarked        Port of Embarkation\n",
        "#                 (C = Cherbourg; Q = Queenstown; S = Southampton)"
      ],
      "metadata": {
        "trusted": true,
        "id": "0MAullKypnor"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After looking at the data for the first time, you should ask yourself a few questions:\n",
        "\n",
        "1. Do I need all of the variables?\n",
        "2. Should I transform any variables?\n",
        "3. Are there NA values, outliers or other strange values?\n",
        "4. Should I create new variables?\n",
        "\n",
        "For the rest of this lesson we will address each of these questions in the context of this data set."
      ],
      "metadata": {
        "id": "87Wq-ynBpnor"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Do I Need All of the Variables?"
      ],
      "metadata": {
        "id": "vYhJmRZvpnor"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Getting rid of unnecessary variables is a good first step when dealing with any data set, since dropping variables reduces complexity and can make computation on the data faster. Whether you should get rid of a variable or not will depend on size of the data set and the goal of your analysis. With a data set as small as the Titanic data, there's no real need to drop variables from a computing perspective (we have plenty of memory and processing power to deal with such a small data set) but it can still be helpful to drop variables that will only distract from your goal.\n",
        "\n",
        "This data set is provided in conjunction with a predictive modeling competition where the goal is to use the training data to predict whether passengers of the titanic listed in a second data set survived or not. We won't be dealing with the second data set (known the test set) right now, but we will revisit this competition and make predictions in a future lesson on predictive modeling.\n",
        "\n",
        "Let's go through each variable and consider whether we should keep it or not in the context of predicting survival:\n",
        "\"PassengerId\" is just a number assigned to each passenger. It is nothing more than an arbitrary identifier; we could keep it for identification purposes, but let's remove it anyway:"
      ],
      "metadata": {
        "id": "wkCG4RVhpnor"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "del titanic_train[\"PassengerId\"]     # Remove PassengerId"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:12.278346Z",
          "iopub.execute_input": "2024-08-22T08:30:12.278648Z",
          "iopub.status.idle": "2024-08-22T08:30:12.283177Z",
          "shell.execute_reply.started": "2024-08-22T08:30:12.278603Z",
          "shell.execute_reply": "2024-08-22T08:30:12.282177Z"
        },
        "trusted": true,
        "id": "F09CQGgRpnos"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\"Survived\" indicates whether each passenger lived or died. Since predicting survival is our goal, we definitely need to keep it.\n",
        "\n",
        "Features that describe passengers numerically or group them into a few broad categories could be useful for predicting survival. The variables Pclass, Sex, Age, SibSp, Parch, Fare and Embarked appear to fit this description, so let's keep all of them.\n",
        "\n",
        "We have 3 more features to consider: Name, Ticket and Cabin.\n",
        "\n",
        "\"Name\" appears to be a character string of the name of each passenger. Let's look at name a little closer:"
      ],
      "metadata": {
        "id": "ww5YCq6spnos"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sorted(titanic_train[\"Name\"])[0:15]   # Check the first 15 sorted names"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:16.647338Z",
          "iopub.execute_input": "2024-08-22T08:30:16.647635Z",
          "iopub.status.idle": "2024-08-22T08:30:16.653860Z",
          "shell.execute_reply.started": "2024-08-22T08:30:16.647589Z",
          "shell.execute_reply": "2024-08-22T08:30:16.653094Z"
        },
        "trusted": true,
        "id": "P8qUdfS2pnos",
        "outputId": "b8674b4a-8508-4f56-bba5-2328ee04b59a"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 11,
          "output_type": "execute_result",
          "data": {
            "text/plain": "['Abbing, Mr. Anthony',\n 'Abbott, Mr. Rossmore Edward',\n 'Abbott, Mrs. Stanton (Rosa Hunt)',\n 'Abelson, Mr. Samuel',\n 'Abelson, Mrs. Samuel (Hannah Wizosky)',\n 'Adahl, Mr. Mauritz Nils Martin',\n 'Adams, Mr. John',\n 'Ahlin, Mrs. Johan (Johanna Persdotter Larsson)',\n 'Aks, Mrs. Sam (Leah Rosen)',\n 'Albimona, Mr. Nassef Cassem',\n 'Alexander, Mr. William',\n 'Alhomaki, Mr. Ilmari Rudolf',\n 'Ali, Mr. Ahmed',\n 'Ali, Mr. William',\n 'Allen, Miss. Elisabeth Walton']"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Name\"].describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:22.140340Z",
          "iopub.execute_input": "2024-08-22T08:30:22.140638Z",
          "iopub.status.idle": "2024-08-22T08:30:22.153559Z",
          "shell.execute_reply.started": "2024-08-22T08:30:22.140591Z",
          "shell.execute_reply": "2024-08-22T08:30:22.152366Z"
        },
        "trusted": true,
        "id": "S8-KdMlJpnos",
        "outputId": "6d382d5e-7c4e-40c4-c068-d9cfd249c0bd"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 12,
          "output_type": "execute_result",
          "data": {
            "text/plain": "count                 891\nunique                891\ntop       Elias, Mr. Dibo\nfreq                    1\nName: Name, dtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "From the output above, we see that the Name variable has 889 unique values. Since there are 889 rows in the data set we know each name is unique. It appears that married women have their maiden names listed in parentheses. In general, a categorical variable that is unique to each case isn't useful for prediction. We could extract last names to try to group family members together, but even then the number of categories would be very large. In addition, the Parch and SibSp variables already contain some information about family relationships, so from the perspective of predictive modeling, the Name variable could be removed. On the other hand, it can be nice to have some way to uniquely identify particular cases and names are interesting from a personal and historical perspective, so let's keep Name, knowing that we won't actually use it in any predictive models we make.\n",
        "\n",
        "Next, let's look closer at \"Ticket\":"
      ],
      "metadata": {
        "id": "-nzeLywapnot"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Ticket\"][0:15]       # Check the first 15 tickets"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:27.262783Z",
          "iopub.execute_input": "2024-08-22T08:30:27.263107Z",
          "iopub.status.idle": "2024-08-22T08:30:27.270328Z",
          "shell.execute_reply.started": "2024-08-22T08:30:27.263058Z",
          "shell.execute_reply": "2024-08-22T08:30:27.269321Z"
        },
        "trusted": true,
        "id": "sG7TvFaipnot",
        "outputId": "6d165d98-f26f-46c9-857b-6cebea771077"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 13,
          "output_type": "execute_result",
          "data": {
            "text/plain": "0            A/5 21171\n1             PC 17599\n2     STON/O2. 3101282\n3               113803\n4               373450\n5               330877\n6                17463\n7               349909\n8               347742\n9               237736\n10             PP 9549\n11              113783\n12           A/5. 2151\n13              347082\n14              350406\nName: Ticket, dtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Ticket\"].describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:39.284793Z",
          "iopub.execute_input": "2024-08-22T08:30:39.285136Z",
          "iopub.status.idle": "2024-08-22T08:30:39.297060Z",
          "shell.execute_reply.started": "2024-08-22T08:30:39.285082Z",
          "shell.execute_reply": "2024-08-22T08:30:39.296076Z"
        },
        "trusted": true,
        "id": "v_MLdQhepnot",
        "outputId": "df933340-ae1c-4c22-ed62-5dcfc74d341f"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 14,
          "output_type": "execute_result",
          "data": {
            "text/plain": "count          891\nunique         681\ntop       CA. 2343\nfreq             7\nName: Ticket, dtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Ticket has 680 unique values: almost as many as there are passengers. Categorical variables with almost as many levels as there are records are often not very useful for prediction. We could try to reduce the number of levels by grouping certain tickets together, but the ticket numbers don't appear to follow any logical pattern we could use for grouping. Let's remove it:"
      ],
      "metadata": {
        "id": "GaHW-_jRpnot"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "del titanic_train[\"Ticket\"]        # Remove Ticket"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:46.417965Z",
          "iopub.execute_input": "2024-08-22T08:30:46.418282Z",
          "iopub.status.idle": "2024-08-22T08:30:46.422804Z",
          "shell.execute_reply.started": "2024-08-22T08:30:46.418234Z",
          "shell.execute_reply": "2024-08-22T08:30:46.421859Z"
        },
        "trusted": true,
        "id": "6SDAHSijpnot"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally let's consider the \"Cabin\" variable:"
      ],
      "metadata": {
        "id": "a0k9pT2opnot"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Cabin\"][0:15]       # Check the first 15 tickets"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:30:50.040327Z",
          "iopub.execute_input": "2024-08-22T08:30:50.040612Z",
          "iopub.status.idle": "2024-08-22T08:30:50.047845Z",
          "shell.execute_reply.started": "2024-08-22T08:30:50.040568Z",
          "shell.execute_reply": "2024-08-22T08:30:50.046884Z"
        },
        "trusted": true,
        "id": "Pjr3s0LOpnou",
        "outputId": "9212579a-dd32-4c06-8958-f83c5757c306"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 16,
          "output_type": "execute_result",
          "data": {
            "text/plain": "0      NaN\n1      C85\n2      NaN\n3     C123\n4      NaN\n5      NaN\n6      E46\n7      NaN\n8      NaN\n9      NaN\n10      G6\n11    C103\n12     NaN\n13     NaN\n14     NaN\nName: Cabin, dtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Cabin\"].describe()  # Check number of unique cabins"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-21T15:08:08.407365Z",
          "iopub.execute_input": "2024-08-21T15:08:08.407728Z",
          "iopub.status.idle": "2024-08-21T15:08:08.420358Z",
          "shell.execute_reply.started": "2024-08-21T15:08:08.407676Z",
          "shell.execute_reply": "2024-08-21T15:08:08.419019Z"
        },
        "trusted": true,
        "id": "U268pFedpnou",
        "outputId": "661fadf1-1028-436f-afce-a9399bfb82b6"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 19,
          "output_type": "execute_result",
          "data": {
            "text/plain": "count         204\nunique        147\ntop       B96 B98\nfreq            4\nName: Cabin, dtype: object"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Cabin also has 145 unique values, which indicates it may not be particularly useful for prediction. On the other hand, the names of the levels for the cabin variable seem to have a regular structure: each starts with a capital letter followed by a number. We could use that structure to reduce the number of levels to make categories large enough that they might be useful for prediction. Let's Keep Cabin for now.\n",
        "\n",
        "As you might have noticed, removing variables is often more of an art than a science. It is easiest to start simple: don't be afraid to remove (or simply ignore) confusing, messy or otherwise troublesome variables temporarily when you're just getting starting with an analysis or predictive modeling task. Data projects are iterative processes: you can start with a simple analysis or model using only a few variables and then expand later by adding more and more of the other variables you initially ignored or removed."
      ],
      "metadata": {
        "id": "542Tbhq7pnou"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Should I Transform Any Variables?"
      ],
      "metadata": {
        "id": "gBjL-Mj5pnou"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "When you first load a data set, some of the variables may be encoded as data types that don't fit well with what the data really is or what it means.\n",
        "\n",
        "For instance, Survived is just an integer variable that takes on the value 0 or 1 depending on whether a passenger died or survived respectively. Variables that indicate a state or the presence or absence of something with the numbers 0 and 1 are sometimes called indicator variables or dummy variables (0 indicates absence and 1 indicates presence.). Indicator variables are essentially just a shorthand for encoding a categorical variable with 2 levels. We could instead encode Survived as a categorical variable with more descriptive categories:"
      ],
      "metadata": {
        "id": "8tkf3JFRpnou"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "new_survived = pd.Categorical(titanic_train[\"Survived\"])\n",
        "new_survived.describe()\n",
        "new_survived = new_survived.rename_categories([\"Died\",\"Survived\"])\n",
        "\n",
        "new_survived.describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:32:33.369247Z",
          "iopub.execute_input": "2024-08-22T08:32:33.369561Z",
          "iopub.status.idle": "2024-08-22T08:32:33.390306Z",
          "shell.execute_reply.started": "2024-08-22T08:32:33.369506Z",
          "shell.execute_reply": "2024-08-22T08:32:33.389342Z"
        },
        "trusted": true,
        "id": "cLhheSHnpnov",
        "outputId": "1119a997-e699-4614-a9d8-86157baa2791"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 22,
          "output_type": "execute_result",
          "data": {
            "text/plain": "            counts     freqs\ncategories                  \nDied           549  0.616162\nSurvived       342  0.383838",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>counts</th>\n      <th>freqs</th>\n    </tr>\n    <tr>\n      <th>categories</th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>Died</th>\n      <td>549</td>\n      <td>0.616162</td>\n    </tr>\n    <tr>\n      <th>Survived</th>\n      <td>342</td>\n      <td>0.383838</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Survived looks a little nicer as categorical variable with appropriate category names, but even so, we're not going to change it. Why not? If you remember, our goal with this data set is predicting survival for the Kaggle competition. It turns out that when submitting predictions for the competition, the predictions need to be encoded as 0 or 1. It would only complicate things to transform Survived, only to convert it back to 0 and 1 later.\n",
        "\n",
        "There's one more variable that has a questionable data encoding: Pclass. Pclass is an integer that indicates a passenger's class, with 1 being first class, 2 being second class and 3 being third class. Passenger class is a category, so it doesn't make a lot of sense to encode it as a numeric variable. What's more 1st class would be considered \"above\" or \"higher\" than second class, but when encoded as an integer, 1 comes before 2. We can fix this by transforming Pclass into an ordered categorical variable:"
      ],
      "metadata": {
        "id": "jt63AST_pnov"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "new_Pclass = pd.Categorical(titanic_train[\"Pclass\"],\n",
        "                           ordered=True)\n",
        "\n",
        "new_Pclass = new_Pclass.rename_categories([\"Class1\",\"Class2\",\"Class3\"])\n",
        "\n",
        "new_Pclass.describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:38:01.194834Z",
          "iopub.execute_input": "2024-08-22T08:38:01.195203Z",
          "iopub.status.idle": "2024-08-22T08:38:01.213410Z",
          "shell.execute_reply.started": "2024-08-22T08:38:01.195151Z",
          "shell.execute_reply": "2024-08-22T08:38:01.212555Z"
        },
        "trusted": true,
        "id": "d6jblWA0pnov",
        "outputId": "89bbb81c-ed59-45d8-b301-a660a6e89625"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 25,
          "output_type": "execute_result",
          "data": {
            "text/plain": "            counts     freqs\ncategories                  \nClass1         216  0.242424\nClass2         184  0.206510\nClass3         491  0.551066",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>counts</th>\n      <th>freqs</th>\n    </tr>\n    <tr>\n      <th>categories</th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>Class1</th>\n      <td>216</td>\n      <td>0.242424</td>\n    </tr>\n    <tr>\n      <th>Class2</th>\n      <td>184</td>\n      <td>0.206510</td>\n    </tr>\n    <tr>\n      <th>Class3</th>\n      <td>491</td>\n      <td>0.551066</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Pclass\"] = new_Pclass"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:46:41.331581Z",
          "iopub.execute_input": "2024-08-22T08:46:41.331910Z",
          "iopub.status.idle": "2024-08-22T08:46:41.336383Z",
          "shell.execute_reply.started": "2024-08-22T08:46:41.331860Z",
          "shell.execute_reply": "2024-08-22T08:46:41.335283Z"
        },
        "trusted": true,
        "id": "bWAuZM5vpno0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now it's time to revisit the Cabin variable. It appears that each Cabin is in a general section of the ship indicated by the capital letter at the start of each factor level:"
      ],
      "metadata": {
        "id": "cBjXzs2Zpno0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Cabin\"].unique()   # Check unique cabins"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:38:23.330541Z",
          "iopub.execute_input": "2024-08-22T08:38:23.330840Z",
          "iopub.status.idle": "2024-08-22T08:38:23.337151Z",
          "shell.execute_reply.started": "2024-08-22T08:38:23.330793Z",
          "shell.execute_reply": "2024-08-22T08:38:23.336204Z"
        },
        "trusted": true,
        "id": "WmBa5vIxpno0",
        "outputId": "43281d99-91ae-4e84-a93b-0e3a103b6529"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 27,
          "output_type": "execute_result",
          "data": {
            "text/plain": "array([nan, 'C85', 'C123', 'E46', 'G6', 'C103', 'D56', 'A6',\n       'C23 C25 C27', 'B78', 'D33', 'B30', 'C52', 'B28', 'C83', 'F33',\n       'F G73', 'E31', 'A5', 'D10 D12', 'D26', 'C110', 'B58 B60', 'E101',\n       'F E69', 'D47', 'B86', 'F2', 'C2', 'E33', 'B19', 'A7', 'C49', 'F4',\n       'A32', 'B4', 'B80', 'A31', 'D36', 'D15', 'C93', 'C78', 'D35',\n       'C87', 'B77', 'E67', 'B94', 'C125', 'C99', 'C118', 'D7', 'A19',\n       'B49', 'D', 'C22 C26', 'C106', 'C65', 'E36', 'C54',\n       'B57 B59 B63 B66', 'C7', 'E34', 'C32', 'B18', 'C124', 'C91', 'E40',\n       'T', 'C128', 'D37', 'B35', 'E50', 'C82', 'B96 B98', 'E10', 'E44',\n       'A34', 'C104', 'C111', 'C92', 'E38', 'D21', 'E12', 'E63', 'A14',\n       'B37', 'C30', 'D20', 'B79', 'E25', 'D46', 'B73', 'C95', 'B38',\n       'B39', 'B22', 'C86', 'C70', 'A16', 'C101', 'C68', 'A10', 'E68',\n       'B41', 'A20', 'D19', 'D50', 'D9', 'A23', 'B50', 'A26', 'D48',\n       'E58', 'C126', 'B71', 'B51 B53 B55', 'D49', 'B5', 'B20', 'F G63',\n       'C62 C64', 'E24', 'C90', 'C45', 'E8', 'B101', 'D45', 'C46', 'D30',\n       'E121', 'D11', 'E77', 'F38', 'B3', 'D6', 'B82 B84', 'D17', 'A36',\n       'B102', 'B69', 'E49', 'C47', 'D28', 'E17', 'A24', 'C50', 'B42',\n       'C148'], dtype=object)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we grouped cabin just by this letter, we could reduce the number of levels while potentially extracting some useful information."
      ],
      "metadata": {
        "id": "h6plphBEpno1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "char_cabin = titanic_train[\"Cabin\"].astype(str) # Convert data to str\n",
        "\n",
        "new_Cabin = np.array([cabin[0] for cabin in char_cabin]) # Take first letter\n",
        "\n",
        "\n",
        "new_Cabin = pd.Categorical(new_Cabin)\n",
        "\n",
        "new_Cabin.describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:51:03.402662Z",
          "iopub.execute_input": "2024-08-22T08:51:03.403262Z",
          "iopub.status.idle": "2024-08-22T08:51:03.424109Z",
          "shell.execute_reply.started": "2024-08-22T08:51:03.402950Z",
          "shell.execute_reply": "2024-08-22T08:51:03.423310Z"
        },
        "trusted": true,
        "id": "cSh8Ealipno1",
        "outputId": "4b9e83ef-cef3-463a-e893-432d0784f5ad"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 39,
          "output_type": "execute_result",
          "data": {
            "text/plain": "            counts     freqs\ncategories                  \nA               15  0.016835\nB               47  0.052750\nC               59  0.066218\nD               33  0.037037\nE               32  0.035915\nF               13  0.014590\nG                4  0.004489\nT                1  0.001122\nn              687  0.771044",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>counts</th>\n      <th>freqs</th>\n    </tr>\n    <tr>\n      <th>categories</th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>A</th>\n      <td>15</td>\n      <td>0.016835</td>\n    </tr>\n    <tr>\n      <th>B</th>\n      <td>47</td>\n      <td>0.052750</td>\n    </tr>\n    <tr>\n      <th>C</th>\n      <td>59</td>\n      <td>0.066218</td>\n    </tr>\n    <tr>\n      <th>D</th>\n      <td>33</td>\n      <td>0.037037</td>\n    </tr>\n    <tr>\n      <th>E</th>\n      <td>32</td>\n      <td>0.035915</td>\n    </tr>\n    <tr>\n      <th>F</th>\n      <td>13</td>\n      <td>0.014590</td>\n    </tr>\n    <tr>\n      <th>G</th>\n      <td>4</td>\n      <td>0.004489</td>\n    </tr>\n    <tr>\n      <th>T</th>\n      <td>1</td>\n      <td>0.001122</td>\n    </tr>\n    <tr>\n      <th>n</th>\n      <td>687</td>\n      <td>0.771044</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The output of describe() shows we succeeded in condensing Cabin into a handful of broader categories, but we also discovered something interesting: 688 of the records have Cabin are \"n\" which is shortened from \"nan\". In other words, more than 2/3 of the passengers do not have a cabin listed at all. Discovering and deciding how to handle these sorts of peculiarities is an important part working with data and there often isn't a single correct answer.\n",
        "\n",
        "Since there are so many missing values, the Cabin variable might be devoid of useful information for prediction. On the other hand, a missing cabin variable could be an indication that a passenger died: after all, how would we know what cabin a passenger stayed in if they weren't around to tell the tale?\n",
        "\n",
        "Let's keep the new cabin variable:"
      ],
      "metadata": {
        "id": "Ds37Bt8qpno1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Cabin\"] = new_Cabin"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:52:07.878158Z",
          "iopub.execute_input": "2024-08-22T08:52:07.878486Z",
          "iopub.status.idle": "2024-08-22T08:52:07.883796Z",
          "shell.execute_reply.started": "2024-08-22T08:52:07.878440Z",
          "shell.execute_reply": "2024-08-22T08:52:07.882402Z"
        },
        "trusted": true,
        "id": "FRO22K0Spno1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is as far as we'll go with transformations right now, but know that the transformations we've covered here are just the tip of the iceberg."
      ],
      "metadata": {
        "id": "yFkkuZp4pno1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Are there NA Values, Outliers or Other Strange Values?"
      ],
      "metadata": {
        "id": "hR12r1Hppno2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data sets are often littered with missing data, extreme data points called outliers and other strange values. Missing values, outliers and strange values can negatively affect statistical tests and models and may even cause certain functions to fail.\n",
        "\n",
        "In Python, you can detect missing values with the pd.isnull() function:"
      ],
      "metadata": {
        "id": "tJ5XeHaFpno2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dummy_vector = pd.Series([1,None,3,None,7,8])\n",
        "\n",
        "dummy_vector.isnull()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:09:17.729094Z",
          "iopub.execute_input": "2024-08-22T10:09:17.729429Z",
          "iopub.status.idle": "2024-08-22T10:09:17.738166Z",
          "shell.execute_reply.started": "2024-08-22T10:09:17.729381Z",
          "shell.execute_reply": "2024-08-22T10:09:17.737292Z"
        },
        "trusted": true,
        "id": "GcfnGU-bpno2",
        "outputId": "96aca5b6-5246-44b6-fa8e-df4822417772"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 62,
          "output_type": "execute_result",
          "data": {
            "text/plain": "0    False\n1     True\n2    False\n3     True\n4    False\n5    False\ndtype: bool"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Detecting missing values is the easy part: it is far more difficult to decide how to handle them. In cases where you have a lot of data and only a few missing values, it might make sense to simply delete records with missing values present. On the other hand, if you have more than a handful of missing values, removing records with missing values could cause you to get rid of a lot of data. Missing values in categorical data are not particularly troubling because you can simply treat NA as an additional category. Missing values in numeric variables are more troublesome, since you can't just treat a missing value as number. As it happens, the Titanic dataset has some NA's in the Age variable:"
      ],
      "metadata": {
        "id": "ch8HY8Pgpno2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Age\"].describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:54:39.679574Z",
          "iopub.execute_input": "2024-08-22T08:54:39.679900Z",
          "iopub.status.idle": "2024-08-22T08:54:39.690280Z",
          "shell.execute_reply.started": "2024-08-22T08:54:39.679836Z",
          "shell.execute_reply": "2024-08-22T08:54:39.689266Z"
        },
        "trusted": true,
        "id": "djQ-6Roopno2",
        "outputId": "557253cb-4041-469b-8976-66f6ae47cfef"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 44,
          "output_type": "execute_result",
          "data": {
            "text/plain": "count    714.000000\nmean      29.699118\nstd       14.526497\nmin        0.420000\n25%       20.125000\n50%       28.000000\n75%       38.000000\nmax       80.000000\nName: Age, dtype: float64"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Notice the count of age(712) is less than the total row count of hte data set(889). This indicates missing data. We can get the row indexes of the missing values with np.where():"
      ],
      "metadata": {
        "id": "uJbI4aiapno3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "missing = np.where(titanic_train[\"Age\"].isnull() == True)\n",
        "missing"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:56:32.063026Z",
          "iopub.execute_input": "2024-08-22T08:56:32.063362Z",
          "iopub.status.idle": "2024-08-22T08:56:32.071882Z",
          "shell.execute_reply.started": "2024-08-22T08:56:32.063312Z",
          "shell.execute_reply": "2024-08-22T08:56:32.070920Z"
        },
        "trusted": true,
        "id": "UbSCRYGZpno3",
        "outputId": "541dbe77-4bae-4e72-bca9-988bde933e29"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 47,
          "output_type": "execute_result",
          "data": {
            "text/plain": "(array([  5,  17,  19,  26,  28,  29,  31,  32,  36,  42,  45,  46,  47,\n         48,  55,  64,  65,  76,  77,  82,  87,  95, 101, 107, 109, 121,\n        126, 128, 140, 154, 158, 159, 166, 168, 176, 180, 181, 185, 186,\n        196, 198, 201, 214, 223, 229, 235, 240, 241, 250, 256, 260, 264,\n        270, 274, 277, 284, 295, 298, 300, 301, 303, 304, 306, 324, 330,\n        334, 335, 347, 351, 354, 358, 359, 364, 367, 368, 375, 384, 388,\n        409, 410, 411, 413, 415, 420, 425, 428, 431, 444, 451, 454, 457,\n        459, 464, 466, 468, 470, 475, 481, 485, 490, 495, 497, 502, 507,\n        511, 517, 522, 524, 527, 531, 533, 538, 547, 552, 557, 560, 563,\n        564, 568, 573, 578, 584, 589, 593, 596, 598, 601, 602, 611, 612,\n        613, 629, 633, 639, 643, 648, 650, 653, 656, 667, 669, 674, 680,\n        692, 697, 709, 711, 718, 727, 732, 738, 739, 740, 760, 766, 768,\n        773, 776, 778, 783, 790, 792, 793, 815, 825, 826, 828, 832, 837,\n        839, 846, 849, 859, 863, 868, 878, 888]),)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(missing[0])"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T08:57:00.459829Z",
          "iopub.execute_input": "2024-08-22T08:57:00.460191Z",
          "iopub.status.idle": "2024-08-22T08:57:00.465605Z",
          "shell.execute_reply.started": "2024-08-22T08:57:00.460130Z",
          "shell.execute_reply": "2024-08-22T08:57:00.464666Z"
        },
        "trusted": true,
        "id": "99oG-D3epno3",
        "outputId": "99ac48b4-5ed9-45af-9ba9-a0fa92e0a8fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 48,
          "output_type": "execute_result",
          "data": {
            "text/plain": "177"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "With 177 missing values it's probably not a good idea to throw all those records away. Here are a few ways we could deal with them:\n",
        "\n",
        "1. Replace the null values with 0s\n",
        "2. Replace the null values with some central value like the mean or median\n",
        "3. Impute some other value\n",
        "4. Split the data set into two parts: one set with where records have an Age value and another set where age is null.\n",
        "\n",
        "Setting missing values in numeric data to zero makes sense in some cases, but it doesn't make any sense here because a person's age can't be zero. Setting all ages to some central number like the median is a simple fix but there's no telling whether such a central number is a reasonable estimate of age without looking at the distribution of ages. For all we know each age is equally common. We can quickly get a sense of the distribution of ages by creating a histogram of the age variable with df.hist():"
      ],
      "metadata": {
        "id": "BjzhSbN5pno3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.hist(column='Age',    # Column to plot\n",
        "                   figsize=(9,6),   # Plot size\n",
        "                   bins=20)         # Number of histogram bins"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:11:20.058431Z",
          "iopub.execute_input": "2024-08-22T10:11:20.058770Z",
          "iopub.status.idle": "2024-08-22T10:11:20.469733Z",
          "shell.execute_reply.started": "2024-08-22T10:11:20.058714Z",
          "shell.execute_reply": "2024-08-22T10:11:20.468734Z"
        },
        "trusted": true,
        "id": "y4vMlT2Xpno3",
        "outputId": "fd63d424-ee1a-41cc-8027-b97e2fae0ffa"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 65,
          "output_type": "execute_result",
          "data": {
            "text/plain": "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x79aba425b630>]],\n      dtype=object)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 648x432 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAF1CAYAAAA+4Dr7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAF2VJREFUeJzt3W2sZVd5H/D/U08IxkNt3nohtsM4AoEQkxAYESIImsFJZDDCSKUulCZD5MhfSEISR80kqopSFdVIJSlVo0gWb25FGBwDxcUkwXKYpPlgFw/QjsGhOGRscP1CFNtkAEEmffrh7oHLYDLje86Zde85v590dffeZ5911jN3n6P/rLXP3tXdAQA40/7R6A4AAKtJCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIYQQYG6q6lBVPVhV3z+6L8DWJ4QAc1FVu5L8RJJO8qqhnQG2BSEEmJefTXJLkvck2X9iY1U9qar+e1V9pao+UVX/rqr+fMPjz66qm6rqb6rqc1V1+ZnvOjDCjtEdAJbGzyb57SS3Jrmlqta6+/4kv5vkq0memmRXkj9OcleSVNU5SW5K8m+SvDzJ7iQ3VdXt3f3ZM14BcEYZCQFmVlUvSfL0JNd19+Ekf5nkX1TVWUn+aZI3d/fXpmBx7YanvjLJ0e5+d3cf7+5PJflAkn92hksABhBCgHnYn+Rj3f3X0/rvT9uekvUR1y9u2Hfj8tOT/FhVPXTiJ8nrsz5qAiw50zHATKrq7CSXJzmrqu6bNn9/kvOSrCU5nuSCJP9neuzCDU//YpI/7e6fOkPdBbaQ6u7RfQC2sap6XdbP+3hekm9ueOi6JJ/IegD5+yQ/n+QHk3wsyd3d/ZKqenyS25P86yQHp+c9L8mx7r7jzFQAjGI6BpjV/iTv7u67u/u+Ez9J/nPWp1Z+Icm5Se5L8l+TvC/JN5Kku/82yU8neW2S/zvt89asj6QAS85ICHBGVdVbkzy1u/efcmdgqRkJARZqug7ID9e6Fya5IsmHRvcLGM+JqcCiPT7rUzA/kOT+JG9L8uGhPQK2BNMxAMAQpmMAgCGEEABgiC1xTsiTn/zk3rVr19za++pXv5pzzjlnbu1tVatQ5yrUmKxGnWpcHqtQpxpnc/jw4b/u7qecar8tEUJ27dqV2267bW7tHTp0KHv37p1be1vVKtS5CjUmq1GnGpfHKtSpxtlU1V2ns5/pGABgCCEEABhCCAEAhjhlCKmqd1XVA1V1+4ZtT6yqm6rq89PvJ0zbq6r+U1XdWVX/u6qev8jOAwDb1+mMhLwnySUnbTuQ5ObufmaSm6f1JHl5kmdOP1cm+b35dBMAWDanDCHd/WdJ/uakzZcluXZavjbJqzds/y+97pYk51XV0+bVWQBgeWz2nJC17r53Wr4vydq0fH6SL27Y70vTNgCA73Ba946pql1JPtLdz53WH+ru8zY8/mB3P6GqPpLk6u7+82n7zUl+vbu/6yIgVXVl1qdssra29oKDBw/OoZx1x44dy86dO+fW3la1CnWuQo3JatSpxuWxCnWqcTb79u073N17TrXfZi9Wdn9VPa27752mWx6Ytt+T5MIN+10wbfsu3X1NkmuSZM+ePT3PC6aswkVmktWocxVqTFajTjUuj1WoU41nxmanY25Isn9a3p9v35b7hiQ/O31L5kVJHt4wbQMA8C2nHAmpqvcl2ZvkyVX1pSRvTnJ1kuuq6ookdyW5fNr9o0lekeTOJF9L8nML6DMAsAROGUK6+3Xf46GLH2HfTvLGWTsFACw/V0wFAIbYEnfRhVPZdeDGhbV99OpLF9Y2AN+bkRAAYAghBAAYQggBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIXaM7gCMtuvAjQtr++jVly6sbYDtzkgIADCEEAIADCGEAABDOCcEFuh0zze5avfxvOFRnpvifBNguzMSAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwxEwhpKp+pao+U1W3V9X7quqxVXVRVd1aVXdW1fur6jHz6iwAsDw2HUKq6vwkv5RkT3c/N8lZSV6b5K1Jfqe7n5HkwSRXzKOjAMBymXU6ZkeSs6tqR5LHJbk3ycuSXD89fm2SV8/4GgDAEqru3vyTq96U5C1Jvp7kY0nelOSWaRQkVXVhkj+cRkpOfu6VSa5MkrW1tRccPHhw0/042bFjx7Jz5865tbdVrUKdJ2o8cs/Do7uyUGtnJ/d//dE9Z/f55y6mMwuySsfrsluFOtU4m3379h3u7j2n2m/TN7CrqickuSzJRUkeSvIHSS453ed39zVJrkmSPXv29N69ezfble9y6NChzLO9rWoV6jxR46O9udt2c9Xu43nbkUf3djz6+r2L6cyCrNLxuuxWoU41nhmzTMf8ZJK/6u4vd/ffJflgkhcnOW+ankmSC5LcM2MfAYAltOmRkCR3J3lRVT0u69MxFye5LcnHk7wmycEk+5N8eNZOsj2c7m3rH43N3OIegO1h0yMh3X1r1k9A/WSSI1Nb1yT59SS/WlV3JnlSknfOoZ8AwJKZZSQk3f3mJG8+afMXkrxwlnYBgOXniqkAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQ+wY3QFgc3YduHEh7R69+tKFtAtwMiMhAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQM4WQqjqvqq6vqr+oqjuq6ser6olVdVNVfX76/YR5dRYAWB6zjoS8Pckfdfezk/xIkjuSHEhyc3c/M8nN0zoAwHfYdAipqnOTvDTJO5Oku7/Z3Q8luSzJtdNu1yZ59aydBACWzywjIRcl+XKSd1fVp6rqHVV1TpK17r532ue+JGuzdhIAWD7V3Zt7YtWeJLckeXF331pVb0/ylSS/2N3nbdjvwe7+rvNCqurKJFcmydra2gsOHjy4qX48kmPHjmXnzp1za2+r2mp1Hrnn4bm3uXZ2cv/X597slrOV6tx9/rkLaXerHa+LsAo1JqtRpxpns2/fvsPdvedU+80SQp6a5Jbu3jWt/0TWz/94RpK93X1vVT0tyaHuftY/1NaePXv6tttu21Q/HsmhQ4eyd+/eubW3VW21OncduHHubV61+3jedmTH3NvdarZSnUevvnQh7W6143URVqHGZDXqVONsquq0Qsimp2O6+74kX6yqEwHj4iSfTXJDkv3Ttv1JPrzZ1wAAltes//X6xSTvrarHJPlCkp/LerC5rqquSHJXkstnfA0AYAnNFEK6+9NJHmm45eJZ2gUAlp8rpgIAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEDtGdwDYWnYduHEh7V61+3j2LqRlYLsyEgIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADDEzCGkqs6qqk9V1Uem9Yuq6taqurOq3l9Vj5m9mwDAspnHSMibktyxYf2tSX6nu5+R5MEkV8zhNQCAJTNTCKmqC5JcmuQd03oleVmS66ddrk3y6lleAwBYTtXdm39y1fVJ/n2Sxyf5tSRvSHLLNAqSqrowyR9293Mf4blXJrkySdbW1l5w8ODBTffjZMeOHcvOnTuTJEfueXhu7Z5s9/nnLqzt07Gxzq1gEf/Wa2cn93997s1uOatQ59rZyT954tj3zKJttffkoqxCnWqczb59+w53955T7bdjsy9QVa9M8kB3H66qvY/2+d19TZJrkmTPnj29d++jbuJ7OnToUE6094YDN86t3ZMdff3ehbV9OjbWuRUs4t/6qt3H87Yjmz5Mt41VqPOq3cdz+RY6Xhdhq70nF2UV6lTjmTHLp96Lk7yqql6R5LFJ/nGStyc5r6p2dPfxJBckuWf2bgIAy2bT54R092909wXdvSvJa5P8SXe/PsnHk7xm2m1/kg/P3EsAYOks4johv57kV6vqziRPSvLOBbwGALDNzWUSursPJTk0LX8hyQvn0S4AsLxcMRUAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIZY7nuHA1vKrgM3ju7Cphy9+tLRXYClZCQEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCDewWzHb9QZiACwfIyEAwBBCCAAwhBACAAwhhAAAQwghAMAQvh0DcAqn+62yq3Yfzxse5TfQjl596Wa6BEvBSAgAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQQggAMIQQAgAMIYQAAEMIIQDAEEIIADCEEAIADCGEAABDCCEAwBBCCAAwhBACAAwhhAAAQwghAMAQmw4hVXVhVX28qj5bVZ+pqjdN259YVTdV1een30+YX3cBgGUxy0jI8SRXdfdzkrwoyRur6jlJDiS5ubufmeTmaR0A4DtsOoR0973d/clp+W+T3JHk/CSXJbl22u3aJK+etZMAwPKp7p69kapdSf4syXOT3N3d503bK8mDJ9ZPes6VSa5MkrW1tRccPHhw5n6ccOzYsezcuTNJcuSeh+fW7sl2n3/uwto+HRvrPF2L/PdYhLWzk/u/ProXi7cKdarxkY3+HNmMzXz2bDdqnM2+ffsOd/eeU+03cwipqp1J/jTJW7r7g1X10MbQUVUPdvc/eF7Inj17+rbbbpupHxsdOnQoe/fuTZLsOnDj3No92dGrL11Y26djY52na5H/Hotw1e7jeduRHaO7sXCrUKcaH9noz5HN2Mxnz3ajxtlU1WmFkJm+HVNV35fkA0ne290fnDbfX1VPmx5/WpIHZnkNAGA5zfLtmEryziR3dPdvb3johiT7p+X9ST68+e4BAMtqlrHRFyf5mSRHqurT07bfTHJ1kuuq6ookdyW5fLYuAgDLaNMhpLv/PEl9j4cv3my7AMBqcMVUAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhljuGzkAbHGLup/TdrwnDavHSAgAMIQQAgAMYTpmizqdIdqrdh/PGxY0lAsAi2YkBAAYQggBAIYwHTODRZ3VDgCrwEgIADCEEAIADCGEAABDOCcEYAkt8py191xyzsLaZrUYCQEAhhBCAIAhhBAAYAghBAAYQggBAIbw7RgAHpUj9zy8sJtnHr360oW0y9ZkJAQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhhBCAIAhhBAAYAghBAAYQggBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhtgxugMAcCbsOnDjae971e7jecOj2P/o1Zdupksrz0gIADCEEAIADGE6BoAt49FMmbD9GQkBAIYQQgCAIYQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhiISGkqi6pqs9V1Z1VdWARrwEAbG9zv2JqVZ2V5HeT/FSSLyX5RFXd0N2fnfdrAcBWsB2v9PqeS84Z3YWFjIS8MMmd3f2F7v5mkoNJLlvA6wAA29giQsj5Sb64Yf1L0zYAgG+p7p5vg1WvSXJJd//8tP4zSX6su3/hpP2uTHLltPqsJJ+bYzeenOSv59jeVrUKda5Cjclq1KnG5bEKdapxNk/v7qecaqdF3EX3niQXbli/YNr2Hbr7miTXLOD1U1W3dfeeRbS9laxCnatQY7IadapxeaxCnWo8MxYxHfOJJM+sqouq6jFJXpvkhgW8DgCwjc19JKS7j1fVLyT54yRnJXlXd39m3q8DAGxvi5iOSXd/NMlHF9H2aVrINM8WtAp1rkKNyWrUqcblsQp1qvEMmPuJqQAAp8Nl2wGAIZYuhCzrJeOr6l1V9UBV3b5h2xOr6qaq+vz0+wkj+zirqrqwqj5eVZ+tqs9U1Zum7UtTZ1U9tqr+Z1X9r6nG35q2X1RVt07H7funk7q3tao6q6o+VVUfmdaXscajVXWkqj5dVbdN25bmeE2Sqjqvqq6vqr+oqjuq6seXsMZnTX/DEz9fqapfXsI6f2X63Lm9qt43fR4NfV8uVQjZcMn4lyd5TpLXVdVzxvZqbt6T5JKTth1IcnN3PzPJzdP6dnY8yVXd/ZwkL0ryxunvt0x1fiPJy7r7R5I8L8klVfWiJG9N8jvd/YwkDya5YmAf5+VNSe7YsL6MNSbJvu5+3oavOi7T8Zokb0/yR9397CQ/kvW/6VLV2N2fm/6Gz0vygiRfS/KhLFGdVXV+kl9Ksqe7n5v1L468NoPfl0sVQrLEl4zv7j9L8jcnbb4sybXT8rVJXn1GOzVn3X1vd39yWv7brH/YnZ8lqrPXHZtWv2/66SQvS3L9tH1b15gkVXVBkkuTvGNaryxZjf+ApTleq+rcJC9N8s4k6e5vdvdDWaIaH8HFSf6yu+/K8tW5I8nZVbUjyeOS3JvB78tlCyGrdsn4te6+d1q+L8nayM7MU1XtSvKjSW7NktU5TVN8OskDSW5K8pdJHuru49Muy3Dc/sck/yrJ/5vWn5TlqzFZD5Afq6rD01Wgk+U6Xi9K8uUk756m1t5RVedkuWo82WuTvG9aXpo6u/ueJP8hyd1ZDx8PJzmcwe/LZQshK6vXv+a0FF91qqqdST6Q5Je7+ysbH1uGOrv776dh3wuyPnr37MFdmquqemWSB7r78Oi+nAEv6e7nZ30K+I1V9dKNDy7B8bojyfOT/F53/2iSr+akKYklqPFbpvMhXpXkD05+bLvXOZ3PclnWg+UPJDkn3z3Ff8YtWwg5rUvGL5H7q+ppSTL9fmBwf2ZWVd+X9QDy3u7+4LR56epMkmlY++NJfjzJedMQabL9j9sXJ3lVVR3N+pToy7J+XsEy1ZjkW/+7THc/kPVzCF6Y5Tpev5TkS91967R+fdZDyTLVuNHLk3yyu++f1pepzp9M8lfd/eXu/rskH8z6e3Xo+3LZQsiqXTL+hiT7p+X9ST48sC8zm84beGeSO7r7tzc8tDR1VtVTquq8afnsJD+V9XNfPp7kNdNu27rG7v6N7r6gu3dl/T34J939+ixRjUlSVedU1eNPLCf56SS3Z4mO1+6+L8kXq+pZ06aLk3w2S1TjSV6Xb0/FJMtV591JXlRVj5s+a0/8LYe+L5fuYmVV9Yqsz0efuGT8WwZ3aS6q6n1J9mb9rof3J3lzkv+W5LokP5jkriSXd/fJJ69uG1X1kiT/I8mRfPtcgt/M+nkhS1FnVf1w1k/+Oivr/wm4rrv/bVX9UNZHDZ6Y5FNJ/mV3f2NcT+ejqvYm+bXufuWy1TjV86FpdUeS3+/ut1TVk7Ikx2uSVNXzsn6C8WOSfCHJz2U6drMkNSbfCpJ3J/mh7n542rZsf8vfSvLPs/5NxE8l+fmsnwMy7H25dCEEANgelm06BgDYJoQQAGAIIQQAGEIIAQCGEEIAgCGEEABgCCEEABhCCAEAhvj/N26DHinhEdEAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "From the histogram, we see that ages between 20 and 30 are the most common, so filling in missing values with a central number like the mean or median wouldn't be entirely unreasonable. Let's fill in the missing values with the median value of 28:"
      ],
      "metadata": {
        "id": "wqJD_m63pno3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "new_age_var = np.where(titanic_train[\"Age\"].isnull(), # Logical check\n",
        "                       28,                       # Value if check is true\n",
        "                       titanic_train[\"Age\"])     # Value if check is false\n",
        "\n",
        "titanic_train[\"Age\"] = new_age_var\n",
        "\n",
        "titanic_train[\"Age\"].describe()"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:13:12.568786Z",
          "iopub.execute_input": "2024-08-22T10:13:12.569141Z",
          "iopub.status.idle": "2024-08-22T10:13:12.582139Z",
          "shell.execute_reply.started": "2024-08-22T10:13:12.569091Z",
          "shell.execute_reply": "2024-08-22T10:13:12.581308Z"
        },
        "trusted": true,
        "id": "5sHijDU5pno4",
        "outputId": "67d40eca-a24b-4f9a-b6b6-f1d82b30ba0f"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 66,
          "output_type": "execute_result",
          "data": {
            "text/plain": "count    891.000000\nmean      29.361582\nstd       13.019697\nmin        0.420000\n25%       22.000000\n50%       28.000000\n75%       35.000000\nmax       80.000000\nName: Age, dtype: float64"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "missing = np.where(titanic_train[\"Age\"].isnull() == True)\n",
        "missing"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:13:55.662665Z",
          "iopub.execute_input": "2024-08-22T10:13:55.663233Z",
          "iopub.status.idle": "2024-08-22T10:13:55.669649Z",
          "shell.execute_reply.started": "2024-08-22T10:13:55.662964Z",
          "shell.execute_reply": "2024-08-22T10:13:55.668893Z"
        },
        "trusted": true,
        "id": "u181WMzMpno4",
        "outputId": "19a9a63a-769b-479c-dd97-efc98cd06817"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 67,
          "output_type": "execute_result",
          "data": {
            "text/plain": "(array([], dtype=int64),)"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since we just added a bunch of 28s to age, let's look at the histogram again for a sanity check. The bar representing 28 to be much taller this time."
      ],
      "metadata": {
        "id": "z3IW9gMvpno4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train.hist(column='Age',    # Column to plot\n",
        "                   figsize=(9,6),   # Plot size\n",
        "                   bins=20)         # Number of histogram bins"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:14:24.468833Z",
          "iopub.execute_input": "2024-08-22T10:14:24.469183Z",
          "iopub.status.idle": "2024-08-22T10:14:24.869132Z",
          "shell.execute_reply.started": "2024-08-22T10:14:24.469122Z",
          "shell.execute_reply": "2024-08-22T10:14:24.868182Z"
        },
        "trusted": true,
        "id": "RzJv7FZYpno4",
        "outputId": "be030378-8f27-4763-f469-78d14cb17063"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 68,
          "output_type": "execute_result",
          "data": {
            "text/plain": "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x79aba405a4a8>]],\n      dtype=object)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 648x432 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAF1CAYAAADV14H4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAF9dJREFUeJzt3X+w5WddH/D3pwlSzDIJGHqNm+jiNIWJRCPsAI7UuQsVw48hOLVpUoqJYtc/QottOnV1OgVrMxNmBKuDZSYaIFZkTflRUoJKmrJS/gDJAjUhKSXABtjmh5YQWOhAN3z6x/0uXMLK3uw9J8+957xeM2fu9/uc5zzn+cw55877Pt/v99zq7gAAjPA3Rk8AAFhegggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIADNTVQeq6v6qevTouQDbgyACzERV7Uryd5N0khcNnQywbQgiwKz8bJIPJHlTksuONVbV91TVf6mqL1bVh6rq31XV+9fd/+SquqmqPl9VH6+qix/5qQOjnDp6AsDC+Nkkr03ywSQfqKqV7r43ye8k+XKS702yK8mfJrkrSarqtCQ3Jfk3SZ6X5PwkN1XVbd19+yNeAfCIsyICbFpVPSvJDyS5vrsPJvlkkn9UVack+ftJXtndX5nCxXXrHvrCJIe6+43dfbS7P5LkbUn+wSNcAjCIIALMwmVJ3tPdfzXt/+HU9oSsrbx+dl3f9ds/kOQZVfWFY7ckL8na6gmwBByaATalqh6T5OIkp1TVPVPzo5OckWQlydEkZyf5X9N956x7+GeT/Fl3/+QjNF1gi6nuHj0HYBurqkuzdh7IBUm+tu6u65N8KGsh5MEkv5Dk+5O8J8lnuvtZVfXYJLcl+ddJ9k+PuyDJke6+45GpABjJoRlgsy5L8sbu/kx333PsluR1WTvM8vIkpye5J8l/TPKWJF9Nku7+UpLnJrkkyf+e+rw6aysqwBKwIgI8oqrq1Um+t7svO2FnYOFZEQHmavqekB+uNU9P8rIk7xg9L2BrcLIqMG+PzdrhmO9Lcm+S1yR559AZAVuGQzMAwDAOzQAAwwgiAMAwW+IckTPPPLN37do10zG//OUv57TTTpvpmFuNGhfHMtSpxsWxDHUuQ43J/Oo8ePDgX3X3EzbSd0sEkV27duWWW26Z6ZgHDhzI6urqTMfcatS4OJahTjUujmWocxlqTOZXZ1XdtdG+Ds0AAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMNsif++CyPt2nfj3MY+dPUL5jY2wCKwIgIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMOcMIhU1TlV9d6qur2qPlZVr5jaX1VVh6vqo9Pt+ese8ytVdWdVfbyqfmqeBQAA29dG/tfM0SRXdveHq+qxSQ5W1U3Tfb/Z3b+xvnNVnZfkkiQ/lOT7kvzXqvo73f3gLCcOAGx/J1wR6e67u/vD0/aXktyRZOd3eMhFSfZ391e7+9NJ7kzy9FlMFgBYLNXdG+9ctSvJ+5I8Jcm/SHJ5ki8muSVrqyb3V9Xrknygu/9gesy1Sf64u9/6kLH2JtmbJCsrK0/bv3//Zmv5FkeOHMmOHTtmOuZWo8bZuPXwA3Mb+/ydp2+on9dyMSxDjcly1LkMNSbzq3PPnj0Hu3v3Rvpu5NBMkqSqdiR5W5Jf6u4vVtXrk/x6kp5+vibJz290vO6+Jsk1SbJ79+5eXV3d6EM35MCBA5n1mFuNGmfj8n03zm3sQy9Z3VA/r+ViWIYak+WocxlqTLZGnRu6aqaqHpW1EPLm7n57knT3vd39YHd/Pcnv5puHXw4nOWfdw8+e2gAAvsVGrpqpJNcmuaO7X7uu/ax13X46yW3T9g1JLqmqR1fVE5Ocm+TPZzdlAGBRbOTQzI8neWmSW6vqo1Pbrya5tKouyNqhmUNJfjFJuvtjVXV9ktuzdsXNFa6YAQCO54RBpLvfn6SOc9e7v8Njrkpy1SbmBQAsAd+sCgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMOcMIhU1TlV9d6qur2qPlZVr5jaH19VN1XVJ6afj5vaq6p+u6rurKq/qKqnzrsIAGB72siKyNEkV3b3eUmemeSKqjovyb4kN3f3uUlunvaT5HlJzp1ue5O8fuazBgAWwgmDSHff3d0fnra/lOSOJDuTXJTkuqnbdUlePG1flOT3e80HkpxRVWfNfOYAwLb3sM4RqapdSX40yQeTrHT33dNd9yRZmbZ3Jvnsuod9bmoDAPgW1d0b61i1I8mfJbmqu99eVV/o7jPW3X9/dz+uqt6V5Orufv/UfnOSX+7uWx4y3t6sHbrJysrK0/bv3z+biiZHjhzJjh07ZjrmVqPG2bj18ANzG/v8nadvqJ/XcjEsQ43JctS5DDUm86tzz549B7t790b6nrqRTlX1qCRvS/Lm7n771HxvVZ3V3XdPh17um9oPJzln3cPPntq+RXdfk+SaJNm9e3evrq5uZCobduDAgcx6zK1GjbNx+b4b5zb2oZesbqif13IxLEONyXLUuQw1Jlujzo1cNVNJrk1yR3e/dt1dNyS5bNq+LMk717X/7HT1zDOTPLDuEA4AwDdsZEXkx5O8NMmtVfXRqe1Xk1yd5PqqelmSu5JcPN337iTPT3Jnkq8k+bmZzhgAWBgnDCLTuR7119z9nOP07yRXbHJeAMAS8M2qAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMc8IgUlVvqKr7quq2dW2vqqrDVfXR6fb8dff9SlXdWVUfr6qfmtfEAYDtbyMrIm9KcuFx2n+zuy+Ybu9Okqo6L8klSX5oesx/qKpTZjVZAGCxnDCIdPf7knx+g+NdlGR/d3+1uz+d5M4kT9/E/ACABVbdfeJOVbuSvKu7nzLtvyrJ5Um+mOSWJFd29/1V9bokH+juP5j6XZvkj7v7rccZc2+SvUmysrLytP3798+gnG86cuRIduzYMdMxtxo1zsathx+Y29jn7zx9Q/28lothGWpMlqPOZagxmV+de/bsOdjduzfS99STfI7XJ/n1JD39fE2Sn384A3T3NUmuSZLdu3f36urqSU7l+A4cOJBZj7nVqHE2Lt9349zGPvSS1Q3181ouhmWoMVmOOpehxmRr1HlSV810973d/WB3fz3J7+abh18OJzlnXdezpzYAgG9zUkGkqs5at/vTSY5dUXNDkkuq6tFV9cQk5yb5881NEQBYVCc8NFNVb0mymuTMqvpcklcmWa2qC7J2aOZQkl9Mku7+WFVdn+T2JEeTXNHdD85n6gDAdnfCINLdlx6n+drv0P+qJFdtZlIAwHLwzaoAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwzAmDSFW9oaruq6rb1rU9vqpuqqpPTD8fN7VXVf12Vd1ZVX9RVU+d5+QBgO1tIysib0py4UPa9iW5ubvPTXLztJ8kz0ty7nTbm+T1s5kmALCIThhEuvt9ST7/kOaLklw3bV+X5MXr2n+/13wgyRlVddasJgsALJbq7hN3qtqV5F3d/ZRp/wvdfca0XUnu7+4zqupdSa7u7vdP992c5Je7+5bjjLk3a6smWVlZedr+/ftnU9HkyJEj2bFjx0zH3GrUOBu3Hn5gbmOfv/P0DfXzWi6GZagxWY46l6HGZH517tmz52B3795I31M3+2Td3VV14jTz7Y+7Jsk1SbJ79+5eXV3d7FS+xYEDBzLrMbcaNc7G5ftunNvYh16yuqF+XsvFsAw1JstR5zLUmGyNOk/2qpl7jx1ymX7eN7UfTnLOun5nT20AAN/mZFdEbkhyWZKrp5/vXNf+8qran+QZSR7o7rs3PUuW3q45rloAMM4Jg0hVvSXJapIzq+pzSV6ZtQByfVW9LMldSS6eur87yfOT3JnkK0l+bg5zBgAWxAmDSHdf+tfc9Zzj9O0kV2x2UgDAcvDNqgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMCf877vAydu178YN9bvy/KO5fIN9jzl09QtOZkoAW4oVEQBgGEEEABhGEAEAhhFEAIBhBBEAYBhXzTAzG71C5OE4matJANg+rIgAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDAu34Vtah6XSyf+mR7wyLIiAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwgggAMIwgAgAMI4gAAMMIIgDAMIIIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAwwgiAMAwp27mwVV1KMmXkjyY5Gh3766qxyf5oyS7khxKcnF337+5aQIAi2gWKyJ7uvuC7t497e9LcnN3n5vk5mkfAODbzOPQzEVJrpu2r0vy4jk8BwCwADYbRDrJe6rqYFXtndpWuvvuafueJCubfA4AYEFVd5/8g6t2dvfhqvpbSW5K8k+T3NDdZ6zrc393P+44j92bZG+SrKysPG3//v0nPY/jOXLkSHbs2DHTMbearVbjrYcfmPmYK49J7v2/Mx92y9lKdZ6/8/S5jLvV3q/zsAw1JstR5zLUmMyvzj179hxcd8rGd7SpIPItA1W9KsmRJP8kyWp3311VZyU50N1P+k6P3b17d99yyy0zmccxBw4cyOrq6kzH3Gq2Wo279t048zGvPP9oXnPrps6p3ha2Up2Hrn7BXMbdau/XeViGGpPlqHMZakzmV2dVbTiInPShmao6raoee2w7yXOT3JbkhiSXTd0uS/LOk30OAGCxbeZPsJUk76iqY+P8YXf/SVV9KMn1VfWyJHcluXjz0wQAFtFJB5Hu/lSSHzlO+/9J8pzNTAoAWA5b46A0sGXM41yfZO08mNW5jAxsZ77iHQAYRhABAIYRRACAYQQRAGAYQQQAGGbhr5qZ1xUAyfy+gRIAloUVEQBgGEEEABhGEAEAhhFEAIBhBBEAYJiFv2oG2DrmeRXbPLlCDubHiggAMIwgAgAMI4gAAMMIIgDAME5WXULb9YRBABaPFREAYBgrIgAnsNFVxCvPP5rLH+aKo0uDWXZWRACAYQQRAGAYQQQAGEYQAQCGEUQAgGEEEQBgGEEEABhGEAEAhhFEAIBhBBEAYBhBBAAYRhABAIYRRACAYQQRAGAYQQQAGObU0RMAWGa79t04t7EPXf2CuY0NsyKIbFEb+eV05flHc/kcf4kBwLwJIpswz79kAGAZCCIAC2pefyy96cLT5jIuy8nJqgDAMIIIADCMQzMAPCy3Hn5gbifKu9Jn+VgRAQCGEUQAgGEEEQBgGEEEABhGEAEAhhFEAIBhXL4LwFJ4ON80+3D/l5fLjk+eIALAluF/eC2fuR2aqaoLq+rjVXVnVe2b1/MAANvXXIJIVZ2S5HeSPC/JeUkurarz5vFcAMD2Na8VkacnubO7P9XdX0uyP8lFc3ouAGCbmtc5IjuTfHbd/ueSPGNOzwUAQ23Xc1vedOFpo6eQ6u7ZD1r1M0ku7O5fmPZfmuQZ3f3ydX32Jtk77T4pycdnPI0zk/zVjMfcatS4OJahTjUujmWocxlqTOZX5w909xM20nFeKyKHk5yzbv/sqe0buvuaJNfM6flTVbd09+55jb8VqHFxLEOdalwcy1DnMtSYbI0653WOyIeSnFtVT6yq70pySZIb5vRcAMA2NZcVke4+WlUvT/KnSU5J8obu/tg8ngsA2L7m9oVm3f3uJO+e1/gbMLfDPluIGhfHMtSpxsWxDHUuQ43JFqhzLierAgBshH96BwAMs3BBZFG/Wr6q3lBV91XVbevaHl9VN1XVJ6afjxs5x82qqnOq6r1VdXtVfayqXjG1L0ydVfU3q+rPq+p/TDX+2tT+xKr64PS+/aPpJO9trapOqaqPVNW7pv1FrPFQVd1aVR+tqlumtoV5vyZJVZ1RVW+tqv9ZVXdU1Y8tYI1Pml7DY7cvVtUvLWCd/3z6vXNbVb1l+n00/HO5UEFkwb9a/k1JLnxI274kN3f3uUlunva3s6NJruzu85I8M8kV0+u3SHV+Ncmzu/tHklyQ5MKqemaSVyf5ze7+20nuT/KygXOclVckuWPd/iLWmCR7uvuCdZdALtL7NUl+K8mfdPeTk/xI1l7Thaqxuz8+vYYXJHlakq8keUcWqM6q2pnknyXZ3d1PydqFJJdkC3wuFyqIZIG/Wr6735fk8w9pvijJddP2dUle/IhOasa6++7u/vC0/aWs/cLbmQWqs9ccmXYfNd06ybOTvHVq39Y1JklVnZ3kBUl+b9qvLFiN38HCvF+r6vQkP5Hk2iTp7q919xeyQDUex3OSfLK778ri1XlqksdU1alJvjvJ3dkCn8tFCyLH+2r5nYPm8khY6e67p+17kqyMnMwsVdWuJD+a5INZsDqnQxYfTXJfkpuSfDLJF7r76NRlEd63/z7Jv0ry9Wn/e7J4NSZrIfI9VXVw+rboZLHer09M8pdJ3jgdZvu9qjoti1XjQ12S5C3T9sLU2d2Hk/xGks9kLYA8kORgtsDnctGCyNLqtcufFuISqKrakeRtSX6pu7+4/r5FqLO7H5yWgM/O2irekwdPaaaq6oVJ7uvug6Pn8gh4Vnc/NWuHg6+oqp9Yf+cCvF9PTfLUJK/v7h9N8uU85PDEAtT4DdP5ES9K8p8eet92r3M6v+WirIXL70tyWr79cP8QixZETvjV8gvm3qo6K0mmn/cNns+mVdWjshZC3tzdb5+aF67OJJmWuN+b5MeSnDEtlybb/33740leVFWHsnZ49NlZO89gkWpM8o2/MtPd92XtnIKnZ7Her59L8rnu/uC0/9asBZNFqnG95yX5cHffO+0vUp1/L8mnu/svu/v/JXl71j6rwz+XixZElu2r5W9Ictm0fVmSdw6cy6ZN5xFcm+SO7n7tursWps6qekJVnTFtPybJT2btXJj3JvmZqdu2rrG7f6W7z+7uXVn7DP637n5JFqjGJKmq06rqsce2kzw3yW1ZoPdrd9+T5LNV9aSp6TlJbs8C1fgQl+abh2WSxarzM0meWVXfPf2uPfZaDv9cLtwXmlXV87N2fPrYV8tfNXhKM1FVb0mymrX/lHhvklcm+c9Jrk/y/UnuSnJxdz/0hNZto6qeleS/J7k13zy34Fezdp7IQtRZVT+ctRPCTsnaHwLXd/e/raofzNrqweOTfCTJP+7ur46b6WxU1WqSf9ndL1y0Gqd63jHtnprkD7v7qqr6nizI+zVJquqCrJ10/F1JPpXk5zK9d7MgNSbfCJOfSfKD3f3A1LZor+WvJfmHWbtC8SNJfiFr54QM/VwuXBABALaPRTs0AwBsI4IIADCMIAIADCOIAADDCCIAwDCCCAAwjCACAAwjiAAAw/x/hYMMqxF9XHEAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Some of the ages we assigned are probably way off, but it might be better than throwing entire records away. In practice, imputing the missing data (estimating age based on other variables) might have been a better option, but we'll stick with this for now.\n",
        "\n",
        "Next, let's consider outliers. Outliers are extreme numerical values: values that lie far away from the typical values a variable takes on. Creating plots is one of the quickest ways to detect outliers. For instance, the histogram above shows that 1 or 2 passengers were near age 80. Ages near 80 are uncommon for this data set, but in looking at the general shape of the data seeing one or two 80 year olds doesn't seem particularly surprising.\n",
        "\n",
        "Now let's investigate the \"Fare\" variable. This time we'll use a boxplot, since boxplots are designed to show the spread of the data and help identify outliers:"
      ],
      "metadata": {
        "id": "5e-PCFa1pno5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Fare\"].plot(kind=\"box\",\n",
        "                           figsize=(9,9),\n",
        "                          color = 'Blue')"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:17:08.039916Z",
          "iopub.execute_input": "2024-08-22T10:17:08.040254Z",
          "iopub.status.idle": "2024-08-22T10:17:08.366476Z",
          "shell.execute_reply.started": "2024-08-22T10:17:08.040204Z",
          "shell.execute_reply": "2024-08-22T10:17:08.365463Z"
        },
        "trusted": true,
        "id": "FsXW9Ww5pno5",
        "outputId": "6fdec4f2-69db-433a-b4b3-c14ea90b1d0f"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 70,
          "output_type": "execute_result",
          "data": {
            "text/plain": "<matplotlib.axes._subplots.AxesSubplot at 0x79ab96479748>"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "<Figure size 648x648 with 1 Axes>",
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiEAAAIMCAYAAAAjJXxvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAHBlJREFUeJzt3X+sX3d93/HX+9rpvXWCSNy4QBKIK5W2wYtL4UIhnaauLLstQzZ/tF1R10aV5ejacNXNkWZi9geTFqsFETa8kdtkqZROW3+oW+WrKloc0VQahbq9aWncwCoCSwQJJU79Y/hm9ojz2R/++s6GEDv4Hn/utR8P6av7PZ/vufe+/c/V0+ec7/lWay0AABfbWO8BAIDLkwgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6WN17gCS59tpr2/r163uPAQAsgUcfffS51tq6c+23LCJk/fr1mZ+f7z0GALAEquqp89nP6RgAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECDG5qaipjY2OpqoyNjWVqaqr3SMAyIEKAQU1NTWXfvn2Znp7OkSNHMj09nX379gkRIKt7DwBc2h5++OFs27Ytn/zkJ5Nk8evs7GzPsYBloFprvWfI5ORkm5+f7z0GMICqypEjR/LqV796ce3o0aO5+uqrsxz+/gBLr6oeba1Nnms/p2OAQVVV7rzzzrPW7rzzzlRVp4mA5UKEAIO69dZbc88992T79u05evRotm/fnnvuuSe33npr79GAzpyOAQY3NTWVhx9+OK21VFVuvfXWPPTQQ73HAgZyvqdjXJgKDE5wAC/F6RgAoAsRAgB0cV4RUlVPVtWBqvpcVc2P1tZW1cNV9cXR12tG61VVn6iqJ6rqsap6y5D/AABgZXolR0L+YWvtzWdcaPLBJJ9qrb0xyadG20nyM0neOHrcnuSepRoWALh0XMjpmM1JHhg9fyDJe89Y/612yp8mubqqXncBvwcAuASdb4S0JPuq6tGqun209prW2tdGz/82yWtGz69P8pUzvverozUAgEXn+xbdv99ae7qqvj/Jw1X1P898sbXWquoV3XBkFDO3J8kb3vCGV/KtAMAl4LyOhLTWnh59fTbJHyR5e5Kvnz7NMvr67Gj3p5O8/oxvv2G09q0/897W2mRrbXLdunXf/b8AAFiRzhkhVXVlVb3q9PMk/zjJXyeZS3LbaLfbkuwdPZ9L8sujd8m8I8nRM07bAAAkOb/TMa9J8gejD5taneS/tNb+e1X9eZLfq6otSZ5K8vOj/R9M8u4kTyR5PsmvLPnUAMCKd84Iaa19OcmPvsT63yV510ustyTvX5LpAIBLljumAgBdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQxXlHSFWtqqq/rKo/HG3/QFXtr6onqup3q+p7Ruvjo+0nRq+vH2Z0AGAleyVHQn41yRfO2P71JB9vrf1gksNJtozWtyQ5PFr/+Gg/AICznFeEVNUNSf5Jkv842q4kP5Xk90e7PJDkvaPnm0fbGb3+rtH+AACLzvdIyL9N8i+TvDja/r4kR1prL4y2v5rk+tHz65N8JUlGrx8d7Q8AsOicEVJV70nybGvt0aX8xVV1e1XNV9X8wYMHl/JHAwArwPkcCfmJJJuq6skkv5NTp2H+XZKrq2r1aJ8bkjw9ev50ktcnyej1Vyf5u2/9oa21e1trk621yXXr1l3QPwIAWHnOGSGttTtbaze01tYn+YUkf9Ra+8UkjyT52dFutyXZO3o+N9rO6PU/aq21JZ0aAFjxLuQ+ITuT7KiqJ3Lqmo/7R+v3J/m+0fqOJB+8sBEBgEvR6nPv8v+11v44yR+Pnn85ydtfYp/jSX5uCWYDAC5h7pgKAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6OKcEVJVE1X1Z1X1V1X1eFX969H6D1TV/qp6oqp+t6q+Z7Q+Ptp+YvT6+mH/CQDASnQ+R0JOJPmp1tqPJnlzkp+uqnck+fUkH2+t/WCSw0m2jPbfkuTwaP3jo/0AAM5yzghppxwbbV4xerQkP5Xk90frDyR57+j55tF2Rq+/q6pqySYGAC4J53VNSFWtqqrPJXk2ycNJvpTkSGvthdEuX01y/ej59Um+kiSj148m+b6lHBoAWPnOK0Jaaydba29OckOStyf5kQv9xVV1e1XNV9X8wYMHL/THAQArzCt6d0xr7UiSR5K8M8nVVbV69NINSZ4ePX86yeuTZPT6q5P83Uv8rHtba5Ottcl169Z9l+MDACvV+bw7Zl1VXT16/r1Jbk3yhZyKkZ8d7XZbkr2j53Oj7Yxe/6PWWlvKoQGAlW/1uXfJ65I8UFWrcipafq+19odV9fkkv1NV/ybJXya5f7T//Un+U1U9keRQkl8YYG4AYIU7Z4S01h5L8mMvsf7lnLo+5FvXjyf5uSWZDgC4ZLljKgDQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECDC4jRs3pqoWHxs3buw9ErAMiBBgUBs3bsyBAwcyNnbqz83Y2FgOHDggRAARAgzrwIEDqap89KMfzcLCQj760Y+mqnLgwIHeowGdiRBgcB/+8IezY8eOrFmzJjt27MiHP/zh3iMBy4AIAQa3f//+l90GLk8iBBhUVeXBBx/M5s2b89xzz2Xz5s158MEHU1W9RwM6EyHAoN7//vcnSebm5rJu3brMzc2dtQ5cvlb3HgC4tO3ZsydJct999+XEiRMZHx/P1q1bF9eBy1e11nrPkMnJyTY/P997DABgCVTVo621yXPt53QMANCFCAEAuhAhAEAXIgQA6EKEAABdiBBgcDMzM5mYmEhVZWJiIjMzM71HApYBEQIMamZmJrOzs9m9e3cWFhaye/fuzM7OChHAfUKAYU1MTGT37t3ZsWPH4trdd9+dXbt25fjx4x0nA4ZyvvcJESHAoKoqCwsLWbNmzeLa888/nyuvvDLL4e8PsPTcrAxYFsbHxzM7O3vW2uzsbMbHxztNBCwXPjsGGNTWrVuzc+fOJMn09HRmZ2ezc+fOTE9Pd54M6E2EAIM6/UF1u3btyh133JHx8fFMT0/7ADvANSEAwNJyTQgAsKyJEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBBgcBs3bkxVLT42btzYeyRgGRAhwKA2btyYAwcOZNOmTTl48GA2bdqUAwcOCBFAhADDOh0ge/fuzbXXXpu9e/cuhghweRMhwODuv//+l90GLk8iBBjcli1bXnYbuDyt7j0AcGm7+eabMzc3l1e96lU5duxYrrrqqhw7diw333xz79GAzhwJAQZ155135oorrsixY8eSJMeOHcsVV1yRO++8s/NkQG8iBBjUXXfdlYceeiittcXHQw89lLvuuqv3aEBn1VrrPUMmJyfb/Px87zGAAaxatSrHjx/PFVdcsbj2zW9+MxMTEzl58mTHyYChVNWjrbXJc+3nSAgwqJtuuimf/vSnz1r79Kc/nZtuuqnTRMByIUKAQX3oQx/Kli1b8sgjj+Sb3/xmHnnkkWzZsiUf+tCHeo8GdObdMcCg3ve+9yVJZmZm8oUvfCE33XRT7rrrrsV14PLlmhAAYEm5JgQAWNZECDA4n6ILvBQRAgzKp+gC34kIAQblU3SB70SEAIPzKbrASxEhwOB8ii7wUkQIMKjTn6K7efPmPPfcc9m8eXPm5uZ8ii7gZmXAsB577LFs3Lgxc3NzWbduXZJTYfLYY491ngzoTYQAgxMcwEtxOgYA6OKcEVJVr6+qR6rq81X1eFX96mh9bVU9XFVfHH29ZrReVfWJqnqiqh6rqrcM/Y8AAFae8zkS8kKSO1prb0ryjiTvr6o3Jflgkk+11t6Y5FOj7ST5mSRvHD1uT3LPkk8NAKx454yQ1trXWmt/MXr+jSRfSHJ9ks1JHhjt9kCS946eb07yW+2UP01ydVW9bsknBwBWtFd0TUhVrU/yY0n2J3lNa+1ro5f+NslrRs+vT/KVM77tq6M1AIBF5x0hVXVVkv+a5J+31v73ma+11lqS9kp+cVXdXlXzVTV/8ODBV/KtAMAl4LwipKquyKkA+c+ttf82Wv766dMso6/PjtafTvL6M779htHaWVpr97bWJltrk6fvHQAAXD7O590xleT+JF9ord19xktzSW4bPb8tyd4z1n959C6ZdyQ5esZpGwCAJOd3s7KfSPJLSQ5U1edGa7uS/FqS36uqLUmeSvLzo9ceTPLuJE8keT7JryzpxADAJeGcEdJa+3SS+g4vv+sl9m9J3n+BcwEAlzh3TAUAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQoDBzczMZGJiIlWViYmJzMzM9B4JWAZECDComZmZzM7OZvfu3VlYWMju3bszOzsrRIBUa633DJmcnGzz8/O9xwAGMDExkd27d2fHjh2La3fffXd27dqV48ePd5wMGEpVPdpamzznfiIEGFJVZWFhIWvWrFlce/7553PllVdmOfz9AZbe+UaI0zHAoMbHxzM7O3vW2uzsbMbHxztNBCwXq3sPAFzatm7dmp07dyZJpqenMzs7m507d2Z6errzZEBvIgQY1J49e5Iku3btyh133JHx8fFMT08vrgOXL9eEAABLyjUhAMCyJkIAgC5ECDC4qampjI2NpaoyNjaWqamp3iMBy4AIAQY1NTWVffv2ZXp6OkeOHMn09HT27dsnRAAXpgLDGhsbyzXXXJNDhw4trq1duzaHDx/Oiy++2HEyYCguTAWWhdZaDh06lE2bNuXgwYPZtGlTDh065G6pgAgBhrd+/frs3bs31157bfbu3Zv169f3HglYBkQIMLgnn3wy27dvz9GjR7N9+/Y8+eSTvUcClgHXhACDqqpMTEyc9Ym5p7eXw98fYOm5JgRYFtauXZvjx49nw4YNeeqpp7Jhw4YcP348a9eu7T0a0JnPjgEGtbCwkNe+9rV5/PHHc+ONNyZJXvva1+bw4cOdJwN6EyHAoE6cOJEvfelLWbNmzeLa888/nyuvvLLjVMBy4HQMMKjx8fHMzs6etTY7O5vx8fFOEwHLhSMhwKC2bt2anTt3Jkmmp6czOzubnTt3Znp6uvNkQG8iBBjUnj17kiS7du3KHXfckfHx8UxPTy+uA5cvb9EFAJaUt+gCAMuaCAEAuhAhwOCmpqYyNjaWqsrY2FimpqZ6jwQsAyIEGNTU1FT27duXqkpy6jbu+/btEyKACAGGtW/fviQ5K0LOXAcuXyIEGNzY2Fg+8pGPZGFhIR/5yEcyNuZPDyBCgIvgbW97W3bs2JE1a9Zkx44dedvb3tZ7JGAZECHA4Pbv35/t27fn6NGj2b59e/bv3997JGAZcLMyYFCrV6/OyZMnv2191apVeeGFFzpMBAzNzcqAZWHbtm2pqqxatSrJqfioqmzbtq3zZEBvPjsGGNTpz4i57777cvLkyaxevTpbt2712TGA0zEAwNJyOgYAWNZECADQhQgBALoQIcDgNm7cmKpafGzcuLH3SMAyIEKAQW3cuDEHDhzIpk2bcvDgwWzatCkHDhwQIoAIAYZ1OkD27t2ba6+9Nnv37l0MEeDyJkKAwd1///0vuw1cnkQIMLgtW7a87DZweRIhwKBuvvnmzM3NZfPmzXnuueeyefPmzM3N5eabb+49GtCZO6YCg7vqqquysLCwuH3llVfm2LFjHScChuSOqcCyMDU1lYWFhWzbti1HjhzJtm3bsrCwkKmpqd6jAZ05EgIMamxsLNdcc00OHTq0uLZ27docPnw4L774YsfJgKE4EgIsC621HDp06Kz7hBw6dCjL4T9AQF+rew8AXPquuuqqPPTQQ1m3bl3Gx8dz1VVXuSYEcCQEGN6xY8fy1re+Nc8880ze+ta3ChAgiSMhwEWwatWqfOYzn8l11123uH3y5MnOUwG9ORICDO7kyZPZsGFDnnrqqWzYsEGAAEkcCQEugle96lV5/PHHc+ONNy5uf+Mb3+g8FdCbIyHA4BYWFvKxj33srK8A7hMCDGpiYiKTk5OZn5/PiRMnMj4+vrh9/Pjx3uMBA1iy+4RU1W9W1bNV9ddnrK2tqoer6oujr9eM1quqPlFVT1TVY1X1lgv7ZwAr3datW7N///7s3r07CwsL2b17d/bv35+tW7f2Hg3o7JxHQqrqHyQ5luS3Wmt/b7T2kSSHWmu/VlUfTHJNa21nVb07yUySdyf58ST/rrX24+cawpEQuLTNzMzkvvvuWzwSsnXr1uzZs6f3WMBAzvdIyHmdjqmq9Un+8IwI+ZskP9la+1pVvS7JH7fWfriqfmP0/Le/db+X+/kiBAAuHUPftv01Z4TF3yZ5zej59Um+csZ+Xx2tAQCc5YLfHdNOHUp5xVe3VtXtVTVfVfMHDx680DEAgBXmu42Qr49Ow2T09dnR+tNJXn/GfjeM1r5Na+3e1tpka21y3bp13+UYAMBK9d1GyFyS20bPb0uy94z1Xx69S+YdSY6e63oQ4NI3MzOTiYmJVFUmJiYyMzPTeyRgGTift+j+dpLPJvnhqvpqVW1J8mtJbq2qLyb5R6PtJHkwyZeTPJHkviTbB5kaWDFmZmYyOzt71lt0Z2dnhQjgZmXAsNysDC4/5/vuGJ8dAwzqxIkT+ZM/+ZPvuA1cvnx2DHBRjI2NnfUVwF8D4KJ4z3vek4MHD+Y973lP71GAZcLpGGBw4+PjmZuby+m344+Pj+fEiROdpwJ6cyQEGNyJEydyyy235Jlnnsktt9wiQIAkjoQAF8lnP/vZXHfddamq3qMAy4QjIcBFcfp2AMvhtgDA8iBCAIAuRAhwUWzYsCFPPfVUNmzY0HsUYJlwTQhwUTz++OO58cYbe48BLCOOhAAAXYgQAKALEQIAdCFCAIAuRAhwUXh3DPCtvDsGuCi8Owb4Vo6EAABdiBAAoAsRAgB0IUKAi+L0p+f6FF3gNBECXBTvfOc788wzz+Sd73xn71GAZcK7Y4CL4jOf+Uyuu+663mMAy4gjIcBFccstt+SZZ57JLbfc0nsUYJlwJAS4KBwJAb6VIyHA4FavXv2y28DlSYQAL6vqwh7JeF544deTtMXHqe3xJfjZwEomQoCX1dqFPT7wga1ZvXpnPvaxu5M8n4997O6sXr0zH/jA1gv+2cDK5pgoMKg9e/YkSXbt2pXkjuzaNZ7p6enFdeDyVW0Z/HdicnKyzc/P9x4DGFiVIxhwOaiqR1trk+faz+kYAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADoQoQAAF2IEACgCxECAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdLG69wDAMNauTQ4f7j3Ft6vqPcG3u+aa5NCh3lPA5UeEwCXq8OGktd5TrAzLMYzgcuB0DADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdCFCAIAuRAgA0IUIAQC6ECEAQBciBADowgfYwSXMB7MBy5kIgUuYT9E9P2IN+nA6BgDoQoQAAF2IEACgCxECAHThwlS4hLng8vxcc03vCeDyJELgErUc3xlTtTznAvpwOgYA6GKQCKmqn66qv6mqJ6rqg0P8DgBgZVvyCKmqVUn+Q5KfSfKmJO+rqjct9e8BAFa2IY6EvD3JE621L7fW/m+S30myeYDfAwCsYENEyPVJvnLG9ldHawAAi7q9O6aqbk9ye5K84Q1v6DUGcA5L/Tbfpfx53mkDK9sQR0KeTvL6M7ZvGK2dpbV2b2ttsrU2uW7dugHGAJZCa8v3AaxsQ0TInyd5Y1X9QFV9T5JfSDI3wO8BAFawJT8d01p7oao+kOShJKuS/GZr7fGl/j0AwMo2yDUhrbUHkzw4xM8GAC4N7pgKAHQhQgCALkQIANCFCAEAuhAhAEAXIgQA6EKEAABdiBAAoAsRAgB0IUIAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdFGttd4zpKoOJnmq9xzA4K5N8lzvIYDB3dhaW3eunZZFhACXh6qab61N9p4DWB6cjgEAuhAhAEAXIgS4mO7tPQCwfLgmBADowpEQAKCL1b0HAFa+qjqZ5MAZS+9trT3ZaRxghXA6BrhgVXWstXbVd/F9q1trLwwxE7D8OR0DDKKq1lfV/6iqvxg9bhmt/+RofS7J50dr/6yq/qyqPldVv1FVq7oOD1wUIgRYCt87CojPVdUfjNaeTXJra+0tSf5pkk+csf9bkvxqa+2Hquqm0es/0Vp7c5KTSX7xYg4P9OGaEGAp/J9RQJzpiiT/vqpOh8UPnfHan7XW/tfo+buSvDXJn1dVknxvTgUMcIkTIcBQ/kWSryf50Zw66nr8jNcWznheSR5ord15EWcDlgGnY4ChvDrJ11prLyb5pSTf6TqPTyX52ar6/iSpqrVVdeNFmhHoSIQAQ/lkktuq6q+S/EjOPvqxqLX2+ST/Ksm+qnosycNJXnfRpgS68RZdAKALR0IAgC5ECADQhQgBALoQIQBAFyIEAOhChAAAXYgQAKALEQIAdPH/AHKLhsz/XsvnAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In a boxplot, the central box represents 50% of the data and the central bar represents the median. The dotted lines with bars on the ends are \"whiskers\" which encompass the great majority of the data and points beyond the whiskers indicate uncommon values. In this case, we have some uncommon values that are so far away from the typical value that the box appears squashed in the plot: this is a clear indication of outliers. Indeed, it looks like one passenger paid almost twice as much as any other passenger. Even the passengers that paid between 200 and 300 are far higher than the vast majority of the other passengers.\n",
        "\n",
        "For interest's sake, let's check the name of this high roller:"
      ],
      "metadata": {
        "id": "3Tn82KIJpno5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "index = np.where(titanic_train[\"Fare\"] == max(titanic_train[\"Fare\"]) )\n",
        "print(index)\n",
        "\n",
        "titanic_train.loc[index]"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:23:05.750079Z",
          "iopub.execute_input": "2024-08-22T10:23:05.750385Z",
          "iopub.status.idle": "2024-08-22T10:23:05.783237Z",
          "shell.execute_reply.started": "2024-08-22T10:23:05.750340Z",
          "shell.execute_reply": "2024-08-22T10:23:05.782372Z"
        },
        "trusted": true,
        "id": "GUSasZwSpno5",
        "outputId": "3b824d4e-60fe-4c16-d643-e8fea8941aa4"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "text": "(array([258, 679, 737]),)\n",
          "output_type": "stream"
        },
        {
          "execution_count": 77,
          "output_type": "execute_result",
          "data": {
            "text/plain": "     Survived  Pclass   ...    Cabin Embarked\n258         1  Class1   ...        n        C\n679         1  Class1   ...        B        C\n737         1  Class1   ...        B        C\n\n[3 rows x 10 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Survived</th>\n      <th>Pclass</th>\n      <th>Name</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>258</th>\n      <td>1</td>\n      <td>Class1</td>\n      <td>Ward, Miss. Anna</td>\n      <td>female</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>512.3292</td>\n      <td>n</td>\n      <td>C</td>\n    </tr>\n    <tr>\n      <th>679</th>\n      <td>1</td>\n      <td>Class1</td>\n      <td>Cardeza, Mr. Thomas Drake Martinez</td>\n      <td>male</td>\n      <td>36.0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>512.3292</td>\n      <td>B</td>\n      <td>C</td>\n    </tr>\n    <tr>\n      <th>737</th>\n      <td>1</td>\n      <td>Class1</td>\n      <td>Lesurer, Mr. Gustave J</td>\n      <td>male</td>\n      <td>35.0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>512.3292</td>\n      <td>B</td>\n      <td>C</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In the graph there appears to be one passenger who paid more than all the others, but the output above shows that there were actually three passengers who all paid the same high fare.\n",
        "\n",
        "Similar to NA values, there's no single cure for outliers. You can keep them, delete them or transform them in some way to try to reduce their impact. Even if you decide to keep outliers unchanged it is still worth identifying them since they can have disproportionately large influence on your results. Let's keep the three high rollers unchanged.\n",
        "\n",
        "Data sets can have other strange values beyond missing values and outliers that you may need to address. Sometimes data is mislabeled or simply erroneous; bad data can corrupt any sort of analysis so it is important to address these sorts of issues before doing too much work."
      ],
      "metadata": {
        "id": "_ly363gZpno5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Should I Create New Variables?"
      ],
      "metadata": {
        "id": "LNusywyPpno6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The variables present when you load a data set aren't always the most useful variables for analysis. Creating new variables that are derivations or combinations existing ones is a common step to take before jumping into an analysis or modeling task.\n",
        "\n",
        "For example, imagine you are analyzing web site auctions where one of the data fields is a text description of the item being sold. A raw block of text is difficult to use in any sort of analysis, but you could create new variables from it such as a variable storing the length of the description or variables indicating the presence of certain keywords.\n",
        "\n",
        "Creating a new variable can be as simple as taking one variable and adding, multiplying or dividing by another. Let's create a new variable, Family, that combines SibSp and Parch to indicate the total number of family members (siblings, spouses, parents and children) a passenger has on board:"
      ],
      "metadata": {
        "id": "gPJ7yXPBpno6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "titanic_train[\"Family\"] = titanic_train[\"SibSp\"] + titanic_train[\"Parch\"]"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:26:21.423815Z",
          "iopub.execute_input": "2024-08-22T10:26:21.424152Z",
          "iopub.status.idle": "2024-08-22T10:26:21.429635Z",
          "shell.execute_reply.started": "2024-08-22T10:26:21.424102Z",
          "shell.execute_reply": "2024-08-22T10:26:21.428459Z"
        },
        "trusted": true,
        "id": "zNT3eZL2pno6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "For interest's sake, let's find out who had the most family members on board:"
      ],
      "metadata": {
        "id": "YclJR_LTpno6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "most_family = np.where(titanic_train[\"Family\"] == max(titanic_train[\"Family\"]))\n",
        "\n",
        "titanic_train.loc[most_family]"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-08-22T10:26:38.921544Z",
          "iopub.execute_input": "2024-08-22T10:26:38.921892Z",
          "iopub.status.idle": "2024-08-22T10:26:38.957144Z",
          "shell.execute_reply.started": "2024-08-22T10:26:38.921828Z",
          "shell.execute_reply": "2024-08-22T10:26:38.956155Z"
        },
        "trusted": true,
        "id": "g2nw99B_pno6",
        "outputId": "63ce9f81-36e8-41f6-e56b-ed915812b155"
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 79,
          "output_type": "execute_result",
          "data": {
            "text/plain": "     Survived  Pclass  ...   Embarked Family\n159         0  Class3  ...          S     10\n180         0  Class3  ...          S     10\n201         0  Class3  ...          S     10\n324         0  Class3  ...          S     10\n792         0  Class3  ...          S     10\n846         0  Class3  ...          S     10\n863         0  Class3  ...          S     10\n\n[7 rows x 11 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Survived</th>\n      <th>Pclass</th>\n      <th>Name</th>\n      <th>Sex</th>\n      <th>Age</th>\n      <th>SibSp</th>\n      <th>Parch</th>\n      <th>Fare</th>\n      <th>Cabin</th>\n      <th>Embarked</th>\n      <th>Family</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>159</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Master. Thomas Henry</td>\n      <td>male</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>180</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Miss. Constance Gladys</td>\n      <td>female</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>201</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Mr. Frederick</td>\n      <td>male</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>324</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Mr. George John Jr</td>\n      <td>male</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>792</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Miss. Stella Anna</td>\n      <td>female</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>846</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Mr. Douglas Bullen</td>\n      <td>male</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n    <tr>\n      <th>863</th>\n      <td>0</td>\n      <td>Class3</td>\n      <td>Sage, Miss. Dorothy Edith \"Dolly\"</td>\n      <td>female</td>\n      <td>28.0</td>\n      <td>8</td>\n      <td>2</td>\n      <td>69.55</td>\n      <td>n</td>\n      <td>S</td>\n      <td>10</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "There were 7 people on board with 8 siblings/spouses and 2 parents/children--they were probably all siblings of one another. Tragically, all 7 of them passed away. The 8th sibling is likely in the test data for which we are supposed make predictions. Would you predict that the final sibling survived or died?"
      ],
      "metadata": {
        "id": "lg1-sLgNpno7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Wrap Up"
      ],
      "metadata": {
        "id": "Ce1WiOwcpno7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this lesson, we covered several general questions you should address when you first inspect a data set. Your first goal should be to explore the structure of the data to clean it and prepare the variables for your analysis. Once your data is it the right form, you can move from exploring structure to investigating relationships between variables. Since data cleaning and formatting is a vital and often time-consuming part of data analysis, it is important that we spend some time learning how to work with different types of data. Over the next few lessons, we'll learn about cleaning and preprocessing text data, numeric data and dates."
      ],
      "metadata": {
        "id": "EdjQO4jYpno7"
      }
    }
  ]
}