fraud_detection/fintech_creditcard_fraud.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "b2eebf0d-31ff-4ce0-b2b7-4d82ee61150b",
      "metadata": {
        "id": "b2eebf0d-31ff-4ce0-b2b7-4d82ee61150b"
      },
      "source": [
        "# Detecting Data Quality Issues in Credit Card Fraud Detection Dataset\n",
        "\n",
        "This example uses **Datalab** to auto-detect various issues in a tabular dataset commonly encountered in financial applications. Specifically, the [Credit Card Fraud Detection dataset](https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud), which you should first download and ensure the data file is named **credit_card_fraud_dataset.csv**. This dataset contains thousands of transaction records labeled as fraudulent or non-fraudulent, and features about each transaction such as the transaction amount plus other variables anonymized for privacy. \n",
        "\n",
        "### Cleanlab Helps Uncover:\n",
        "- **Label errors**: Mislabeled transactions, such as fraudulent cases incorrectly marked as non-fraudulent.\n",
        "- **Outliers**: Transactions with abnormal patterns that deviate significantly from the rest of the dataset.\n",
        "- **Near-duplicates**: Repeated transactions or entries that may distort results or impact model performance.\n",
        "\n",
        "Using Cleanlab, we automatically identify examples that are likely mislabeled or otherwise problematic, improving the overall data quality for better fraud detection performance. You can adapt this tutorial to detect and correct issues in your own financial tabular datasets."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "791717d7-d140-4d85-b516-9a6e6f28c7c0",
      "metadata": {
        "id": "791717d7-d140-4d85-b516-9a6e6f28c7c0"
      },
      "source": [
        "## 1. Install and Import Dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f7TpME1a6-Db",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f7TpME1a6-Db",
        "outputId": "9f5642dc-ae12-464b-e65f-77d0f57b4ce0"
      },
      "outputs": [],
      "source": [
        "!pip install \"cleanlab[all]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "2af19c9f-970f-4f2e-ab48-392b376c0b98",
      "metadata": {
        "id": "2af19c9f-970f-4f2e-ab48-392b376c0b98"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.model_selection import cross_val_predict\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.neighbors import NearestNeighbors\n",
        "from cleanlab import Datalab\n",
        "\n",
        "# Optional: set seed for reproducibility\n",
        "SEED = 42\n",
        "np.random.seed(SEED)\n",
        "random.seed(SEED)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3ef577c5-b3f1-4d5b-9b26-9324651b12fd",
      "metadata": {
        "id": "3ef577c5-b3f1-4d5b-9b26-9324651b12fd"
      },
      "source": [
        "## 2. Load and Process the Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "ea80ab8d-6461-47d9-ac45-7048540b4650",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "ea80ab8d-6461-47d9-ac45-7048540b4650",
        "outputId": "35249fcf-22d4-433c-faa9-5871138a6db0"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"fraud_data\",\n  \"rows\": 100000,\n  \"fields\": [\n    {\n      \"column\": \"TransactionID\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 28867,\n        \"min\": 1,\n        \"max\": 100000,\n        \"num_unique_values\": 100000,\n        \"samples\": [\n          75722,\n          80185,\n          19865\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionDate\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"num_unique_values\": 100000,\n        \"samples\": [\n          \"2024-08-18 01:11:35.918051\",\n          \"2024-06-09 07:44:35.939541\",\n          \"2024-06-10 08:55:35.558368\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1442.4159985963513,\n        \"min\": 1.05,\n        \"max\": 4999.77,\n        \"num_unique_values\": 90621,\n        \"samples\": [\n          3273.37,\n          4040.01,\n          4120.55\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"MerchantID\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 288,\n        \"min\": 1,\n        \"max\": 1000,\n        \"num_unique_values\": 1000,\n        \"samples\": [\n          702,\n          152,\n          346\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"purchase\",\n          \"refund\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 10,\n        \"samples\": [\n          \"Houston\",\n          \"Dallas\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"IsFraud\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "fraud_data"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1fa2024e-e3de-4936-9fa2-090dc948076d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>TransactionID</th>\n",
              "      <th>TransactionDate</th>\n",
              "      <th>Amount</th>\n",
              "      <th>MerchantID</th>\n",
              "      <th>TransactionType</th>\n",
              "      <th>Location</th>\n",
              "      <th>IsFraud</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>2024-04-03 14:15:35.462794</td>\n",
              "      <td>4189.27</td>\n",
              "      <td>688</td>\n",
              "      <td>refund</td>\n",
              "      <td>San Antonio</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>2024-03-19 13:20:35.462824</td>\n",
              "      <td>2659.71</td>\n",
              "      <td>109</td>\n",
              "      <td>refund</td>\n",
              "      <td>Dallas</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>2024-01-08 10:08:35.462834</td>\n",
              "      <td>784.00</td>\n",
              "      <td>394</td>\n",
              "      <td>purchase</td>\n",
              "      <td>New York</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>2024-04-13 23:50:35.462850</td>\n",
              "      <td>3514.40</td>\n",
              "      <td>944</td>\n",
              "      <td>purchase</td>\n",
              "      <td>Philadelphia</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>2024-07-12 18:51:35.462858</td>\n",
              "      <td>369.07</td>\n",
              "      <td>475</td>\n",
              "      <td>purchase</td>\n",
              "      <td>Phoenix</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1fa2024e-e3de-4936-9fa2-090dc948076d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1fa2024e-e3de-4936-9fa2-090dc948076d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1fa2024e-e3de-4936-9fa2-090dc948076d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-551315e6-4862-409f-9786-3910638f4897\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-551315e6-4862-409f-9786-3910638f4897')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-551315e6-4862-409f-9786-3910638f4897 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "   TransactionID             TransactionDate   Amount  MerchantID  \\\n",
              "0              1  2024-04-03 14:15:35.462794  4189.27         688   \n",
              "1              2  2024-03-19 13:20:35.462824  2659.71         109   \n",
              "2              3  2024-01-08 10:08:35.462834   784.00         394   \n",
              "3              4  2024-04-13 23:50:35.462850  3514.40         944   \n",
              "4              5  2024-07-12 18:51:35.462858   369.07         475   \n",
              "\n",
              "  TransactionType      Location  IsFraud  \n",
              "0          refund   San Antonio        0  \n",
              "1          refund        Dallas        0  \n",
              "2        purchase      New York        0  \n",
              "3        purchase  Philadelphia        0  \n",
              "4        purchase       Phoenix        0  "
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "fraud_data = pd.read_csv(\"credit_card_fraud_dataset.csv\")\n",
        "fraud_data.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "a2e0a535-7ece-4607-ba19-ee6e7eac9374",
      "metadata": {
        "id": "a2e0a535-7ece-4607-ba19-ee6e7eac9374"
      },
      "outputs": [],
      "source": [
        "# Select relevant features and labels\n",
        "X_raw = fraud_data[[\"Amount\", \"TransactionType\", \"Location\"]]\n",
        "y = fraud_data[\"IsFraud\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2600e93b-60d9-4d54-b8a6-092563c19aff",
      "metadata": {
        "id": "2600e93b-60d9-4d54-b8a6-092563c19aff"
      },
      "source": [
        "We will now preprocess the dataset to prepare it for analysis. This involves:\n",
        "1. Selecting relevant features (e.g., `Amount`, `TransactionType`, `Location`).\n",
        "2. Encoding categorical variables (e.g., `TransactionType` and `Location`) using one-hot encoding.\n",
        "3. Standardizing numerical variables (e.g., `Amount`) to ensure all features are on a similar scale.\n",
        "\n",
        "Next, we assign the preprocessed features to `X` and the labels (`IsFraud`) to `y`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "814c39d6-4490-47db-8a0a-ded45fc5a09a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "814c39d6-4490-47db-8a0a-ded45fc5a09a",
        "outputId": "6d8567f0-8396-450c-abc1-9fba07e1c4f5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "     Amount  TransactionType_refund  Location_Dallas  Location_Houston  \\\n",
            "0  1.173161                    True            False             False   \n",
            "1  0.112740                    True             True             False   \n",
            "2 -1.187661                   False            False             False   \n",
            "3  0.705284                   False            False             False   \n",
            "4 -1.475326                   False            False             False   \n",
            "\n",
            "   Location_Los Angeles  Location_New York  Location_Philadelphia  \\\n",
            "0                 False              False                  False   \n",
            "1                 False              False                  False   \n",
            "2                 False               True                  False   \n",
            "3                 False              False                   True   \n",
            "4                 False              False                  False   \n",
            "\n",
            "   Location_Phoenix  Location_San Antonio  Location_San Diego  \\\n",
            "0             False                  True               False   \n",
            "1             False                 False               False   \n",
            "2             False                 False               False   \n",
            "3             False                 False               False   \n",
            "4              True                 False               False   \n",
            "\n",
            "   Location_San Jose  \n",
            "0              False  \n",
            "1              False  \n",
            "2              False  \n",
            "3              False  \n",
            "4              False  \n"
          ]
        }
      ],
      "source": [
        "# One-hot encode categorical features\n",
        "categorical_features = [\"TransactionType\", \"Location\"]\n",
        "X_encoded = pd.get_dummies(X_raw, columns=categorical_features, drop_first=True)\n",
        "\n",
        "# Standardize numerical features\n",
        "numeric_features = [\"Amount\"]\n",
        "scaler = StandardScaler()\n",
        "X_encoded[numeric_features] = scaler.fit_transform(X_encoded[numeric_features])\n",
        "\n",
        "# Display preprocessed data\n",
        "print(X_encoded.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d394b6cd-34fd-4ec0-8f6f-99d13dd481ab",
      "metadata": {
        "id": "d394b6cd-34fd-4ec0-8f6f-99d13dd481ab"
      },
      "source": [
        "## 3. Train a Classification Model and Compute Out-of-Sample Predicted Probabilities\n",
        "\n",
        "To detect potential label errors in the **Credit Card Fraud Detection dataset**, Cleanlab requires **probabilistic predictions** for every data point. However, predictions generated on the same data used for training can be **overfitted** and unreliable. For accurate results, Cleanlab works best with **out-of-sample** predicted class probabilities—i.e., predictions for data points excluded from the model during training.\n",
        "\n",
        "#### How We Generate Out-of-Sample Predictions\n",
        "\n",
        "We use **K-fold cross-validation**, which:\n",
        "1. Splits the dataset into `K` folds.\n",
        "2. Trains the model on `K-1` folds and predicts probabilities on the excluded fold.\n",
        "3. Repeats this for all folds so that every data point gets a prediction from a model that has not seen it during training.\n",
        "\n",
        "This ensures every data point has **out-of-sample predicted probabilities**.\n",
        "\n",
        "#### Model: Logistic Regression\n",
        "\n",
        "For this example, we use **Logistic Regression**, a simple and interpretable model commonly used in fraud detection tasks. It predicts the probability of each class (`0` for non-fraud, `1` for fraud) based on the input features of an example in the dataset. The same approach will work with *any* Machine Learning model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "dd2ccbc6-824a-44cd-bd1d-26e0a39e23a4",
      "metadata": {
        "id": "dd2ccbc6-824a-44cd-bd1d-26e0a39e23a4"
      },
      "outputs": [],
      "source": [
        "clf = LogisticRegression(max_iter=1000, random_state=SEED)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "bb9327e7-bc63-45a0-8d2c-7950186c961f",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bb9327e7-bc63-45a0-8d2c-7950186c961f",
        "outputId": "1024212d-a062-4a11-8599-f792c7d48892"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of predicted probabilities: (100000, 2)\n"
          ]
        }
      ],
      "source": [
        "num_crossval_folds = 5\n",
        "pred_probs = cross_val_predict(\n",
        "    clf,\n",
        "    X_encoded,     # Preprocessed feature matrix\n",
        "    y,             # Labels\n",
        "    cv=num_crossval_folds,\n",
        "    method=\"predict_proba\"  # Get predicted probabilities\n",
        ")\n",
        "\n",
        "print(\"Shape of predicted probabilities:\", pred_probs.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "2681aad2-d84d-4713-bed8-aa1204223fd5",
      "metadata": {
        "id": "2681aad2-d84d-4713-bed8-aa1204223fd5"
      },
      "source": [
        "## 4. Construct a K Nearest Neighbors Graph (Optional)\n",
        "\n",
        "The **KNN graph** represents the similarity between examples in the dataset.\n",
        "Here, we'll define similarity using the **Euclidean distance** between our normalized feature values.\n",
        "\n",
        "Note that this step is *optional*. If you pass in numerical `features` to Datalab but no KNN graph, then Datalab will internally construct its own KNN graph.\n",
        "You can provide your KNN graph as done here to exert greater control over this process, or do this whenever your data aren't in a numerical format or you have a massive dataset (use [approxmate KNN](https://docs.cleanlab.ai/stable/tutorials/datalab/workflows.html#Accelerate-Issue-Checks-with-Pre-computed-kNN-Graphs) in that case).\n",
        "\n",
        "Here we use scikit-learn's `NearestNeighbors` class to construct this graph:\n",
        "1. Compute pairwise distances between all examples.\n",
        "2. Represent the graph as a sparse matrix, with nonzero entries indicating the distance to nearest neighbors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "6c49780c-5d65-4360-ad0b-90e6717ecddb",
      "metadata": {
        "id": "6c49780c-5d65-4360-ad0b-90e6717ecddb"
      },
      "outputs": [],
      "source": [
        "# Create a KNN model with Euclidean distance as the metric\n",
        "knn = NearestNeighbors(metric=\"euclidean\")\n",
        "\n",
        "# Fit the KNN model to the preprocessed feature values\n",
        "knn.fit(X_encoded.values)\n",
        "\n",
        "# Construct the KNN graph as a sparse matrix\n",
        "knn_graph = knn.kneighbors_graph(mode=\"distance\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b27cf2de-e276-438a-8f0a-9a5de3d1757a",
      "metadata": {
        "id": "b27cf2de-e276-438a-8f0a-9a5de3d1757a"
      },
      "source": [
        "## 5. Use Datalab to Find Dataset Issues\n",
        "\n",
        "With the given labels, predicted probabilities, and the KNN graph (optional), Datalab can automatically identifies various issues in the dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "58ca8740-1e44-4959-8567-ec4ee2535bfa",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "58ca8740-1e44-4959-8567-ec4ee2535bfa",
        "outputId": "dc353a68-3607-46e0-fb87-07fcf651c288"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Finding label issues ...\n",
            "Finding outlier issues ...\n",
            "Finding near_duplicate issues ...\n",
            "Finding non_iid issues ...\n",
            "Finding class_imbalance issues ...\n",
            "Finding underperforming_group issues ...\n",
            "\n",
            "Audit complete. 12043 issues found in the dataset.\n"
          ]
        }
      ],
      "source": [
        "from cleanlab import Datalab\n",
        "# Wrap the dataset into a dictionary\n",
        "data = {\"X\": X_encoded.values, \"y\": y}\n",
        "\n",
        "# Create a Datalab object\n",
        "lab = Datalab(data, label_name=\"y\")\n",
        "\n",
        "# Use Cleanlab to find issues in the dataset\n",
        "lab.find_issues(pred_probs=pred_probs, knn_graph=knn_graph)  # could provide features here instead of knn_graph"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "ef82352b-eb2e-4c7a-91d0-436fc61e16ac",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ef82352b-eb2e-4c7a-91d0-436fc61e16ac",
        "outputId": "73358440-6c05-4fb6-9beb-ad5fc35055cc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset Information: num_examples: 100000, num_classes: 2\n",
            "\n",
            "Here is a summary of various issues found in your data:\n",
            "\n",
            "     issue_type  num_issues\n",
            " near_duplicate        8639\n",
            "        outlier        1797\n",
            "class_imbalance        1000\n",
            "          label         607\n",
            "\n",
            "Learn about each issue: https://docs.cleanlab.ai/stable/cleanlab/datalab/guide/issue_type_description.html\n",
            "See which examples in your dataset exhibit each issue via: `datalab.get_issues(<ISSUE_NAME>)`\n",
            "\n",
            "Data indices corresponding to top examples of each issue are shown below.\n",
            "\n",
            "\n",
            "------------------ near_duplicate issues -------------------\n",
            "\n",
            "About this issue:\n",
            "\tA (near) duplicate issue refers to two or more examples in\n",
            "    a dataset that are extremely similar to each other, relative\n",
            "    to the rest of the dataset.  The examples flagged with this issue\n",
            "    may be exactly duplicated, or lie atypically close together when\n",
            "    represented as vectors (i.e. feature embeddings).\n",
            "    \n",
            "\n",
            "Number of examples with this issue: 8639\n",
            "Overall dataset quality in terms of this issue: 0.5894\n",
            "\n",
            "Examples representing most severe instances of this issue:\n",
            "       is_near_duplicate_issue  near_duplicate_score near_duplicate_sets  distance_to_nearest_neighbor\n",
            "62583                     True                   0.0             [55080]                           0.0\n",
            "30333                     True                   0.0             [13617]                           0.0\n",
            "12827                     True                   0.0             [15703]                           0.0\n",
            "66741                     True                   0.0             [82920]                           0.0\n",
            "45125                     True                   0.0             [95476]                           0.0\n",
            "\n",
            "\n",
            "---------------------- outlier issues ----------------------\n",
            "\n",
            "About this issue:\n",
            "\tExamples that are very different from the rest of the dataset \n",
            "    (i.e. potentially out-of-distribution or rare/anomalous instances).\n",
            "    \n",
            "\n",
            "Number of examples with this issue: 1797\n",
            "Overall dataset quality in terms of this issue: 0.3784\n",
            "\n",
            "Examples representing most severe instances of this issue:\n",
            "       is_outlier_issue  outlier_score\n",
            "43484              True       0.003062\n",
            "4659               True       0.007290\n",
            "67602              True       0.007582\n",
            "91994              True       0.007898\n",
            "52696              True       0.008608\n",
            "\n",
            "\n",
            "------------------ class_imbalance issues ------------------\n",
            "\n",
            "About this issue:\n",
            "\tExamples belonging to the most under-represented class in the dataset.\n",
            "\n",
            "Number of examples with this issue: 1000\n",
            "Overall dataset quality in terms of this issue: 0.0100\n",
            "\n",
            "Examples representing most severe instances of this issue:\n",
            "       is_class_imbalance_issue  class_imbalance_score  given_label\n",
            "68852                      True                   0.01            1\n",
            "22652                      True                   0.01            1\n",
            "33819                      True                   0.01            1\n",
            "5781                       True                   0.01            1\n",
            "44573                      True                   0.01            1\n",
            "\n",
            "Additional Information: \n",
            "Rarest Class: 1\n",
            "\n",
            "\n",
            "----------------------- label issues -----------------------\n",
            "\n",
            "About this issue:\n",
            "\tExamples whose given label is estimated to be potentially incorrect\n",
            "    (e.g. due to annotation error) are flagged as having label issues.\n",
            "    \n",
            "\n",
            "Number of examples with this issue: 607\n",
            "Overall dataset quality in terms of this issue: 0.9939\n",
            "\n",
            "Examples representing most severe instances of this issue:\n",
            "       is_label_issue  label_score  given_label  predicted_label\n",
            "6901             True     0.006965            1                0\n",
            "7933             True     0.007031            1                0\n",
            "13204            True     0.007065            1                0\n",
            "16276            True     0.007086            1                0\n",
            "7546             True     0.007124            1                0\n"
          ]
        }
      ],
      "source": [
        "lab.report()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qBcATrTFCWqJ",
      "metadata": {
        "id": "qBcATrTFCWqJ"
      },
      "source": [
        "## Label Issues\n",
        "The report indicates that Cleanlab identified several label issues in the dataset. These are data entries where the given labels may not match the actual label, as estimated by Cleanlab. Each issue includes a numeric label score that quantifies how likely the label is correct (lower scores indicate higher likelihood of being mislabeled)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "pee_lWpiCiIV",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pee_lWpiCiIV",
        "outputId": "d5bcf570-0051-4b92-df49-20c3479b88b1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "   is_label_issue  label_score  given_label  predicted_label\n",
            "0           False     0.990469            0                0\n",
            "1           False     0.991203            0                0\n",
            "2           False     0.988302            0                0\n",
            "3           False     0.990321            0                0\n",
            "4           False     0.991149            0                0\n"
          ]
        }
      ],
      "source": [
        "# Retrieve label issues\n",
        "label_issues = lab.get_issues(\"label\")\n",
        "\n",
        "print(label_issues.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "8pGqVz8RDeoF",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8pGqVz8RDeoF",
        "outputId": "322a6eb8-4b2f-4597-9b8f-614c97887a45"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "     is_label_issue  label_score  given_label  predicted_label\n",
            "190            True     0.007187            1                0\n",
            "191            True     0.007622            1                0\n",
            "208            True     0.007177            1                0\n",
            "319            True     0.008984            1                0\n",
            "506            True     0.009220            1                0\n"
          ]
        }
      ],
      "source": [
        "# Filter rows with label issues\n",
        "label_issues_filtered = label_issues[label_issues['is_label_issue'] == True]\n",
        "print(label_issues_filtered.head())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "m1KP2zEWDfaE",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "m1KP2zEWDfaE",
        "outputId": "6fc9c1b0-30a0-4c3f-f015-44e42202c166"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \")\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"Amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 438.6116871789898,\n        \"min\": 25.91,\n        \"max\": 1093.22,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          25.91,\n          598.78,\n          963.84\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"refund\",\n          \"purchase\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"San Jose\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"given_label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"predicted_label\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-b5f43500-9ac4-48f8-ae62-3335bd3814fc\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Amount</th>\n",
              "      <th>TransactionType</th>\n",
              "      <th>Location</th>\n",
              "      <th>given_label</th>\n",
              "      <th>predicted_label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6901</th>\n",
              "      <td>346.13</td>\n",
              "      <td>purchase</td>\n",
              "      <td>San Jose</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7933</th>\n",
              "      <td>25.91</td>\n",
              "      <td>refund</td>\n",
              "      <td>San Jose</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13204</th>\n",
              "      <td>963.84</td>\n",
              "      <td>purchase</td>\n",
              "      <td>San Jose</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16276</th>\n",
              "      <td>1093.22</td>\n",
              "      <td>purchase</td>\n",
              "      <td>San Jose</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7546</th>\n",
              "      <td>598.78</td>\n",
              "      <td>refund</td>\n",
              "      <td>San Jose</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b5f43500-9ac4-48f8-ae62-3335bd3814fc')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b5f43500-9ac4-48f8-ae62-3335bd3814fc button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b5f43500-9ac4-48f8-ae62-3335bd3814fc');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-51b6006f-0304-4e73-a574-1d60b6abb89c\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-51b6006f-0304-4e73-a574-1d60b6abb89c')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-51b6006f-0304-4e73-a574-1d60b6abb89c button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "        Amount TransactionType  Location  given_label  predicted_label\n",
              "6901    346.13        purchase  San Jose            1                0\n",
              "7933     25.91          refund  San Jose            1                0\n",
              "13204   963.84        purchase  San Jose            1                0\n",
              "16276  1093.22        purchase  San Jose            1                0\n",
              "7546    598.78          refund  San Jose            1                0"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Sort the label issues by label_score (lower scores indicate higher likelihood of being mislabeled)\n",
        "sorted_issues = label_issues.sort_values(\"label_score\").index\n",
        "\n",
        "# View the most likely label errors\n",
        "X_raw.iloc[sorted_issues].assign(\n",
        "    given_label=y.iloc[sorted_issues],\n",
        "    predicted_label=label_issues[\"predicted_label\"].iloc[sorted_issues]\n",
        ").head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "-ApyX5r6FTmI",
      "metadata": {
        "id": "-ApyX5r6FTmI"
      },
      "source": [
        "### Example Review of Label Issues\n",
        "\n",
        "The dataframe below shows the original label (`given_label`) for examples that Cleanlab finds most likely to be mislabeled, as well as an alternative `predicted_label` for each example.\n",
        "\n",
        "| Amount  | TransactionType | Location  | given_label | predicted_label |\n",
        "|---------|------------------|-----------|-------------|-----------------|\n",
        "| 346.13  | purchase         | San Jose  | 1           | 0               |\n",
        "| 25.91   | refund           | San Jose  | 1           | 0               |\n",
        "| 963.84  | purchase         | San Jose  | 1           | 0               |\n",
        "| 1093.22 | purchase         | San Jose  | 1           | 0               |\n",
        "| 598.78  | refund           | San Jose  | 1           | 0               |\n",
        "\n",
        "These examples may have been labeled incorrectly and should be carefully re-examined:\n",
        "- **Entry 1**: A purchase of 346.13 labeled as fraudulent (`1`) is predicted to be non-fraudulent (`0`).\n",
        "- **Entry 2**: A refund of  25.91 is similarly labeled as fraudulent but predicted as non-fraudulent.\n",
        "- **Entry 4**: A purchase of $1093.22 also seems misclassified as fraudulent.\n",
        "\n",
        "The predicted labels suggest a potential mislabeling pattern for transactions in `San Jose`. Transactions with relatively lower amounts or refunds might have been mislabeled as fraudulent. This should be reviewed with additional domain knowledge or transaction metadata for confirmation.\n",
        "\n",
        "Such insights are crucial for improving the dataset's quality and ensuring the model learns from accurate labels.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "_zzPdWl0GFOY",
      "metadata": {
        "id": "_zzPdWl0GFOY"
      },
      "source": [
        "\n",
        "### Outlier Issues\n",
        "\n",
        "According to the report, our dataset contains some outliers. We can see which examples are outliers (and a numeric quality score quantifying how typical each example appears to be) via the `get_issues` method. We sort the resulting DataFrame by Cleanlab’s outlier quality score to see the most severe outliers in our dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "D7VClp15GIXC",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "D7VClp15GIXC",
        "outputId": "ec6c23c3-1802-42ba-d2aa-69537251da5d"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"X_raw\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"Amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1519.3915375570575,\n        \"min\": 1147.93,\n        \"max\": 4999.73,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          2114.37,\n          4005.05,\n          3255.47\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"refund\",\n          \"purchase\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Location\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 4,\n        \"samples\": [\n          \"Philadelphia\",\n          \"San Antonio\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-46bca668-ea2a-40b7-a280-39e7933beb22\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Amount</th>\n",
              "      <th>TransactionType</th>\n",
              "      <th>Location</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>43484</th>\n",
              "      <td>4999.73</td>\n",
              "      <td>purchase</td>\n",
              "      <td>Chicago</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4659</th>\n",
              "      <td>2114.37</td>\n",
              "      <td>refund</td>\n",
              "      <td>Philadelphia</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67602</th>\n",
              "      <td>3255.47</td>\n",
              "      <td>purchase</td>\n",
              "      <td>San Jose</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91994</th>\n",
              "      <td>1147.93</td>\n",
              "      <td>refund</td>\n",
              "      <td>Chicago</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52696</th>\n",
              "      <td>4005.05</td>\n",
              "      <td>purchase</td>\n",
              "      <td>San Antonio</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-46bca668-ea2a-40b7-a280-39e7933beb22')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-46bca668-ea2a-40b7-a280-39e7933beb22 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-46bca668-ea2a-40b7-a280-39e7933beb22');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-274c6028-115e-4dcc-a3d3-3c4f0df19f78\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-274c6028-115e-4dcc-a3d3-3c4f0df19f78')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-274c6028-115e-4dcc-a3d3-3c4f0df19f78 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "        Amount TransactionType      Location\n",
              "43484  4999.73        purchase       Chicago\n",
              "4659   2114.37          refund  Philadelphia\n",
              "67602  3255.47        purchase      San Jose\n",
              "91994  1147.93          refund       Chicago\n",
              "52696  4005.05        purchase   San Antonio"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "outlier_results = lab.get_issues(\"outlier\")\n",
        "sorted_outliers = outlier_results.sort_values(\"outlier_score\").index\n",
        "\n",
        "X_raw.iloc[sorted_outliers].head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "qQYl8X5RG9F6",
      "metadata": {
        "id": "qQYl8X5RG9F6"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "\n",
        "#### **Key Observations**:\n",
        "1. **Entry 1**: A purchase transaction with an unusually high amount of `$4999.73` in Chicago may represent a legitimate but rare high-value transaction or could be indicative of an error.\n",
        "2. **Entry 2**: A refund for `$2114.37` in Philadelphia seems unusually high compared to typical refund amounts and should be verified.\n",
        "3. **Entry 5**: Another high-value purchase transaction of `$4005.05` in San Antonio is rare and should be reviewed for validity.\n",
        "\n",
        "#### **Next Steps**:\n",
        "- **Investigate Outliers**:\n",
        "  - Validate whether these transactions are legitimate or the result of data errors.\n",
        "  - Cross-check these entries against metadata such as timestamps, merchants, and customer profiles for better context.\n",
        "- **Handle Outliers**:\n",
        "  - **Retain**: If the transaction is valid, keep it in the dataset for training.\n",
        "  - **Remove**: If the transaction is deemed erroneous or unrepresentative, exclude it from the dataset to avoid skewing the model's learning.\n",
        "\n",
        "These steps will ensure that the dataset is representative and does not include suspicious entries that could affect the performance of fraud detection models.\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "id": "STlYZFJRRDtO",
      "metadata": {
        "id": "STlYZFJRRDtO"
      },
      "source": [
        "### Near-Duplicate Issues\n",
        "\n",
        "According to the report, our dataset contains some sets of nearly duplicated examples. We can see which examples are (nearly) duplicated (and a numeric quality score quantifying how dissimilar each example is from its nearest neighbor in the dataset) via `get_issues`. We sort the resulting DataFrame by Cleanlab’s near-duplicate quality score to see the examples in our dataset that are most nearly duplicated (the score is equal to 0 if examples are exactly duplicated)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "VHcPnNYbQZ-n",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "VHcPnNYbQZ-n",
        "outputId": "7dc6f1fe-ac78-4c77-96e5-176c7f3a6a16"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"duplicate_results\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"is_near_duplicate_issue\",\n      \"properties\": {\n        \"dtype\": \"boolean\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          true\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"near_duplicate_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.0,\n        \"max\": 0.0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"near_duplicate_sets\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"distance_to_nearest_neighbor\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 0.0,\n        \"max\": 0.0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-42757ddd-f3d9-4e8c-b95c-254d8e92043d\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>is_near_duplicate_issue</th>\n",
              "      <th>near_duplicate_score</th>\n",
              "      <th>near_duplicate_sets</th>\n",
              "      <th>distance_to_nearest_neighbor</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>62583</th>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[55080]</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30333</th>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[13617]</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12827</th>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[15703]</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66741</th>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[82920]</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45125</th>\n",
              "      <td>True</td>\n",
              "      <td>0.0</td>\n",
              "      <td>[95476]</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-42757ddd-f3d9-4e8c-b95c-254d8e92043d')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-42757ddd-f3d9-4e8c-b95c-254d8e92043d button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-42757ddd-f3d9-4e8c-b95c-254d8e92043d');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c0ddef4d-eaea-4231-8ef5-001220e1699a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c0ddef4d-eaea-4231-8ef5-001220e1699a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c0ddef4d-eaea-4231-8ef5-001220e1699a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "       is_near_duplicate_issue  near_duplicate_score near_duplicate_sets  \\\n",
              "62583                     True                   0.0             [55080]   \n",
              "30333                     True                   0.0             [13617]   \n",
              "12827                     True                   0.0             [15703]   \n",
              "66741                     True                   0.0             [82920]   \n",
              "45125                     True                   0.0             [95476]   \n",
              "\n",
              "       distance_to_nearest_neighbor  \n",
              "62583                           0.0  \n",
              "30333                           0.0  \n",
              "12827                           0.0  \n",
              "66741                           0.0  \n",
              "45125                           0.0  "
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "duplicate_results = lab.get_issues(\"near_duplicate\")\n",
        "duplicate_results.sort_values(\"near_duplicate_score\").head()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0FyG5cJtRNGb",
      "metadata": {
        "id": "0FyG5cJtRNGb"
      },
      "source": [
        "The results above show which examples Cleanlab considers nearly duplicated (rows where is_near_duplicate_issue == True). Here, we see some examples that Cleanlab has flagged as being nearly duplicated. Let’s view these examples to see how similar they are."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "IqgcWEVIROAP",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 143
        },
        "id": "IqgcWEVIROAP",
        "outputId": "eb36a8cd-a66e-4f3d-eb68-c7aac6ef27b5"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"X_raw\",\n  \"rows\": 3,\n  \"fields\": [\n    {\n      \"column\": \"Amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.011547005383782014,\n        \"min\": 3374.61,\n        \"max\": 3374.63,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          3374.63,\n          3374.61\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"refund\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Location\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"New York\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-a18947c4-2bfe-4434-842a-663e654b0ee3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Amount</th>\n",
              "      <th>TransactionType</th>\n",
              "      <th>Location</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>73</th>\n",
              "      <td>3374.61</td>\n",
              "      <td>refund</td>\n",
              "      <td>New York</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19427</th>\n",
              "      <td>3374.61</td>\n",
              "      <td>refund</td>\n",
              "      <td>New York</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30450</th>\n",
              "      <td>3374.63</td>\n",
              "      <td>refund</td>\n",
              "      <td>New York</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-a18947c4-2bfe-4434-842a-663e654b0ee3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-a18947c4-2bfe-4434-842a-663e654b0ee3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-a18947c4-2bfe-4434-842a-663e654b0ee3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-aa0fc0d2-3e8f-4bb9-8cb6-735e3e9ed7ed\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-aa0fc0d2-3e8f-4bb9-8cb6-735e3e9ed7ed')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-aa0fc0d2-3e8f-4bb9-8cb6-735e3e9ed7ed button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "        Amount TransactionType  Location\n",
              "73     3374.61          refund  New York\n",
              "19427  3374.61          refund  New York\n",
              "30450  3374.63          refund  New York"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Identify the row with the lowest near_duplicate_score\n",
        "lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].idxmin()\n",
        "\n",
        "# Extract the indices of the lowest scoring duplicate and its near duplicate sets\n",
        "indices_to_display = [lowest_scoring_duplicate] + duplicate_results.loc[lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
        "\n",
        "# Display the relevant rows from the original dataset\n",
        "X_raw.iloc[indices_to_display]\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6nhecZHHSuv9",
      "metadata": {
        "id": "6nhecZHHSuv9"
      },
      "source": [
        "These examples are exact duplicates! Perhaps the same information was accidentally recorded multiple times in this data.\n",
        "\n",
        "Similarly, let’s take a look at another example and the identified near-duplicate sets:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "94gQWzVkRW53",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "94gQWzVkRW53",
        "outputId": "106f3513-d065-4483-dc76-e6c28e614b39"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"X_raw\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"Amount\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 1796.39,\n        \"max\": 1796.39,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1796.39\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TransactionType\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"refund\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Location\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"New York\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1dc32bdb-7824-4e6d-9218-a44a95f200c3\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Amount</th>\n",
              "      <th>TransactionType</th>\n",
              "      <th>Location</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>167</th>\n",
              "      <td>1796.39</td>\n",
              "      <td>refund</td>\n",
              "      <td>New York</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53564</th>\n",
              "      <td>1796.39</td>\n",
              "      <td>refund</td>\n",
              "      <td>New York</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1dc32bdb-7824-4e6d-9218-a44a95f200c3')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1dc32bdb-7824-4e6d-9218-a44a95f200c3 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1dc32bdb-7824-4e6d-9218-a44a95f200c3');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3336eb5f-e09b-4355-a801-b5fde09f5360\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3336eb5f-e09b-4355-a801-b5fde09f5360')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3336eb5f-e09b-4355-a801-b5fde09f5360 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "        Amount TransactionType  Location\n",
              "167    1796.39          refund  New York\n",
              "53564  1796.39          refund  New York"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Identify the next row not in the previous near duplicate set\n",
        "second_lowest_scoring_duplicate = duplicate_results[\"near_duplicate_score\"].drop(indices_to_display).idxmin()\n",
        "\n",
        "# Extract the indices of the second lowest scoring duplicate and its near duplicate sets\n",
        "next_indices_to_display = [second_lowest_scoring_duplicate] + duplicate_results.loc[second_lowest_scoring_duplicate, \"near_duplicate_sets\"].tolist()\n",
        "\n",
        "# Display the relevant rows from the original dataset\n",
        "X_raw.iloc[next_indices_to_display]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6vexriCMTCAG",
      "metadata": {
        "id": "6vexriCMTCAG"
      },
      "source": [
        "We identified another set of exact duplicates in our dataset! Including near/exact duplicates in a dataset may have unintended effects on models; be wary about splitting them across training/test sets. Learn more about handling near duplicates detected in a dataset from the [FAQ](https://docs.cleanlab.ai/stable/tutorials/faq.html#How-to-handle-near-duplicate-data-identified-by-Datalab?).\n",
        "\n",
        "This tutorial highlights a straightforward approach to detect potentially incorrect information in any tabular dataset. Just use Cleanlab with any ML model – the better the model, the more accurate the data issues detected by Cleanlab will be!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}