DP0701EN-3-3-1-Clustering-k-means-py-v1.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<a href=\"https://cognitiveclass.ai\"><img src = \"https://ibm.box.com/shared/static/9gegpsmnsoo25ikkbl4qzlvlyjbgxs5x.png\" width = 400> </a>\n",
    "\n",
    "<h1 align=center><font size = 5><em>k</em>-means Clustering</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Introduction\n",
    "\n",
    "There are many models for clustering out there. In this lab, we will be presenting the model that is considered the one of the simplest model among them. Despite its simplicity, *k*-means is vastly used for clustering in many data science applications, especially useful if you need to quickly discover insights from unlabeled data.\n",
    "\n",
    "Some real-world applications of *k*-means include:\n",
    "- customer segmentation,\n",
    "- understand what the visitors of a website are trying to accomplish,\n",
    "- pattern recognition, and,\n",
    "- data compression.\n",
    "\n",
    "In this lab, we will learn *k*-means clustering with 3 examples:\n",
    "- *k*-means on a randomly generated dataset.\n",
    "- Using *k*-means for customer segmentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Table of Contents\n",
    "\n",
    "1. <a href=\"#item1\"><em>k</em>-means on a Randomly Generated Dataset</a>    \n",
    "2. <a href=\"#item2\">Using <em>k</em> for Customer Segmentation</a> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Before we start with the main lab content, let's download all the dependencies that we will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import random # library for random number generation\n",
    "import numpy as np # library for vectorized computation\n",
    "import pandas as pd # library to process data as dataframes\n",
    "\n",
    "import matplotlib.pyplot as plt # plotting library\n",
    "# backend for rendering plots within the browser\n",
    "%matplotlib inline \n",
    "\n",
    "from sklearn.cluster import KMeans \n",
    "from sklearn.datasets.samples_generator import make_blobs\n",
    "\n",
    "print('Libraries imported.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<a id='item1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## 1. *k*-means on a Randomly Generated Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's first demonstrate how *k*-means works with an example of engineered datapoints. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### 30 data points belonging to 2 different clusters (x1 is the first feature and x2 is the second feature)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# data\n",
    "x1 = [-4.9, -3.5, 0, -4.5, -3, -1, -1.2, -4.5, -1.5, -4.5, -1, -2, -2.5, -2, -1.5, 4, 1.8, 2, 2.5, 3, 4, 2.25, 1, 0, 1, 2.5, 5, 2.8, 2, 2]\n",
    "x2 = [-3.5, -4, -3.5, -3, -2.9, -3, -2.6, -2.1, 0, -0.5, -0.8, -0.8, -1.5, -1.75, -1.75, 0, 0.8, 0.9, 1, 1, 1, 1.75, 2, 2.5, 2.5, 2.5, 2.5, 3, 6, 6.5]\n",
    "\n",
    "print('Datapoints defined!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Define a function that assigns each datapoint to a cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "colors_map = np.array(['b', 'r'])\n",
    "def assign_members(x1, x2, centers):\n",
    "    compare_to_first_center = np.sqrt(np.square(np.array(x1) - centers[0][0]) + np.square(np.array(x2) - centers[0][1]))\n",
    "    compare_to_second_center = np.sqrt(np.square(np.array(x1) - centers[1][0]) + np.square(np.array(x2) - centers[1][1]))\n",
    "    class_of_points = compare_to_first_center > compare_to_second_center\n",
    "    colors = colors_map[class_of_points + 1 - 1]\n",
    "    return colors, class_of_points\n",
    "\n",
    "print('assign_members function defined!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Define a function that updates the centroid of each cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# update means\n",
    "def update_centers(x1, x2, class_of_points):\n",
    "    center1 = [np.mean(np.array(x1)[~class_of_points]), np.mean(np.array(x2)[~class_of_points])]\n",
    "    center2 = [np.mean(np.array(x1)[class_of_points]), np.mean(np.array(x2)[class_of_points])]\n",
    "    return [center1, center2]\n",
    "\n",
    "print('assign_members function defined!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Define a function that plots the data points along with the cluster centroids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def plot_points(centroids=None, colors='g', figure_title=None):\n",
    "    # plot the figure\n",
    "    fig = plt.figure(figsize=(15, 10))  # create a figure object\n",
    "    ax = fig.add_subplot(1, 1, 1)\n",
    "    \n",
    "    centroid_colors = ['bx', 'rx']\n",
    "    if centroids:\n",
    "        for (i, centroid) in enumerate(centroids):\n",
    "            ax.plot(centroid[0], centroid[1], centroid_colors[i], markeredgewidth=5, markersize=20)\n",
    "    plt.scatter(x1, x2, s=500, c=colors)\n",
    "    \n",
    "    # define the ticks\n",
    "    xticks = np.linspace(-6, 8, 15, endpoint=True)\n",
    "    yticks = np.linspace(-6, 6, 13, endpoint=True)\n",
    "\n",
    "    # fix the horizontal axis\n",
    "    ax.set_xticks(xticks)\n",
    "    ax.set_yticks(yticks)\n",
    "\n",
    "    # add tick labels\n",
    "    xlabels = xticks\n",
    "    ax.set_xticklabels(xlabels)\n",
    "    ylabels = yticks\n",
    "    ax.set_yticklabels(ylabels)\n",
    "\n",
    "    # style the ticks\n",
    "    ax.xaxis.set_ticks_position('bottom')\n",
    "    ax.yaxis.set_ticks_position('left')\n",
    "    ax.tick_params('both', length=2, width=1, which='major', labelsize=15)\n",
    "    \n",
    "    # add labels to axes\n",
    "    ax.set_xlabel('x1', fontsize=20)\n",
    "    ax.set_ylabel('x2', fontsize=20)\n",
    "    \n",
    "    # add title to figure\n",
    "    ax.set_title(figure_title, fontsize=24)\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "print('plot_points function defined!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Initialize *k*-means - plot data points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "plot_points(figure_title='Scatter Plot of x2 vs x1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Initialize *k*-means - randomly define clusters and add them to plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "centers = [[-2, 2], [2, -2]]\n",
    "plot_points(centers, figure_title='k-means Initialization')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Run *k*-means (4-iterations only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "number_of_iterations = 4\n",
    "for i in range(number_of_iterations):\n",
    "    input('Iteration {} - Press Enter to update the members of each cluster'.format(i + 1))\n",
    "    colors, class_of_points = assign_members(x1, x2, centers)\n",
    "    title = 'Iteration {} - Cluster Assignment'.format(i + 1)\n",
    "    plot_points(centers, colors, figure_title=title)\n",
    "    input('Iteration {} - Press Enter to update the centers'.format(i + 1))\n",
    "    centers = update_centers(x1, x2, class_of_points)\n",
    "    title = 'Iteration {} - Centroid Update'.format(i + 1)\n",
    "    plot_points(centers, colors, figure_title=title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now, we have visually observed how <em>k</em>-means works, let's look at an example with many more datapoints. For this example, we will use the <strong>random</strong> library to generate thousands of datapoints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Generating the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "First, we need to set up a random seed. We use the Numpy's **random.seed()** function, and we will set the seed to 0. In other words, **random.seed(0)**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Next we will be making *random clusters* of points by using the **make_blobs** class. The **make_blobs** class can take in many inputs, but we will use these specific ones.\n",
    "\n",
    "<b> <u> Input </u> </b>\n",
    "<ul>\n",
    "    <li> <b>n_samples</b>: The total number of points equally divided among clusters. </li>\n",
    "    <ul> <li> Value will be: 5000 </li> </ul>\n",
    "    <li> <b>centers</b>: The number of centers to generate, or the fixed center locations. </li>\n",
    "    <ul> <li> Value will be: [[4, 4], [-2, -1], [2, -3],[1,1]] </li> </ul>\n",
    "    <li> <b>cluster_std</b>: The standard deviation of the clusters. </li>\n",
    "    <ul> <li> Value will be: 0.9 </li> </ul>\n",
    "</ul>\n",
    "\n",
    "<b> <u> Output </u> </b>\n",
    "<ul>\n",
    "    <li> <b>X</b>: Array of shape [n_samples, n_features]. (Feature Matrix)</li>\n",
    "    <ul> <li> The generated samples. </li> </ul> \n",
    "    <li> <b>y</b>: Array of shape [n_samples]. (Response Vector)</li>\n",
    "    <ul> <li> The integer labels for cluster membership of each sample. </li> </ul>\n",
    "</ul>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "X, y = make_blobs(n_samples=5000, centers=[[4, 4], [-2, -1], [2, -3], [1, 1]], cluster_std=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Display the scatter plot of the randomly generated data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 10))\n",
    "plt.scatter(X[:, 0], X[:, 1], marker='.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Setting up *k*-means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now that we have our random data, let's set up our *k*-means clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "The KMeans class has many parameters that can be used, but we will use these three:\n",
    "<ul>\n",
    "    <li> <strong>init</strong>: Initialization method of the centroids. </li>\n",
    "    <ul>\n",
    "        <li> Value will be: \"k-means++\". k-means++ selects initial cluster centers for <em>k</em>-means clustering in a smart way to speed up convergence.</li>\n",
    "    </ul>\n",
    "    <li> <strong>n_clusters</strong>: The number of clusters to form as well as the number of centroids to generate. </li>\n",
    "    <ul> <li> Value will be: 4 (since we have 4 centers)</li> </ul>\n",
    "    <li> <strong>n_init</strong>: Number of times the <em>k</em>-means algorithm will be run with different centroid seeds. The final results will be the best output of n_init consecutive runs in terms of inertia. </li>\n",
    "    <ul> <li> Value will be: 12 </li> </ul>\n",
    "</ul>\n",
    "\n",
    "Initialize KMeans with these parameters, where the output parameter is called **k_means**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": true,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "k_means = KMeans(init=\"k-means++\", n_clusters=4, n_init=12)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's fit the KMeans model with the feature matrix we created above, <b> X </b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "k_means.fit(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's grab the labels for each point in the model using KMeans **.labels\\_** attribute and save it as **k_means_labels**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "k_means_labels = k_means.labels_\n",
    "k_means_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will also get the coordinates of the cluster centers using KMeans **.cluster\\_centers\\_** and save it as **k_means_cluster_centers**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "k_means_cluster_centers = k_means.cluster_centers_\n",
    "k_means_cluster_centers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Visualizing the Resulting Clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "So now that we have the random data generated and the KMeans model initialized, let's plot them and see what the clusters look like."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Please read through the code and comments to understand how to plot the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# initialize the plot with the specified dimensions.\n",
    "fig = plt.figure(figsize=(15, 10))\n",
    "\n",
    "# colors uses a color map, which will produce an array of colors based on\n",
    "# the number of labels. We use set(k_means_labels) to get the\n",
    "# unique labels.\n",
    "colors = plt.cm.Spectral(np.linspace(0, 1, len(set(k_means_labels))))\n",
    "\n",
    "# create a plot\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "\n",
    "# loop through the data and plot the datapoints and centroids.\n",
    "# k will range from 0-3, which will match the number of clusters in the dataset.\n",
    "for k, col in zip(range(len([[4,4], [-2, -1], [2, -3], [1, 1]])), colors):\n",
    "\n",
    "    # create a list of all datapoints, where the datapoitns that are \n",
    "    # in the cluster (ex. cluster 0) are labeled as true, else they are\n",
    "    # labeled as false.\n",
    "    my_members = (k_means_labels == k)\n",
    "    \n",
    "    # define the centroid, or cluster center.\n",
    "    cluster_center = k_means_cluster_centers[k]\n",
    "    \n",
    "    # plot the datapoints with color col.\n",
    "    ax.plot(X[my_members, 0], X[my_members, 1], 'w', markerfacecolor=col, marker='.')\n",
    "    \n",
    "    # plot the centroids with specified color, but with a darker outline\n",
    "    ax.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,  markeredgecolor='k', markersize=6)\n",
    "\n",
    "# title of the plot\n",
    "ax.set_title('KMeans')\n",
    "\n",
    "# remove x-axis ticks\n",
    "ax.set_xticks(())\n",
    "\n",
    "# remove y-axis ticks\n",
    "ax.set_yticks(())\n",
    "\n",
    "# show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<a id='item2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## 2. Using *k*-means for Customer Segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Imagine that you have a customer dataset, and you are interested in exploring the behavior of your customers using their historical data.\n",
    "\n",
    "Customer segmentation is the practice of partitioning a customer base into groups of individuals that have similar characteristics. It is a significant strategy as a business can target these specific groups of customers and effectively allocate marketing resources. For example, one group might contain customers who are high-profit and low-risk, that is, more likely to purchase products, or subscribe to a service. A business task is to retain those customers. Another group might include customers from non-profit organizations, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Downloading Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's download the data and save it as a CSV file called **customer_segmentation.csv**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "!wget -q -O 'customer_segmentation.csv' https://cocl.us/customer_dataset\n",
    "print('Data downloaded!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now that the data is downloaded, let's read it into a *pandas* dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "customers_df = pd.read_csv('customer_segmentation.csv')\n",
    "customers_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As you can see, **Address** in this dataset is a categorical variable. k-means algorithm isn't directly applicable to categorical variables because Euclidean distance function isn't really meaningful for discrete variables. So, lets drop this feature and run clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df = customers_df.drop('Address', axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now let's normalize the dataset. But why do we need normalization in the first place? Normalization is a statistical method that helps mathematical-based algorithms interpret features with different magnitudes and distributions equally. We use **StandardScaler()** to normalize our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "X = df.values[:,1:]\n",
    "X = np.nan_to_num(X)\n",
    "cluster_dataset = StandardScaler().fit_transform(X)\n",
    "cluster_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's run our model and group our customers into three clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_clusters = 3\n",
    "\n",
    "k_means = KMeans(init=\"k-means++\", n_clusters=num_clusters, n_init=12)\n",
    "k_means.fit(cluster_dataset)\n",
    "labels = k_means.labels_\n",
    "\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2 id=\"insights\">Insights</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Note that each row in our dataset represents a customer, and therefore, each row is assigned a label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df[\"Labels\"] = labels\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can easily check the centroid values by averaging the features in each cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "collapsed": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "df.groupby('Labels').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<em>k</em>-means will partition your customers into three groups since we specified the algorithm to generate 3 clusters. The customers in each cluster are similar to each other in terms of the features included in the dataset.\n",
    "\n",
    "Now we can create a profile for each group, considering the common characteristics of each cluster. \n",
    "For example, the 3 clusters can be:\n",
    "\n",
    "- OLDER, HIGH INCOME, AND INDEBTED\n",
    "- MIDDLE AGED, MIDDLE INCOME, AND FINANCIALLY RESPONSIBLE\n",
    "- YOUNG, LOW INCOME, AND INDEBTED"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can devise your own profiles based on the means above and come up with labels that you think best describe each cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "I hope that you are able to see the power of *k*-means here. This clustering algorithm provided us with insight into the dataset and lead us to group the data into three clusters. Perhaps the same results would have been achieved but using multiple tests and experiments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Thank you for completing this lab!\n",
    "\n",
    "This notebook was created by [Saeed Aghabozorgi](https://ca.linkedin.com/in/saeedaghabozorgi) and [Alex Aklson](https://www.linkedin.com/in/aklson/). We hope you found this lab interesting and educational. Feel free to contact us if you have any questions!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "This notebook is part of a course on **Coursera** called *Applied Data Science Capstone*. If you accessed this notebook outside the course, you can take this course online by clicking [here](http://cocl.us/DP0701EN_Coursera_Week3_LAB1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cognitiveclass.ai/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}