Unsupervised learning & clustering¶
1. Reading data¶
The woldbank_development_2015.csv (can be found in the same folder with this notebook) file contains the World Development Indicators for the 2015 year, downloaded from The World Bank's webpage.
- Look at the data in any text editor. Build up an overall sense how the data is built up and how is the missing values are represented.
- Read the file into a pandas dataframe and tell pandas which special pattern means if a value is missing.
- The data is in a long format. Convert it into a wide format, where each row is a single country and the columns are the measured features, the Series Codes (eg the first column is 'EG.CFT.ACCS.ZS', the second is 'EG.ELC.ACCS.ZS'. Order of the columns does not matter)!
- Keep only those rows, which represents counties and NOT regions. Luckily they are well separated in the order they occur!
- Convert the features to numeric format, which will be needed for modeling!
2. Data preprocessing and inspection¶
- Visualize the missing values!
- Keep only those countries which has less than 700 missing features in the original table and keep features which were missing in less than 20 country in the original table
- Visualize the missing values again! Now really only a few entries are missing. Impute the missing values with its feature's mean value!
- How many counties and features do we have left?
- Read the kept features' descriptions. In the original table the Series Name describe the meaning of the features. What do you think, based only on these information, which counties are the most similar to Hungary? And Norway?
3. PCA¶
- Perform PCA with 3 principal components on the filtered, imputed data (from now on, data refers to the filtered, imputed dataset)
- Plot the three embedded 2D combination next to each other (0 vs 1, 0 vs 2 and 1 vs 2)
- It seems that the embedding is really dominated by a single direction. Normalize the data (each feature should have zero mean and unit variance after normalization) and re-do the PCA and the plotting (do not delete the previous plots, just make new ones).
4. T-SNE¶
- Perform T-SNE on the scaled data with 2 components
- Plot the embeddings results. Add a text label for each point to make it possible to interpret the results. It will not be possible to read all, but try to make it useful, see the attached image as an example!
- Highlight Hungary and Norway! Which countries are the closest one to Hungary and Norway?
5. Hierarchical clustering¶
- Perform hierarchical clustering on the filtered and scaled data (hint: use seaborn)
- Try to plot in a way that all country's name is visible
- Write down your impressions that you got from this plot!
Hints:¶
- On total you can get 10 points for fully completing all tasks.
- Decorate your notebook with, questions, explanation etc, make it self contained and understandable!
- Comments you code when necessary
- Write functions for repetitive tasks!
- Use the pandas package for data loading and handling
- Use matplotlib and seaborn for plotting or bokeh and plotly for interactive investigation
- Use the scikit learn package for almost everything
- Use for loops only if it is really necessary!
- Code sharing is not allowed between student! Sharing code will result in zero points.
- If you use code found on web, it is OK, but, make its source clear!
Solution¶
1. Reading data¶
The woldbank_development_2015.csv (can be found in the same folder with this notebook) file contains the World Development Indicators for the 2015 year, downloaded from The World Bank's webpage.
- Look at the data in any text editor. Build up an overall sense how the data is built up and how is the missing values are represented.
- Read the file into a pandas dataframe and tell pandas which special pattern means if a value is missing.
- The data is in a long format. Convert it into a wide format, where each row is a single country and the columns are the measured features, the Series Codes (eg the first column is 'EG.CFT.ACCS.ZS', the second is 'EG.ELC.ACCS.ZS'. Order of the columns does not matter)!
In [1]:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
%matplotlib inline
In [2]:
long = pd.read_csv('woldbank_development_2015.csv', na_values=['..', '...'])
long.head()
Out[2]:
In [3]:
pd.unique(long['Country Name'])[210:225]
Out[3]:
In [4]:
%%time
series_codes = pd.unique(long['Series Code'].dropna())
wide = pd.DataFrame()
# had to manually split when countries stops and regions starts
for c in pd.unique(long['Country Name'])[:217]:
tmp = long[long['Country Name'] == c][['Series Code', '2015 [YR2015]']].set_index('Series Code').T
# tmp is here:
## index: 2015 [YR2015]
## columns: Series Code
tmp['country'] = c
wide = wide.append(tmp[['country'] + list(series_codes)])
wide = wide.set_index('country').astype(float)
In [5]:
wide.head()
Out[5]:
In [6]:
# keep only the countries first, then pivot it
long2 = long[long['Country Name'].isin(pd.unique(long['Country Name'])[:217])]
pivoted = long2.pivot(index='Country Name', columns='Series Code', values='2015 [YR2015]')
pivoted[series_codes].head()
Out[6]:
In [7]:
np.allclose(pivoted[series_codes].fillna(0), wide.fillna(0))
# in general we do not want to compare floats with == too much
Out[7]:
In [8]:
np.allclose??
2. Data preprocessing and inspection¶
- Visualize the missing values!
- Keep only those countries which has less than 700 missing features in the original table and keep features which were missing in less than 20 country in the original table
- Visualize the missing values again! Now really only a few entries are missing. Impute the missing values with its feature's mean value!
- How many counties and features do we have left?
- Read the kept features' descriptions. In the original table the Series Name describe the meaning of the features. What do you think, based only on these information, which counties are the most similar to Hungary? And Norway?
In [10]:
wide.isna()
Out[10]:
In [9]:
plt.imshow(wide.isna())
Out[9]:
In [11]:
keep_cols = wide.isna().sum()[wide.isna().sum() < 20].index.values
keep_rows = wide.isna().sum(1)[wide.isna().sum(1) < 700].index.values
filtered_wide = wide[wide.index.isin(keep_rows)][keep_cols]
filtered_wide.shape
Out[11]:
In [12]:
plt.imshow(filtered_wide.isna())
Out[12]:
In [13]:
filtered_wide = filtered_wide.fillna(filtered_wide.mean())
In [14]:
long[long['Series Code'].isin(keep_cols)]['Series Name'].drop_duplicates().tolist()
Out[14]:
3. PCA¶
- Perform PCA with 3 principal components on the filtered, imputed data (from now on, data refers to the filtered, imputed dataset)
- Plot the three embedded 2D combination next to each other (0 vs 1, 0 vs 2 and 1 vs 2)
- It seems that the embedding is really dominated by a single direction. Normalize the data (each feature should have zero mean and unit variance after normalization) and re-do the PCA and the plotting (do not delete the previous plots, just make new ones).
In [15]:
pca = PCA(3)
pca_transformed = pca.fit_transform(filtered_wide)
In [16]:
plt.figure(figsize=(17, 6))
plt.subplot(131)
plt.scatter(pca_transformed[:,0], pca_transformed[:,1])
plt.subplot(132)
plt.scatter(pca_transformed[:,0], pca_transformed[:,2])
plt.subplot(133)
plt.scatter(pca_transformed[:,1], pca_transformed[:,2])
Out[16]:
In [17]:
filtered_scaled_wide = (filtered_wide - filtered_wide.mean())/filtered_wide.std()
In [18]:
pca = PCA(3)
pca_transformed = pca.fit_transform(filtered_scaled_wide)
In [19]:
plt.figure(figsize=(17, 6))
plt.subplot(131)
plt.scatter(pca_transformed[:,0], pca_transformed[:,1])
plt.subplot(132)
plt.scatter(pca_transformed[:,0], pca_transformed[:,2])
plt.subplot(133)
plt.scatter(pca_transformed[:,1], pca_transformed[:,2])
Out[19]:
4. T-SNE¶
- Perform T-SNE on the scaled data with 2 components
- Plot the embeddings results. Add a text label for each point to make it possible to interpret the results. It will not be possible to read all, but try to make it useful, see the attached image as an example!
- Highlight Hungary and Norway! Which countries are the closest one to Hungary and Norway?
In [33]:
tsne = TSNE()
ts_transformed = tsne.fit_transform(filtered_scaled_wide)
plt.scatter(ts_transformed[:,0], ts_transformed[:,1])
Out[33]:
In [21]:
countries = filtered_scaled_wide.index.tolist()
In [22]:
plt.figure(figsize=(16, 12))
plt.scatter(ts_transformed[:,0], ts_transformed[:,1])
for idx, c in enumerate(countries):
if c in ['Hungary', 'Norway']:
plt.text(ts_transformed[idx, 0], ts_transformed[idx, 1], c, fontsize=15)
plt.scatter([ts_transformed[idx,0]], [ts_transformed[idx,1]], c='r', s=150)
plt.text(ts_transformed[idx, 0], ts_transformed[idx, 1], c, fontsize=8)
In [23]:
plt.figure(figsize=(30, 20))
plt.scatter(ts_transformed[:,0], ts_transformed[:,1])
for idx, c in enumerate(countries):
if c in ['Hungary', 'Norway']:
plt.text(ts_transformed[idx, 0], ts_transformed[idx, 1], c, fontsize=15)
plt.scatter([ts_transformed[idx,0]], [ts_transformed[idx,1]], c='r', s=150)
plt.text(ts_transformed[idx, 0], ts_transformed[idx, 1], c, fontsize=8)
5. Hierarchical clustering¶
- Perform hierarchical clustering on the filtered and scaled data (hint: use seaborn)
- Try to plot in a way that all country's name is visible
- Write down your impressions that you got from this plot!
In [34]:
sns.clustermap(filtered_scaled_wide, col_cluster=False, figsize=(10, 45))
Out[34]:
