A Data-Driven Approach To Cryptocurrency Speculation

How do Bitcoin markets behave? What are the causes of the sudden spikes and dips in cryptocurrency values? Are the markets for different altcoins inseparably linked or largely independent? How can we predict what will happen next?

Articles on cryptocurrencies, such as Bitcoin and Ethereum, are rife with speculation these days, with hundreds of self-proclaimed experts advocating for the trends that they expect to emerge. What is lacking from many of these analyses is a strong foundation of data and statistics to backup the claims.

The goal of this article is to provide an easy introduction to cryptocurrency analysis using Python. We will walk through a simple Python script to retrieve, analyze, and visualize data on different cryptocurrencies. In the process, we will uncover an interesting trend in how these volatile markets behave, and how they are evolving.

This is not a post explaining what cryptocurrencies are (if you want one, I would recommend this great overview), nor is it an opinion piece on which specific currencies will rise and which will fall. Instead, all that we are concerned about in this tutorial is procuring the raw data and uncovering the stories hidden in the numbers.

Step 1 - Setup Your Data Laboratory

The tutorial is intended to be accessible for enthusiasts, engineers, and data scientists at all skill levels. The only skills that you will need are a basic understanding of Python and enough knowledge of the command line to setup a project.

A completed version of the notebook with all of the results is available here.

Step 1.1 - Install Anaconda

The easiest way to install the dependencies for this project from scratch is to use Anaconda, a prepackaged Python data science ecosystem and dependency manager.

To setup Anaconda, I would recommend following the official installation instructions - https://www.continuum.io/downloads.

If you're an advanced user, and you don't want to use Anaconda, that's totally fine; I'll assume you don't need help installing the required dependencies. Feel free to skip to section 2.

Step 1.2 - Setup an Anaconda Project Environment

Once Anaconda is installed, we'll want to create a new environment to keep our dependencies organized.

Run conda create --name cryptocurrency-analysis python=3 to create a new Anaconda environment for our project.

Next, run source activate cryptocurrency-analysis (on Linux/macOS) or activate cryptocurrency-analysis (on windows) to activate this environment.

Finally, run conda install numpy pandas nb_conda jupyter plotly quandl to install the required dependencies in the environment. This could take a few minutes to complete.

Why use environments? If you plan on developing multiple Python projects on your computer, it is helpful to keep the dependencies (software libraries and packages) separate in order to avoid conflicts. Anaconda will create a special environment directory for the dependencies for each project to keep everything organized and separated.

Step 1.3 - Start An Interative Jupyter Notebook

Once the environment and dependencies are all set up, run jupyter notebook to start the iPython kernel, and open your browser to https://localhost:8888/. Create a new Python notebook, making sure to use the Python [conda env:cryptocurrency-analysis] kernel.

Step 1.4 - Import the Dependencies At The Top of The Notebook

Once you've got a blank Jupyter notebook open, the first thing we'll do is import the required dependencies.

import os
import numpy as np
import pandas as pd
import pickle
import quandl
from datetime import datetime

We'll also import Plotly and enable the offline mode.

import plotly.offline as py
import plotly.graph_objs as go
import plotly.figure_factory as ff
py.init_notebook_mode(connected=True)

Step 2 - Retrieve Bitcoin Pricing Data

Now that everything is set up, we're ready to start retrieving data for analysis. First, we need to get Bitcoin pricing data using Quandl's free Bitcoin API.

Step 2.1 - Define Quandl Helper Function

To assist with this data retrieval we'll define a function to download and cache datasets from Quandl.

def get_quandl_data(quandl_id):
    '''Download and cache Quandl dataseries'''
    cache_path = '{}.pkl'.format(quandl_id).replace('/','-')
    try:
        f = open(cache_path, 'rb')
        df = pickle.load(f)   
        print('Loaded {} from cache'.format(quandl_id))
    except (OSError, IOError) as e:
        print('Downloading {} from Quandl'.format(quandl_id))
        df = quandl.get(quandl_id, returns="pandas")
        df.to_pickle(cache_path)
        print('Cached {} at {}'.format(quandl_id, cache_path))
    return df

We're using pickle to serialize and save the downloaded data as a file, which will prevent our script from re-downloading the same data each time we run the script. The function will return the data as a Pandas dataframe. If you're not familiar with dataframes, you can think of them as super-powered spreadsheets.

Step 2.2 - Pull Kraken Exchange Pricing Data

Let's first pull the historical Bitcoin exchange rate for the Kraken Bitcoin exchange.

# Pull Kraken BTC price exchange data
btc_usd_price_kraken = get_quandl_data('BCHARTS/KRAKENUSD')

We can inspect the first 5 rows of the dataframe using the head() method.

btc_usd_price_kraken.head()

Next, we'll generate a simple chart as a quick visual verification that the data looks correct.

# Chart the BTC pricing data
btc_trace = go.Scatter(x=btc_usd_price_kraken.index, y=btc_usd_price_kraken['Weighted Price'])
py.iplot([btc_trace])

Here, we're using Plotly for generating our visualizations. This is a less traditional choice than some of the more established Python data visualization libraries such as Matplotlib, but I think Plotly is a great choice since it produces fully-interactive charts using D3.js. These charts have attractive visual defaults, are easy to explore, and are very simple to embed in web pages.

As a quick sanity check, you should compare the generated chart with publicly available graphs on Bitcoin prices(such as those on Coinbase), to verify that the downloaded data is legit.

Step 2.3 - Pull Pricing Data From More BTC Exchanges

You might have noticed a hitch in this dataset - there are a few notable down-spikes, particularly in late 2014 and early 2016. These spikes are specific to the Kraken dataset, and we obviously don't want them to be reflected in our overall pricing analysis.

The nature of Bitcoin exchanges is that the pricing is determined by supply and demand, hence no single exchange contains a true "master price" of Bitcoin. To solve this issue, along with that of down-spikes (which are likely the result of technical outages and data set glitches) we will pull data from three more major Bitcoin exchanges to calculate an aggregate Bitcoin price index.

First, we will download the data from each exchange into a dictionary of dataframes.

# Pull pricing data for 3 more BTC exchanges
exchanges = ['COINBASE','BITSTAMP','ITBIT']
exchange_data = {}
exchange_data['KRAKEN'] = btc_usd_price_kraken
for exchange in exchanges:
    exchange_code = 'BCHARTS/{}USD'.format(exchange)
    btc_exchange_df = get_quandl_data(exchange_code)
    exchange_data[exchange] = btc_exchange_df

Step 2.4 - Merge All Of The Pricing Data Into A Single Dataframe

Next, we will define a simple function to merge a common column of each dataframe into a new combined dataframe.

def merge_dfs_on_column(dataframes, labels, col):
    '''Merge a single column of each dataframe into a new combined dataframe'''
    series_dict = {}
    for index in range(len(dataframes)):
        series_dict[labels[index]] = dataframes[index][col]
        
    return pd.DataFrame(series_dict)

Now we will merge all of the dataframes together on their "Weighted Price" column.

# Merge the BTC price dataseries' into a single dataframe
btc_usd_datasets = merge_dfs_on_column(list(exchange_data.values()), list(exchange_data.keys()), 'Weighted Price')

Finally, we can preview last five rows the result using the tail() method, to make sure it looks ok.

btc_usd_datasets.tail()

The prices look to be as expected: they are in similar ranges, but with slight variations based on the supply and demand of each individual Bitcoin exchange.

Step 2.5 - Visualize The Pricing Datasets

The next logical step is to visualize how these pricing datasets compare. For this, we'll define a helper function to provide a single-line command to generate a graph from the dataframe.

def df_scatter(df, title, seperate_y_axis=False, y_axis_label='', scale='linear', initial_hide=False):
    '''Generate a scatter plot of the entire dataframe'''
    label_arr = list(df)
    series_arr = list(map(lambda col: df[col], label_arr))
    
    layout = go.Layout(
        title=title,
        legend=dict(orientation="h"),
        xaxis=dict(type='date'),
        yaxis=dict(
            title=y_axis_label,
            showticklabels= not seperate_y_axis,
            type=scale
        )
    )
    
    y_axis_config = dict(
        overlaying='y',
        showticklabels=False,
        type=scale )
    
    visibility = 'visible'
    if initial_hide:
        visibility = 'legendonly'
        
    # Form Trace For Each Series
    trace_arr = []
    for index, series in enumerate(series_arr):
        trace = go.Scatter(
            x=series.index, 
            y=series, 
            name=label_arr[index],
            visible=visibility
        )
        
        # Add seperate axis for the series
        if seperate_y_axis:
            trace['yaxis'] = 'y{}'.format(index + 1)
            layout['yaxis{}'.format(index + 1)] = y_axis_config    
        trace_arr.append(trace)
    fig = go.Figure(data=trace_arr, layout=layout)
    py.iplot(fig)

In the interest of brevity, I won't go too far into how this helper function works. Check out the documentation for Pandas and Plotly if you would like to learn more.

We can now easily generate a graph for the Bitcoin pricing data.

# Plot all of the BTC exchange prices
df_scatter(btc_usd_datasets, 'Bitcoin Price (USD) By Exchange')

Step 2.6 - Clean and Aggregate the Pricing Data

We can see that, although the four series follow roughly the same path, there are various irregularities in each that we'll want to get rid of.

Let's remove all of the zero values from the dataframe, since we know that the price of Bitcoin has never been equal to zero in the timeframe that we are examining.

# Remove "0" values
btc_usd_datasets.replace(0, np.nan, inplace=True)

When we re-chart the dataframe, we'll see a much cleaner looking chart without the down-spikes.

# Plot the revised dataframe
df_scatter(btc_usd_datasets, 'Bitcoin Price (USD) By Exchange')

We can now calculate a new column, containing the average daily Bitcoin price across all of the exchanges.

# Calculate the average BTC price as a new column
btc_usd_datasets['avg_btc_price_usd'] = btc_usd_datasets.mean(axis=1)

This new column is our Bitcoin pricing index! Let's chart that column to make sure it looks ok.

# Plot the average BTC price
btc_trace = go.Scatter(x=btc_usd_datasets.index, y=btc_usd_datasets['avg_btc_price_usd'])
py.iplot([btc_trace])

Yup, looks good. We'll use this aggregate pricing series later on, in order to convert the exchange rates of other cryptocurrencies to USD.

Step 3 - Retrieve Altcoin Pricing Data

Now that we have a solid time series dataset for the price of Bitcoin, let's pull in some data for non-Bitcoin cryptocurrencies, commonly referred to as altcoins.

Step 3.1 - Define Poloniex API Helper Functions

For retrieving data on cryptocurrencies we'll be using the Poloniex API. To assist in the altcoin data retrieval, we'll define two helper functions to download and cache JSON data from this API.

First, we'll define get_json_data, which will download and cache JSON data from a provided URL.

def get_json_data(json_url, cache_path):
    '''Download and cache JSON data, return as a dataframe.'''
    try:        
        f = open(cache_path, 'rb')
        df = pickle.load(f)   
        print('Loaded {} from cache'.format(json_url))
    except (OSError, IOError) as e:
        print('Downloading {}'.format(json_url))
        df = pd.read_json(json_url)
        df.to_pickle(cache_path)
        print('Cached {} at {}'.format(json_url, cache_path))
    return df

Next, we'll define a function that will generate Poloniex API HTTP requests, and will subsequently call our new get_json_data function to save the resulting data.

base_polo_url = 'https://poloniex.com/public?command=returnChartData&currencyPair={}&start={}&end={}&period={}'
start_date = datetime.strptime('2015-01-01', '%Y-%m-%d') # get data from the start of 2015
end_date = datetime.now() # up until today
pediod = 86400 # pull daily data (86,400 seconds per day)
def get_crypto_data(poloniex_pair):
    '''Retrieve cryptocurrency data from poloniex'''
    json_url = base_polo_url.format(poloniex_pair, start_date.timestamp(), end_date.timestamp(), pediod)
    data_df = get_json_data(json_url, poloniex_pair)
    data_df = data_df.set_index('date')
    return data_df

This function will take a cryptocurrency pair string (such as 'BTC_ETH') and return a dataframe containing the historical exchange rate of the two currencies.

Step 3.2 - Download Trading Data From Poloniex

Most altcoins cannot be bought directly with USD; to acquire these coins individuals often buy Bitcoins and then trade the Bitcoins for altcoins on cryptocurrency exchanges. For this reason, we'll be downloading the exchange rate to BTC for each coin, and then we'll use our existing BTC pricing data to convert this value to USD.

We'll download exchange data for nine of the top cryptocurrencies -
Ethereum, Litecoin, Ripple, Ethereum Classic, Stellar, Dash, Siacoin, Monero, and NEM.

altcoins = ['ETH','LTC','XRP','ETC','STR','DASH','SC','XMR','XEM']
altcoin_data = {}
for altcoin in altcoins:
    coinpair = 'BTC_{}'.format(altcoin)
    crypto_price_df = get_crypto_data(coinpair)
    altcoin_data[altcoin] = crypto_price_df

Now we have a dictionary with 9 dataframes, each containing the historical daily average exchange prices between the altcoin and Bitcoin.

We can preview the last few rows of the Ethereum price table to make sure it looks ok.

altcoin_data['ETH'].tail()

Step 3.3 - Convert Prices to USD

Now we can combine this BTC-altcoin exchange rate data with our Bitcoin pricing index to directly calculate the historical USD values for each altcoin.

# Calculate USD Price as a new column in each altcoin dataframe
for altcoin in altcoin_data.keys():
    altcoin_data[altcoin]['price_usd'] =  altcoin_data[altcoin]['weightedAverage'] * btc_usd_datasets['avg_btc_price_usd']

Here, we've created a new column in each altcoin dataframe with the USD prices for that coin.

Next, we can re-use our merge_dfs_on_column function from earlier to create a combined dataframe of the USD price for each cryptocurrency.

# Merge USD price of each altcoin into single dataframe 
combined_df = merge_dfs_on_column(list(altcoin_data.values()), list(altcoin_data.keys()), 'price_usd')

Easy. Now let's also add the Bitcoin prices as a final column to the combined dataframe.

# Add BTC price to the dataframe
combined_df['BTC'] = btc_usd_datasets['avg_btc_price_usd']

Now we should have a single dataframe containing daily USD prices for the ten cryptocurrencies that we're examining.

Let's reuse our df_scatter function from earlier to chart all of the cryptocurrency prices against each other.

# Chart all of the altocoin prices
df_scatter(combined_df, 'Cryptocurrency Prices (USD)', seperate_y_axis=False, y_axis_label='Coin Value (USD)', scale='log')

Nice! This graph provides a pretty solid "big picture" view of how the exchange rates for each currency have varied over the past few years.

Note that we're using a logarithmic y-axis scale in order to compare all of the currencies on the same plot. You are welcome to try out different parameter values here (such as scale='linear') to get different perspectives on the data.

Step 3.4 - Perform Correlation Analysis

You might notice is that the cryptocurrency exchange rates, despite their wildly different values and volatility, look slightly correlated. Especially since the spike in April 2017, even many of the smaller fluctuations appear to be occurring in sync across the entire market.

A visually-derived hunch is not much better than a guess until we have the stats to back it up.

We can test our correlation hypothesis using the Pandas corr() method, which computes a Pearson correlation coefficient for each column in the dataframe against each other column.

Revision Note 8/22/2017 - This section has been revised in order to use the daily return percentages instead of the absolute price values in calculating the correlation coefficients.

Computing correlations directly on a non-stationary time series (such as raw pricing data) can give biased correlation values. We will work around this by first applying the pct_change() method, which will convert each cell in the dataframe from an absolute price value to a daily return percentage.

First we'll calculate correlations for 2016.

# Calculate the pearson correlation coefficients for cryptocurrencies in 2016
combined_df_2016 = combined_df[combined_df.index.year == 2016]
combined_df_2016.pct_change().corr(method='pearson')

These correlation coefficients are all over the place. Coefficients close to 1 or -1 mean that the series' are strongly correlated or inversely correlated respectively, and coefficients close to zero mean that the values are not correlated, and fluctuate independently of each other.

To help visualize these results, we'll create one more helper visualization function.

def correlation_heatmap(df, title, absolute_bounds=True):
    '''Plot a correlation heatmap for the entire dataframe'''
    heatmap = go.Heatmap(
        z=df.corr(method='pearson').as_matrix(),
        x=df.columns,
        y=df.columns,
        colorbar=dict(title='Pearson Coefficient'),
    )
    
    layout = go.Layout(title=title)
    
    if absolute_bounds:
        heatmap['zmax'] = 1.0
        heatmap['zmin'] = -1.0
        
    fig = go.Figure(data=[heatmap], layout=layout)
    py.iplot(fig)

correlation_heatmap(combined_df_2016.pct_change(), "Cryptocurrency Correlations in 2016")

Here, the dark red values represent strong correlations (note that each currency is, obviously, strongly correlated with itself), and the dark blue values represent strong inverse correlations. All of the light blue/orange/gray/tan colors in-between represent varying degrees of weak/non-existent correlations.

What does this chart tell us? Essentially, it shows that there was little statistically significant linkage between how the prices of different cryptocurrencies fluctuated during 2016.

Now, to test our hypothesis that the cryptocurrencies have become more correlated in recent months, let's repeat the same test using only the data from 2017.

combined_df_2017 = combined_df[combined_df.index.year == 2017]
combined_df_2017.pct_change().corr(method='pearson')

These are somewhat more significant correlation coefficients. Strong enough to use as the sole basis for an investment? Certainly not.

It is notable, however, that almost all of the cryptocurrencies have become more correlated with each other across the board.

correlation_heatmap(combined_df_2017.pct_change(), "Cryptocurrency Correlations in 2017")

Huh. That's rather interesting.

Why is this happening?

Good question. I'm really not sure.

The most immediate explanation that comes to mind is that hedge funds have recently begun publicly trading in crypto-currency markets^[1][2]. These funds have vastly more capital to play with than the average trader, so if a fund is hedging their bets across multiple cryptocurrencies, and using similar trading strategies for each based on independent variables (say, the stock market), it could make sense that this trend of increasing correlations would emerge.

In-Depth - XRP and STR

For instance, one noticeable trait of the above chart is that XRP (the token for Ripple), is the least correlated cryptocurrency. The notable exception here is with STR (the token for Stellar, officially known as "Lumens"), which has a stronger (0.62) correlation with XRP.

What is interesting here is that Stellar and Ripple are both fairly similar fintech platforms aimed at reducing the friction of international money transfers between banks.

It is conceivable that some big-money players and hedge funds might be using similar trading strategies for their investments in Stellar and Ripple, due to the similarity of the blockchain services that use each token. This could explain why XRP is so much more heavily correlated with STR than with the other cryptocurrencies.

Quick Plug - I'm a contributor to Chipper, a (very) early-stage startup using Stellar with the aim of disrupting micro-remittances in Africa.

Your Turn

This explanation is, however, largely speculative. Maybe you can do better. With the foundation we've made here, there are hundreds of different paths to take to continue searching for stories within the data.

Here are some ideas:

• Add data from more cryptocurrencies to the analysis.
• Adjust the time frame and granularity of the correlation analysis, for a more fine or coarse grained view of the trends.
• Search for trends in trading volume and/or blockchain mining data sets. The buy/sell volume ratios are likely more relevant than the raw price data if you want to predict future price fluctuations.
• Add pricing data on stocks, commodities, and fiat currencies to determine which of them correlate with cryptocurrencies (but please remember the old adage that "Correlation does not imply causation").
• Quantify the amount of "buzz" surrounding specific cryptocurrencies using Event Registry, GDELT, and Google Trends.
• Train a predictive machine learning model on the data to predict tomorrow's prices. If you're more ambitious, you could even try doing this with a recurrent neural network (RNN).
• Use your analysis to create an automated "Trading Bot" on a trading site such as Poloniex or Coinbase, using their respective trading APIs. Be careful: a poorly optimized trading bot is an easy way to lose your money quickly.
• Share your findings! The best part of Bitcoin, and of cryptocurrencies in general, is that their decentralized nature makes them more free and democratic than virtually any other asset. Open source your analysis, participate in the community, maybe write a blog post about it.

An HTML version of the Python notebook is available here.

Hopefully, now you have the skills to do your own analysis and to think critically about any speculative cryptocurrency articles you might read in the future, especially those written without any data to back up the provided predictions.

Thanks for reading, and please comment below if you have any ideas, suggestions, or criticisms regarding this tutorial. If you find problems with the code, you can also feel free to open an issue in the Github repository here.

I've got second (and potentially third) part in the works, which will likely be following through on some of the ideas listed above, so stay tuned for more in the coming weeks.

Or How I Learned to Stop Writing Callback Functions and Love Javascript ES8.

Sometimes modern Javascript projects get out of hand. A major culprit in this can be the messy handling of asynchronous tasks, leading to long, complex, and deeply nested blocks of code. Javascript now provides a new syntax for handling these operations, and it can turn even the most convoluted asynchronous operations into concise and highly readable code.

Background

AJAX (Asynchronous JavaScript And XML)

First a brief bit of history. In the late 1990s, Ajax was the first major breakthrough in asynchronous Javascript. This technique allowed websites to pull and display new data after the HTML had been loaded, a revolutionary idea at a time when most websites would download the entire page again to display a content update. The technique (popularized in name by the bundled helper function in jQuery) dominated web-development for all of the 2000s, and Ajax is the primary technique that websites use to retrieve data today, but with XML largely substituted for JSON.

NodeJS

When NodeJS was first released in 2009, a major focus of the server-side environment was allowing programs to gracefully handle concurrency. Most server-side languages at the time handled I/O operations by blocking the code completion until the operation had finished. Nodejs instead utilized an event-loop architecture, such that developers could assign "callback" functions to be triggered once non-blocking asynchronous operations had completed, in a similar manner to how the Ajax syntax worked.

Promises

A few years later, a new standard called "Promises" emerged in both NodeJS and browser environments, offering a powerful and standardized way to compose asynchronous operations. Promises still used a callback based format, but offered a consistent syntax for chaining and composing asynchronous operations. Promises, which had been pioneered by popular open-source libraries, were finally added as a native feature to Javascript in 2015.

Promises were a major improvement, but they still can often be the cause of somewhat verbose and difficult-to-read blocks of code.

Now there is a solution.

Async/await is a new syntax (borrowed from .NET and C#) that allows us to compose Promises as though they were just normal synchronous functions without callbacks. It's a fantastic addition to the Javascript language, added last year in Javascript ES7, and can be used to simplify pretty much any existing JS application.

Examples

We'll be going through a few code examples.

No libraries are required to run these examples. Async/await is fully supported in the latest versions of Chrome, Firefox, Safari, and Edge, so you can try out the examples in your browser console. Additionally, async/await syntax works in Nodejs version 7.6 and higher, and is supported by the Babel and Typescript transpilers, so it can really be used in any Javascript project today.

Setup

If you want to follow along on your machine, we'll be using this dummy API class. The class simulates network calls by returning promises which will resolve with simple data 200ms after being called.

class Api {
  constructor () {
    this.user = { id: 1, name: 'test' }
    this.friends = [ this.user, this.user, this.user ]
    this.photo = 'not a real photo'
  }
  getUser () {
    return new Promise((resolve, reject) => {
      setTimeout(() => resolve(this.user), 200)
    })
  }
  getFriends (userId) {
    return new Promise((resolve, reject) => {
      setTimeout(() => resolve(this.friends.slice()), 200)
    })
  }
  getPhoto (userId) {
    return new Promise((resolve, reject) => {
      setTimeout(() => resolve(this.photo), 200)
    })
  }
  throwError () {
    return new Promise((resolve, reject) => {
      setTimeout(() => reject(new Error('Intentional Error')), 200)
    })
  }
}

Each example will be performing the same three operations in sequence: retrieve a user, retrieve their friends, retrieve their picture. At the end, we will log all three results to the console.

Attempt 1 - Nested Promise Callback Functions

Here is an implemention using nested promise callback functions.

function callbackHell () {
  const api = new Api()
  let user, friends
  api.getUser().then(function (returnedUser) {
    user = returnedUser
    api.getFriends(user.id).then(function (returnedFriends) {
      friends = returnedFriends
      api.getPhoto(user.id).then(function (photo) {
        console.log('callbackHell', { user, friends, photo })
      })
    })
  })
}

This probably looks familiar to anyone who has worked on a Javascript project. The code block, which has a reasonably simple purpose, is long, deeply nested, and ends in this...

      })
    })
  })
}

In a real codebase, each callback function might be quite long, which can result in huge and deeply indented functions. Dealing with this type of code, working with callbacks within callbacks within callbacks, is what is commonly referred to as "callback hell".

Even worse, there's no error checking, so any of the callbacks could fail silently as an unhandled promise rejection.

Attempt 2 - Promise Chain

Let's see if we can do any better.

function promiseChain () {
  const api = new Api()
  let user, friends
  api.getUser()
    .then((returnedUser) => {
      user = returnedUser
      return api.getFriends(user.id)
    })
    .then((returnedFriends) => {
      friends = returnedFriends
      return api.getPhoto(user.id)
    })
    .then((photo) => {
      console.log('promiseChain', { user, friends, photo })
    })
}

One nice feature of promises is that they can be chained by returning another promise inside each callback. This way we can keep all of the callbacks on the same indentation level. We're also using arrow functions to abbreviate the callback function declarations.

This variant is certainly easier to read than the previous, and has a better sense of sequentiality, but is still very verbose and a bit complex looking.

Attempt 3 - Async/Await

What if it were possible to write it without any callback functions? Impossible? How about writing it in 7 lines?

async function asyncAwaitIsYourNewBestFriend () {
  const api = new Api()
  const user = await api.getUser()
  const friends = await api.getFriends(user.id)
  const photo = await api.getPhoto(user.id)
  console.log('asyncAwaitIsYourNewBestFriend', { user, friends, photo })
}

Much better. Calling "await" in front of a promise pauses the flow of the function until the promise has resolved, and assigns the result to the variable to the left of the equal sign. This way we can program an asynchronous operation flow as though it were a normal synchronous series of commands.

I hope you're as excited as I am at this point.

Note that "async" is declared at the beginning of the function declaration. This is required and actually turns the entire function into a promise. We'll dig into that later on.

Loops

Async/await makes lots of previously complex operations really easy. For example, what if we wanted to sequentially retrieve the friends lists for each of the user's friends?

Attempt 1 - Recursive Promise Loop

Here's how fetching each friend list sequentially might look with normal promises.

function promiseLoops () {  
  const api = new Api()
  api.getUser()
    .then((user) => {
      return api.getFriends(user.id)
    })
    .then((returnedFriends) => {
      const getFriendsOfFriends = (friends) => {
        if (friends.length > 0) {
          let friend = friends.pop()
          return api.getFriends(friend.id)
            .then((moreFriends) => {
              console.log('promiseLoops', moreFriends)
              return getFriendsOfFriends(friends)
            })
        }
      }
      return getFriendsOfFriends(returnedFriends)
    })
}

We're creating an inner-function that recursively chains promises for the fetching friends-of-friends until the list is empty. Ugh. It's completely functional, which is nice, but this is still an exceptionally complicated solution for a fairly straightforward task.

Note - Attempting to simplify the promiseLoops() function using Promise.all() will result in a function that behaves in significantly different manner. The intention of this example is to run the operations sequentially (one at a time), whereas Promise.all() is used for running asynchronous operations concurrently (all at once). Promise.all() is still very powerful when combined with async/await, however, as we'll see in the next section.

Attempt 2 - Async/Await For-Loop

This could be so much easier.

async function asyncAwaitLoops () {
  const api = new Api()
  const user = await api.getUser()
  const friends = await api.getFriends(user.id)
  for (let friend of friends) {
    let moreFriends = await api.getFriends(friend.id)
    console.log('asyncAwaitLoops', moreFriends)
  }
}

No need to write any recursive promise closures. Just a for-loop. Async/await is your friend.

Parallel Operations

It's a bit slow to get each additional friend list one-by-one, why not do them in parallel? Can we do that with async/await?

Yeah, of course we can. It solves all of our problems.

async function asyncAwaitLoopsParallel () {
  const api = new Api()
  const user = await api.getUser()
  const friends = await api.getFriends(user.id)
  const friendPromises = friends.map(friend => api.getFriends(friend.id))
  const moreFriends = await Promise.all(friendPromises)
  console.log('asyncAwaitLoopsParallel', moreFriends)
}

To run operations in parallel, form an array of promises to be run, and pass it as the parameter to Promise.all(). This returns a single promise for us to await, which will resolve once all of the operations have completed.

Error Handling

There is, however, one major issue in asynchronous programming that we haven't addressed yet: error handling. The bane of many codebases, asynchronous error handling often involves writing individual error handling callbacks for each operation. Percolating errors to the top of the call stack can be complicated, and normally requires explicitly checking if an error was thrown at the beginning of every callback. This approach is tedious, verbose and error-prone. Furthermore, any exception thrown in a promise will fail silently if not properly caught, leading to "invisible errors" in codebases with incomplete error checking.

Let's go back through the examples and add error handling to each. To test the error handling, we'll be calling an additional function, "api.throwError()", before retrieving the user photo.

Attempt 1 - Promise Error Callbacks

Let's look at a worst-case scenario.

function callbackErrorHell () {
  const api = new Api()
  let user, friends
  api.getUser().then(function (returnedUser) {
    user = returnedUser
    api.getFriends(user.id).then(function (returnedFriends) {
      friends = returnedFriends
      api.throwError().then(function () {
        console.log('Error was not thrown')
        api.getPhoto(user.id).then(function (photo) {
          console.log('callbackErrorHell', { user, friends, photo })
        }, function (err) {
          console.error(err)
        })
      }, function (err) {
        console.error(err)
      })
    }, function (err) {
      console.error(err)
    })
  }, function (err) {
    console.error(err)
  })
}

This is just awful. Besides being really long and ugly, the control flow is very unintuitive to follow since it flows from the outside in, instead of from top to bottom like normal, readable code. Awful. Let's move on.

Attempt 2 - Promise Chain "Catch" Method

We can improve things a bit by using a combined Promise "catch" method.

function callbackErrorPromiseChain () {
  const api = new Api()
  let user, friends
  api.getUser()
    .then((returnedUser) => {
      user = returnedUser
      return api.getFriends(user.id)
    })
    .then((returnedFriends) => {
      friends = returnedFriends
      return api.throwError()
    })
    .then(() => {
      console.log('Error was not thrown')
      return api.getPhoto(user.id)
    })
    .then((photo) => {
      console.log('callbackErrorPromiseChain', { user, friends, photo })
    })
    .catch((err) => {
      console.error(err)
    })
}

This is certainly better; by leveraging a single catch function at the end of the promise chain, we can provide a single error handler for all of the operations. However, it's still a bit complex, and we are still forced to handle the asynchronous errors using a special callback instead of handling them the same way we would normal Javascript errors.

Attempt 3 - Normal Try/Catch Block

We can do better.

async function aysncAwaitTryCatch () {
  try {
    const api = new Api()
    const user = await api.getUser()
    const friends = await api.getFriends(user.id)
    await api.throwError()
    console.log('Error was not thrown')
    const photo = await api.getPhoto(user.id)
    console.log('async/await', { user, friends, photo })
  } catch (err) {
    console.error(err)
  }
}

Here, we've wrapped the entire operation within a normal try/catch block. This way, we can throw and catch errors from synchronous code and asynchronous code in the exact same way. Much simpler.

Composition

I mentioned earlier that any function tagged with "async" actually returns a promise. This allows us to really easily compose asynchronous control flows.

For instance, we can reconfigure the earlier example to return the user data instead of logging it. Then we can retrieve the data by calling the async function as a promise.

async function getUserInfo () {
  const api = new Api()
  const user = await api.getUser()
  const friends = await api.getFriends(user.id)
  const photo = await api.getPhoto(user.id)
  return { user, friends, photo }
}
function promiseUserInfo () {
  getUserInfo().then(({ user, friends, photo }) => {
    console.log('promiseUserInfo', { user, friends, photo })
  })
}

Even better, we can use async/await syntax in the receiver function too, leading to a completely obvious, even trivial, block of asynchronous programing.

async function awaitUserInfo () {
  const { user, friends, photo } = await getUserInfo()
  console.log('awaitUserInfo', { user, friends, photo })
}

What if now we need to retrieve all of the data for the first 10 users?

async function getLotsOfUserData () {
  const users = []
  while (users.length < 10) {
    users.push(await getUserInfo())
  }
  console.log('getLotsOfUserData', users)
}

How about in parallel? And with airtight error handling?

async function getLotsOfUserDataFaster () {
  try {
    const userPromises = Array(10).fill(getUserInfo())
    const users = await Promise.all(userPromises)
    console.log('getLotsOfUserDataFaster', users)
  } catch (err) {
    console.error(err)
  }
}

Conclusion

With the rise of single-page javascript web apps and the widening adoption of NodeJS, handling concurrency gracefully is more important than ever for Javascript developers. Async/await alleviates many of the bug-inducing control-flow issues that have plagued Javascript codebases for decades and is pretty much guaranteed to make any async code block significantly shorter, simpler, and more self-evident. With near-universal support in mainstream browsers and NodeJS, this is the perfect time to integrate these techniques into your own coding practices and projects.

Join The Discussion on Reddit

Async/Await Will Make Your Code Simpler from javascript

Async/Await Will Make Your Code Simpler from webdev

Toggle Code Editor

I originally wrote this post for the SocialCops engineering blog,
check it out here - Would You Survive the Titanic? A Guide to Machine Learning in Python

What if machines could learn?

This has been one of the most intriguing questions in science fiction and philosophy since the advent of machines. With modern technology, such questions are no longer bound to creative conjecture. Machine learning is all around us. From deciding which movie you might want to watch next on Netflix to predicting stock market trends, machine learning has a profound impact on how data is understood in the modern era.

This tutorial aims to give you an accessible introduction on how to use machine learning techniques for your projects and data sets. In roughly 20 minutes, you will learn how to use Python to apply different machine learning techniques — from decision trees to deep neural networks — to a sample data set. This is a practical, not a conceptual, introduction; to fully understand the capabilities of machine learning, I highly recommend that you seek out resources that explain the low-level implementations and theory of these techniques.

Our sample dataset: passengers of the RMS Titanic. We will use an open data set with data on the passengers aboard the infamous doomed sea voyage of 1912. By examining factors such as class, sex, and age, we will experiment with different machine learning algorithms and build a program that can predict whether a given passenger would have survived this disaster.

Setting Up Your Machine Learning Laboratory

The best way to learn about machine learning is to follow along with this tutorial on your computer. To do this, you will need to install a few software packages if you do not have them yet:

• Python (version 3.4.2 was used for this tutorial): https://www.python.org
• SciPy Ecosystem (NumPy, SciPy, Pandas, IPython, matplotlib): https://www.scipy.org
• SciKit-Learn: https://scikit-learn.org/stable/
• TensorFlow: https://www.tensorflow.org

There are multiple ways to install each of these packages. I recommend using the “pip” Python package manager, which will allow you to simply run “pip3 install ” to install each of the dependencies: https://pip.pypa.io/en/stable/.

For actually writing and running the code, I recommend using IPython (which will allow you to run modular blocks of code and immediately view the output values and data visualizations) along with the Jupyter Notebook as a graphical interface: https://jupyter.org.

You will also need the Titanic dataset that we will be analyzing. You can find it here: https://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.xls.

With all of the dependencies installed, simply run “jupyter notebook” on the command line, from the same directory as the titanic3.xls file, and you will be ready to get started.

The Data at First Glance: Who Survived the Titanic and Why?

First, import the required Python dependencies.

Note: This tutorial was written using TensorFlow version 0.8.0. Newer versions of TensorFlow use different import statements and names. If you’re on a newer version of TensorFlow, check out this Github issue or the latest TensorFlow Learn documentation for the newest import statements.

Once we have read the spreadsheet file into a Pandas dataframe (imagine a hyperpowered Excel table), we can peek at the first five rows of data using the head() command.

The column heading variables have the following meanings:

• survival: Survival (0 = no; 1 = yes)
• class: Passenger class (1 = first; 2 = second; 3 = third)
• name: Name
• sex: Sex
• age: Age
• sibsp: Number of siblings/spouses aboard
• parch: Number of parents/children aboard
• ticket: Ticket number
• fare: Passenger fare
• cabin: Cabin
• embarked: Port of embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)
• boat: Lifeboat (if survived)
• body: Body number (if did not survive and body was recovered)

Now that we have the data in a dataframe, we can begin an advanced analysis of the data using powerful single-line Pandas functions. First, let’s examine the overall chance of survival for a Titanic passenger.

The calculation shows that only 38% of the passengers survived. Not the best odds. The reason for this massive loss of life is that the Titanic was only carrying 20 lifeboats, which was not nearly enough for the 1,317 passengers and 885 crew members aboard. It seems unlikely that all of the passengers would have had equal chances at survival, so we will continue breaking down the data to examine the social dynamics that determined who got a place on a lifeboat and who did not.

Social classes were heavily stratified in the early twentieth century. This was especially true on the Titanic, where the luxurious first-class areas were completely off limits to the middle-class passengers in second class, and especially to those who carried a third class “economy price” ticket. To get a view into the composition of each class, we can group data by class, and view the averages for each column:

We can start drawing some interesting insights from this data. For instance, passengers in first class had a 62% chance of survival, compared to a 25.5% chance for those in 3rd class. Additionally, the lower classes generally consisted of younger people, and the ticket prices for first class were predictably much higher than those for second and third class. The average ticket price for first class (£87.5) is equivalent to $13,487 in 2016.

We can extend our statistical breakdown using the grouping function for both class and sex:

While the Titanic was sinking, the officers famously prioritized who was allowed in a lifeboat with the strict maritime tradition of evacuating women and children first. Our statistical results clearly reflect the first part of this policy as, across all classes, women were much more likely to survive than the men. We can also see that the women were younger than the men on average, were more likely to be traveling with family, and paid slightly more for their tickets.

The effectiveness of the second part of this “Women and children first” policy can be deduced by breaking down the survival rate by age.

Here we can see that children were indeed the most likely age group to survive, although this percentage was still tragically below 60%.

Why Machine Learning?

With analysis, we can draw some fairly straightforward conclusions from this data — being a woman, being in 1st class, and being a child were all factors that could boost your chances of survival during this disaster.

Let’s say we wanted to write a program to predict whether a given passenger would survive the disaster. This could be done through an elaborate system of nested if-else statements with some sort of weighted scoring system, but such a program would be long, tedious to write, difficult to generalize, and would require extensive fine tuning.

This is where machine learning comes in: we will build a program that learns from the sample data to predict whether a given passenger would survive.

Preparing The Data

Before we can feed our data set into a machine learning algorithm, we have to remove missing values and split it into training and test sets.

If we perform a count of each column, we will see that much of the data on certain fields is missing. Most machine learning algorithms will have a difficult time handling missing values, so we will need to make sure that each row has a value for each column.

Most of the rows are missing values for “boat” and “cabin”, so we will remove these columns from the data frame. A large number of rows are also missing the “home.dest” field; here we fill the missing values with “NA”. A significant number of rows are also missing an age value. We have seen above that age could have a significant effect on survival chances, so we will have to drop all of rows that are missing an age value. When we run the count command again, we can see that all remaining columns now contain the same number of values.

Now we need to format the remaining data in a way that our machine learning algorithms will accept.

The “sex” and “embarked” fields are both string values that correspond to categories (i.e “Male” and “Female”) so we will run each through a preprocessor. This preprocessor will convert these strings into integer keys, making it easier for the classification algorithms to find patterns. For instance, “Female” and “Male” will be converted to 0 and 1 respectively. The “name”, “ticket”, and “home.dest” columns consist of non-categorical string values. These are difficult to use in a classification algorithm, so we will drop them from the data set.

Next, we separate the data set into two arrays: “X” containing all of the values for each row besides “survived”, and “y” containing only the “survived” value for that row. The classification algorithms will compare the attribute values of “X” to the corresponding values of “y” to detect patterns in how different attributes values tend to affect the survival of a passenger.

Finally, we break the “X” and “y” array into two parts each — a training set and a testing set. We will feed the training set into the classification algorithm to form a trained model. Once the model is formed, we will use it to classify the testing set, allowing us to determine the accuracy of the model. Here we have made a 20/80 split, such that 80% of the dataset will be used for training and 20% will be used for testing.

Classification – The Fun Part

We will start off with a simple decision tree classifier. A decision tree examines one variable at a time and splits into one of two branches based on the result of that value, at which point it does the same for the next variable. A fantastic visual explanation of how decision trees work can be found here: https://www.r2d3.us/visual-intro-to-machine-learning-part-1/.

This is what a trained decision tree for the Titanic dataset looks like if we set the maximum number of levels to 3:

The tree first splits by sex, and then by class, since it has learned during the training phase that these are the two most important features for determining survival. The dark blue boxes indicate passengers who are likely to survive, and the dark orange boxes represent passengers who are almost certainly doomed. Interestingly, after splitting by class, the main deciding factor determining the survival of women is the ticket fare that they paid, while the deciding factor for men is their age (with children being much more likely to survive).

To create this tree, we first initialize an instance of an untrained decision tree classifier. (Here we will set the maximum depth of the tree to 10). Next we “fit” this classifier to our training set, enabling it to learn about how different factors affect the survivability of a passenger. Now that the decision tree is ready, we can “score” it using our test data to determine how accurate it is.

The resulting reading, 0.7703, means that the model correctly predicted the survival of 77% of the test set. Not bad for our first model!

If you are being an attentive, skeptical reader (as you should be), you might be thinking that the accuracy of the model could vary depending on which rows were selected for the training and test sets. We will get around this problem by using a shuffle validator.

This shuffle validator applies the same random 20:80 split as before, but this time it generates 20 unique permutations of this split. By passing this shuffle validator as a parameter to the “cross_val_score” function, we can score our classifier against each of the different splits, and compute the average accuracy and standard deviation from the results.

The result shows that our decision tree classifier has an overall accuracy of 77.34%, although it can go up to 80% and down to 75% depending on the training/test split. Using scikit-learn, we can easily test other machine learning algorithms using the exact same syntax.

The “Random Forest” classification algorithm will create a multitude of (generally very poor) trees for the data set using different random subsets of the input variables, and will return whichever prediction was returned by the most trees. This helps to avoid “overfitting”, a problem that occurs when a model is so tightly fitted to arbitrary correlations in the training data that it performs poorly on test data.

The “Gradient Boosting” classifier will generate many weak, shallow prediction trees and will combine, or “boost”, them into a strong model. This model performs very well on our data set, but has the drawback of being relatively slow and difficult to optimize, as the model construction happens sequentially so it cannot be parallelized.

A “Voting” classifier can be used to apply multiple conceptually divergent classification models to the same data set and will return the majority vote from all of the classifiers. For instance, if the gradient boosting classifier predicts that a passenger will not survive, but the decision tree and random forest classifiers predict that they will live, the voting classifier will chose the latter.

This has been a very brief and non-technical overview of each technique, so I encourage you to learn more about the mathematical implementations of all of these algorithms to obtain a deeper understanding of their relative strengths and weaknesses. Many more classification algorithms are available “out-of-the-box” in scikit-learn and can be explored here: https://scikit-learn.org/stable/modules/ensemble.

Computational Brains — An Introduction to Deep Neural Networks

Neural networks are a rapidly developing paradigm for information processing based loosely on how neurons in the brain processes information. A neural network consists of multiple layers of nodes, where each node performs a unit of computation and passes the result onto the next node. Multiple nodes can pass inputs to a single node and vice versa.

The neural network also contains a set of weights, which can be refined over time as the network learns from sample data. The weights are used to describe and refine the connection strengths between nodes. For instance, in our Titanic data set, node connections transmitting the passenger sex and class will likely be weighted very heavily, since these are important for determining the survival of a passenger.

A Deep Neural Network (DNN) is a neural network that works not just by passing data between nodes, but by passing data between layers of nodes. Each layer of nodes is able to aggregate and recombine the outputs from the previous layer, allowing the network to gradually piece together and make sense of unstructured data (such as an image). Such networks can also be heavily optimized due to their modular nature, allowing the operations of each node layer to be parallelized en masse across multiple CPUs and even GPUs.

We have barely begun to skim the surface of explaining neural networks. For a more in depth explanation of the inner workings of DNNs, this is a good resource: https://deeplearning4j.org/neuralnet-overview.html.

This awesome tool allows you to visualize and modify an active deep neural network: https://playground.tensorflow.org.

The major advantage of neural networks over traditional machine learning techniques is their ability to find patterns in unstructured data (such as images or natural language). Training a deep neural network on the Titanic data set is total overkill, but it’s a cool technology to work with, so we’re going to do it anyway.

An emerging powerhouse in programing neural networks is an open source library from Google called TensorFlow. This library is the foundation for many of the most recent advances in machine learning, such as being used to train computer programs to create unique works of music and visual art (https://magenta.tensorflow.org/welcome-to-magenta). The syntax for using TensorFlow is somewhat abstract, but there is a wrappercalled “skflow” in the TensorFlow package that allows us to build deep neural networks using the now-familiar scikit-learn syntax.

Above, we have written the code to build a deep neural network classifier. The “hidden units” of the classifier represent the neural layers we described earlier, with the corresponding numbers representing the size of each layer.

We can also define our own training model to pass to the TensorFlow estimator function (as seen above). Our defined model is very basic. For more advanced examples of how to work within this syntax, see the skflow documentation here: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/skflow.

Despite the increased power and lengthier runtime of these neural network models, you will notice that the accuracy is still about the same as what we achieved using more traditional tree-based methods. The main advantage of neural networks — unsupervised learning of unstructured data — doesn’t necessarily lend itself well to our Titanic dataset, so this is not too surprising.

I still, however, think that running the passenger data of a 104-year-old shipwreck through a cutting-edge deep neural network is pretty cool.

These Are Not Just Data Points. They’re People.

Given that the accuracy for all of our models is maxing out around 80%, it will be interesting to look at specific passengers for whom these classification algorithms are incorrect.

The above code forms a test data set of the first 20 listed passengers for each class, and trains a deep neural network against the remaining data.

Once the model is trained we can use it to predict the survival of passengers in the test data set, and compare these to the known survival of each passenger using the original dataset.

The above table shows all of the passengers in our test data set whose survival (or lack thereof) was incorrectly classified by the neural network model.

Sometimes when you are dealing the data sets like this, the human side of the story can get lost beneath the complicated math and statistical analysis. By examining passengers for whom our classification model was incorrect, we can begin to uncover some of the most fascinating, and sometimes tragic, stories of humans defying the odds.

For instance, the first three incorrectly classified passengers are all members of the Allison family, who perished even though the model predicted that they would survive. These first class passengers were very wealthy, as can be evidenced by their far-above-average ticket prices. For Betsy (25) and Loraine (2) in particular, not surviving is very surprising, considering that we found earlier that over 96% of first class women lived through the disaster.

From left to right: Hudson (30), Bess (25), Trevor (11 months), and Loraine Allison (2).

So what happened? A surprising amount of information on each Titanic passenger is available online; it turns out that the Allison family was unable to find their youngest son Trevor and was unwilling to evacuate the ship without him. Tragically, Trevor was already safe in a lifeboat with his nurse and was the only member of the Allison family to survive the sinking.

Another interesting misclassification is John Jacob Astor, who perished in the disaster even though the model predicted he would survive. Astor was the wealthiest person on the Titanic, an impressive feat on a ship full of multimillionaire industrialists, railroad tycoons, and aristocrats. Given his immense wealth and influence, which the model may have deduced from his ticket fare (valued at over $35,000 in 2016), it seems likely that he would have been among of the 35% of men in first class to survive. However, this was not the case: although his pregnant wife survived, John Jacob Astor’s body was recovered a week later, along with a gold watch, a diamond ring with three stones, and no less than $92,481 (2016 value) in cash.

John Jacob Astor IV

On the other end of the spectrum is Olaus Jorgensen Abelseth, a 25-year-old Norwegian sailor. Abelseth, as a man in 3rd class, was not expected to survive by our classifier. Once the ship sank, however, he was able to stay alive by swimming for 20 minutes in the frigid North Atlantic water before joining other survivors on a waterlogged collapsible boat and rowing through the night. Abelseth got married three years later, settled down as a farmer in North Dakota, had 4 kids, and died in 1980 at the age of 94.

Olaus Jorgensen Abelseth

Initially I was disappointed by the accuracy of our machine learning models maxing out at about 80% for this data set. It’s easy to forget that these data points each represent real people, each of whom found themselves stuck on a sinking ship without enough lifeboats. When we looked into data points for which our model was wrong, we can uncover incredible stories of human nature driving people to defy their logical fate. It is important to never lose sight of the human element when analyzing this type of data set. This principle will be especially important going forward, as machine learning is increasingly applied to human data sets by organizations such as insurance companies, big banks, and law enforcement agencies.

What next?

So there you have it — a primer for data analysis and machine learning in Python. From here, you can fine-tune the machine learning algorithms to achieve better accuracy on this data set, design your own neural networks using TensorFlow, discover more fascinating stories of passengers whose survival does not match the model, and apply all of these techniques to any other data set. (Check out this Game of Thrones dataset: https://www.kaggle.com/mylesoneill/game-of-thrones). When it comes to machine learning, the possibilities are endless and the opportunities are titanic.

This blog post is a synthesis of two articles I originally wrote for the SocialCops engineering blog. Check them out here - Tracking Air Pollution in Delhi, One Auto Rickshaw at a Time, How We Built Our IoT Devices to Track Air Pollution in Delhi
*

The Most Polluted City On Earth

Delhi is one of the largest and fastest growing cities on the planet. Home to over 24 million people, one of the unifying challenges for this huge and diverse population is also one of the most basic necessities for life: clean air. Delhi was recently granted the dubious title of the “World’s Most Polluted City” by the World Health Organization.

Air pollution has major health consequences for those living in heavily polluted areas. High levels of particulates and hazardous gases can increase the risk of heart disease, asthma, bronchitis, cancer, and more. The WHO estimates that there are 7 million premature deaths linked to air pollution every year, and the Union Environment Ministry estimates that 80 people die every day from air pollution in Delhi.

How it Started

The first week I joined SocialCops, I was given a “hack week” project to present to the rest of the company. My project was simple and open-ended: “build something cool”. As a newcomer to Delhi, I was concerned by the infamous air pollution continually hovering over the city. I decided to build two IoT air pollution sensing devices (one for our office balcony and one for inside our office) to determine how protected we were from the pollution while inside. Over the following weeks, this simple internal hack turned into a much more ambitious project — monitoring air pollution throughout Delhi by attaching these IoT devices to auto rickshaws.

Air Pollution Sensors + Auto Rickshaws

Traditional particulate matter measurement devices use very advanced scales and filters to measure the exact mass of ambient particles below a certain size. As such, these devices are prohibitively expensive (₹1.1 crore or $165,000) and fixed in a single location.

We took a different approach.

Autorickshaws at the Saket Metro Station

Auto rickshaws are a very popular source of transportation in Delhi. Popularly called “autos”, these vehicles can be found all over Delhi at all times of day and night, making them an ideal place to deploy our sensors. Unlike traditional air quality readings that sample from one location repeatedly, the sensors deployed for this project sample data for air pollution in Delhi directly from traffic jams, markets, and residential neighborhoods all over the city.

The Internet of (Air Pollution Monitoring) Things

We have developed a custom internet-connected device to attach to take pollution readings from autos. Each device contains an airborne particle sensor, a GPS unit, and a cellular antenna to send the data over 2g networks in real time; we were able to construct each device for about ₹6,500 ($100) each. The greater mobility and reduced cost of these devices comes at a cost: the particle sensor we are using is less accurate than those used by traditional pollution monitors. The reason is that our sensor determines the number of airborne particles by reading the ambient air opaqueness, instead of measuring the precise mass of the collected particles.

Our solution for this drop in precision is to increase the sample size. A lot.

Each device takes two readings per minute. With five devices deployed, the pollution reading for each hour is an average of 600 data points, and the AQI for each day is calculated from almost 15,000 distinct readings. The up-time for each device is not 100%, as the auto-rickshaw drivers generally drive for 12 hours per day, and we are not always able to transfer the device to another auto driver between shifts. However, the resulting data has still proven sufficient for our experimental purposes.

SocialCops Delhi Air Pollution Dashboard

Hard Decisions Over Hardware

We chose the hardware for this pilot project based on ease of construction, cost, and hackability. There were a few decisions to confront first: what data to collect, how to collect it, and how to retrieve it for further analysis. We decided to focus on collecting data for particulate matter (PM) concentration since this is the primary pollutant in Delhi. There are other factors that influence air pollution, such as humidity, but we can pull most of this data from established external sources and integrate it with our primary data.

To track devices and analyze data in real time, we decided to send data via 2G networks using a GPRS (General Packet Radio Service) antenna and SIM card. We decided to go with the LinkIt One development board for the initial deployment, due to its compatibility with the Arduino IDE and firmware libraries and its included GPS and GPRS antennas. (We originally intended to also use its included battery but decided not to, as explained later in this post.)

For the pollution-sensing module, we decided to use the Shinyei PPD42NS because of its low cost and general dependability and durability. This sensor measures the ambient air opacity using an LED, lens, and photodiode. The attached microcontroller can read this opacity and calculate the number of particles per .01 cubic feet by reading the LPO (Low Pulse Occupancy) outputted by the sensor.

The prototype hardware, mid-assembly

This sensor module is relatively imprecise, as are all dust sensors in this same price range. That’s why one of our main goals with this experiment is to determine whether aggregating a large enough number of readings from this type of sensor can provide pollution data that is as (or more) accurate than the data from traditional air pollution stations. The stations cost ₹1.1 crore ($165,000) each. The final cost of our hardware package, including the case and external power bank, was about ₹6,500 ($100).

Firming up the Firmware

The firmware for this experiment was written in C/C++ using the Arduino IDE, which the LinkIt One is designed to be directly compatible with. The basic flow of the firmware is simple: sample data from the pollution sensor for 30 seconds, calculate the particles per .01 cubic foot from these values, retrieve the GPS location, upload the data to the server, and repeat.

One scenario we needed to prepare for was the IoT device being temporarily unable to connect to the cellular data network. To account for this, we implemented a caching system — if the device can’t connect to the server, it logs current readings to its local 10 MB of flash memory. Every 30 seconds, the device takes a reading and attempts to upload that data to the server. If it is able to connect, then it also scans the local filesystem for any cached data; if there is locally stored data, it is uploaded and deleted from the local storage. This feature allows the device to operate virtually anywhere, regardless of connectivity, and also provides interesting insights into cellular dark zones in Delhi.

With the firmware functionality complete, one of the immediate problems was that the device would occasionally crash while uploading readings. Unlike debugging a more traditional software program, there was no stack-trace to look through to find the cause. The only clue we had was that the light on the microcontroller would turn from green to red, and the device would stop uploading data. Like most microcontrollers, the LinkIt One has very limited SRAM, leading us to suspect that the crash may be due an out-of-memory error. The first optimization we made was using the F() macro to cache constant strings in flash memory during compilation instead of storing them in virtual memory at runtime. This optimization simply required replacing, for example, Serial.println(“string”) with Serial.println(F(“string”)).

This optimization was a good practice, but it still did not solve the device crashes. To further optimize the memory usage, we focused on the largest variable being stored in virtual memory during runtime: the JSON string storing the data to be uploaded to the server. By allocating memory more carefully for this string and ensuring that only one instance of this string existed in SRAM at any time, we were able to close any remaining memory leaks and solve the problem of unpredictable device crashes.

Backend and Security

To store the data being uploaded by each IoT device, we built a backend and API using Node.js, Express, and MongoDB. Beyond standard server-side security protocols (cryptographically secure login, restrictive firewalls, rate limiting, etc.), it was also important to build endpoint-based authentication and field verification for incoming device data.

Autorickshaws at the Saket Metro station.

A unique cryptographically secure key value is hard coded into the firmware for each IoT device. When a reading is uploaded to the server, this key value is included along with the device ID and authenticated against a corresponding key value on the server for that device. This technique of hard coding a password directly into the firmware provides a high degree of security that ensures our database is not corrupted with unauthorized data.

Power Problems

One of our major obstacles was figuring out how to provide enough power for the device to ensure that it would transmit data 24/7. Since the device uses GPS and GPRS antennas and the PPD42NS dust sensor requires constant back-to-back readings to compute an accurate value, the device is power hungry for a microcontroller-based project.

The LinkIt One comes with a rechargeable 1000 mAh battery. In our testing, this battery was only able to power the device for about three hours of operation. Even worse, the battery often had difficulty re-charging or holding a charge once it had been fully discharged, leading us to believe that the battery was becoming over-discharged and damaged by being run for too long. Further compounding this problem, the LinkIt One does not have the capability to programmatically trigger a shutdown, making it impossible to prevent this damage from occurring.

Having discarded the included battery as an option, we began testing the IoT device using low-cost mobile power banks designed for smartphones. These power banks generally come in capacities between 3000 mAh and 15,000 mAh (1-5x an average smartphone battery) and can be purchased for under $20 each. One battery we tested included a solar panel for recharging, but unfortunately, the solar panel wasn’t able to recharge the battery quickly enough. We ended up settling on a reputable 10,000 mAh rechargeable battery, which can run the device for 33 hours straight.

Brawn Over Beauty

Delhi is not a forgiving environment for hardware devices, especially when they are mounted in auto rickshaws in the height of summer when the temperatures regularly top 40℃ (104℉) or 45℃ (113℉) during heat waves. During our initial prototyping phase, the devices were encased within cardboard boxes, which made it very easy to quickly adjust the component placement and encasing structure (wire hole locations, etc).

One of the devices after being deployed for a week.

One order of 100 x 100 x 50 mm enclosure boxes and a power drill later, we assembled our new pilot encasings. Protected by white hard plastic with the dust sensor component mounted externally for unrestricted access to ambient air, the device is not pretty. It is, however, durable enough to survive extremely demanding operating conditions, and looks rough enough to reduce the risk of theft that comes associated with shiny new electronics.

From the Lab to the Streets

Beyond the core technology, one of the most difficult aspects of this pilot project was the logistical challenge of actually deploying these devices on auto rickshaws. Our priority for the deployment was twofold — to keep the device hardware safe and to have the IoT devices transmitting data 24/7 for the duration of the experiment.

Recruiting drivers was easy. We simply went down to the nearby metro station and asked the auto drivers if they would be interested in participating in the pilot project. By paying a modest daily and weekly stipend, we are giving the drivers a supplemental source of income and financial incentive to keep the device safe. Since the battery only lasts for 33 hours, the drivers have to return to our office every day to collect their pay and to swap out their battery for a fully charged one.

The SocialCops team preparing the first device for deployment.

To keep track of the devices and the incoming data, we built an administrative dashboard. This dashboard enables us to view incoming data, as well as a live map of device locations. Since we have drivers’ phone numbers, we can contact drivers if we see a device malfunctioning on the dashboard.

Moving Forward Against Air Pollution in Delhi

This experiment could be deployed at a greater scale in the future, with 100 sensors deployed across a city, for example. The base cost of the required hardware would still be considerably less than acquiring the traditional equipment used to measure air pollution. With 100 sensors deployed, the average pollution level for each day could be calculated from over 250,000 individual readings, and the sensors could be deployed to a wider area of the city.

For the next iteration of these pollution sensors, we already have a few concrete goals in mind. Our first priority is creating a sustainable and scalable system for powering and managing the IoT devices while they are deployed. Replacing five batteries per day is manageable for this deployment, but we want to scale up to having 100+ devices on the streets at the same time. One option is powering the devices from the autos themselves, the same way you could charge a cellphone in a car. The complication here is that auto rickshaws are generally turned off while the drivers are parked and waiting for customers. We would still need to include a power bank that (unlike the current power solution) supports pass-through charging to power the device and charge the battery simultaneously.

Another goal is to perform further research on alternative carriers for the devices like buses and cabs. Minimizing the recurring costs (the drivers’ stipends) will be essential in forming a sustainable system for deploying these devices. Finally, it would be best to move away from DIY-focused prototyping hardware such as the LinkIt One, and to form partnerships with hardware suppliers to produce and encase each device scalably with more standardization and lower variable costs.

New technological capabilities enabled by the Internet of Things have the potential to transform how we assess and alleviate some of the world’s most pressing problems. By further developing and refining these low-cost air pollution sensors, we hope to establish a sustainable model for collecting relevant and detailed public health data in hard-to-reach areas.

I originally wrote this post for the SocialCops engineering blog,
check it out here - The Internet of Coffee: Building a Wifi Coffee Maker

Here at SocialCops, we’ve begun using Internet-of-Things technology to research solutions for some of the world’s most pressing problems: monitoring pollution, conserving energy, protecting against identity fraud, and…making coffee.

“[If we could] turn on the coffee machine via slack, we’d be set,” Christine wistfully wrote one afternoon on our #coffee Slack channel. Being located in Delhi, good chai is readily available to us day and night, but sometimes a good cup of coffee in the morning (and afternoon, and evening) is required to keep us going. Starting a fresh brew during our commutes and having a full pot waiting at the office once we arrive sounds like a fantastical dream, but here at SocialCops we thrive on turning these types of dreams into reality.

Setting Up Our WiFi Coffee Machine

The basic hardware setup was simple: we got a wifi-enabled Particle Photon micro-controller and hooked it up to a 5v electrical relay to control the power supply to the coffee machine.

Warning: this project deals with dangerous high-voltage electrical currents, so NEVER try to follow these steps if any of the components are plugged in, or if you are unsure of what you are doing.

First we cut an extension cord in half, connected the “hot wire” from each half to the relay, and spliced the ground and neutral wires from the two sides back together.

Relay Wiring.

Then we hooked our microcontroller up to the relay, using the D1 pin on the Photon to communicate with the IN1 port on the relay, sending 5v power from the VIN pin of the Photon to the VCC port on the relay, and connecting the GND on the relay to GND on the Photon in order to complete the circuit.

Relay Wiring.

Finally we packaged the Photon and relay together into a small plastic enclosure, and cut holes for power cords into the sides of the container. By plugging one end of the extension cord into the wall, and the other side into the coffee machine, we were now able to control power to the coffee machine by sending HTTP requests over WiFi to the Particle Photon microcontroller!

There were a few extra wires in this setup, resulting in too much clutter on our kitchen counter, so we packaged the whole setup into a larger cardboard box. Taking advantage of the extra area on top of the box, we gave CoffeeBot a face, complete with LEDs for eyes to monitor the hardware status.

Enter CoffeeBot

Our team uses Slack for online communication; one of Slack’s many features is the ability to create Slackbots, or automated users that can respond to chat messages and carry out advanced tasks.

Using BotKit, a NodeJS module for programming Slackbot behaviors, we built CoffeeBot. CoffeeBot is a capable of receiving naturally worded messages on Slack, deciphering them, and sending the corresponding commands to the coffee machine.

For instance:

Other features of CoffeeBot include:

• Getting the status of the machine and how long ago the last brew was started
• Posting to Slack when the coffee’s ready
• Automatically switching off the coffee machine after an hour, in order to save energy.
• And a few other secret behaviors

Caffeinated Continuance

Now every morning before leaving for work, we can start the coffee machine directly from Slack! In the future we want to add more advanced features, such as automatically loading coffee grounds and water, and measuring how many cups of coffee the office consumes per week. Someday we might even have a robot to deliver the fresh coffee directly to our desks, which admittedly sounds like a crazy and fantastical dream right now.

But you know how we feel about fantastical dreams.