Tables - Data Science Primer

The datascience Table object is designed to analyze multiple data points with multiple charecteristics. One way to visualize this is to consider columns of data which contain values for each data point (row) and a typical table will have multiple columns reflecting the multiple charecteristics of each data point. Each column is also an array of values with each element representing the value for a given data point or row. The datascience Table has many methods which are summarized in the detailed online reference. Datascience tables are closely related to the pandas dataframe which serves as a key component of a Python data science toolbox.

Import requisite modules¶

datascience, plotting, and plot fixes

from datascience import *

# import for plotting
%matplotlib inline
import matplotlib.pyplot as plots
plots.style.use('fivethirtyeight')
# Fix for datascience plots
import collections as collections
import collections.abc as abc
collections.Iterable = abc.Iterable

Create a first datascience table object from an array¶

As an example we can create a table directly from data inserted into an array using the with_columns table method.

T=Table().with_columns('Tornados',make_array(0,0,0,1,0,0,0,1,5,1,0,0))

We can create a nice month index by using the numpy arange method combined with the with_columns table method.

import numpy as np
T=T.with_columns('Month',np.arange(1,13,1))

We can sort using the sort method and using either the column label, 'Tornadoes' in the below case, or the column index which is the first which is inxed as the value 0

T.sort(0,descending=True)

We can also use a trick with the modulo operator, %, and then further apply the where method which filters the table to get rows that match the criteria which in this case is that the newly created column, 'Odd' has the value 0.

T=T.with_columns("Odd",T.column(1)%2).where("Odd",0)
T

Census data example¶

We can look at complex real world data. In this example we look at US Census using the datascience table read_table method.

data = 'http://www2.census.gov/programs-surveys/popest/datasets/2010-2020/national/asrh/nc-est2020-agesex-res.csv'
# A local copy can be accessed here in case census.gov moves the file:
# data = path_data + 'nc-est2015-agesex-res.csv'

full_census_table = Table.read_table(data)
full_census_table

partial_census_table = full_census_table.select('SEX', 'AGE', 'POPESTIMATE2010', 'POPESTIMATE2020')
partial_census_table

We can extract a column as an array using the column() method and even multiply by 3 in this case to get a new array

3*partial_census_table.column("AGE")

array([   0,    3,    6,    9,   12,   15,   18,   21,   24,   27,   30,
         33,   36,   39,   42,   45,   48,   51,   54,   57,   60,   63,
         66,   69,   72,   75,   78,   81,   84,   87,   90,   93,   96,
         99,  102,  105,  108,  111,  114,  117,  120,  123,  126,  129,
        132,  135,  138,  141,  144,  147,  150,  153,  156,  159,  162,
        165,  168,  171,  174,  177,  180,  183,  186,  189,  192,  195,
        198,  201,  204,  207,  210,  213,  216,  219,  222,  225,  228,
        231,  234,  237,  240,  243,  246,  249,  252,  255,  258,  261,
        264,  267,  270,  273,  276,  279,  282,  285,  288,  291,  294,
        297,  300, 2997,    0,    3,    6,    9,   12,   15,   18,   21,
         24,   27,   30,   33,   36,   39,   42,   45,   48,   51,   54,
         57,   60,   63,   66,   69,   72,   75,   78,   81,   84,   87,
         90,   93,   96,   99,  102,  105,  108,  111,  114,  117,  120,
        123,  126,  129,  132,  135,  138,  141,  144,  147,  150,  153,
        156,  159,  162,  165,  168,  171,  174,  177,  180,  183,  186,
        189,  192,  195,  198,  201,  204,  207,  210,  213,  216,  219,
        222,  225,  228,  231,  234,  237,  240,  243,  246,  249,  252,
        255,  258,  261,  264,  267,  270,  273,  276,  279,  282,  285,
        288,  291,  294,  297,  300, 2997,    0,    3,    6,    9,   12,
         15,   18,   21,   24,   27,   30,   33,   36,   39,   42,   45,
         48,   51,   54,   57,   60,   63,   66,   69,   72,   75,   78,
         81,   84,   87,   90,   93,   96,   99,  102,  105,  108,  111,
        114,  117,  120,  123,  126,  129,  132,  135,  138,  141,  144,
        147,  150,  153,  156,  159,  162,  165,  168,  171,  174,  177,
        180,  183,  186,  189,  192,  195,  198,  201,  204,  207,  210,
        213,  216,  219,  222,  225,  228,  231,  234,  237,  240,  243,
        246,  249,  252,  255,  258,  261,  264,  267,  270,  273,  276,
        279,  282,  285,  288,  291,  294,  297,  300, 2997])

us_pop = partial_census_table.relabeled('POPESTIMATE2010', '2010').relabeled('POPESTIMATE2020', '2020').relabeled('SEX','GENDER')
us_pop

type(us_pop)

datascience.tables.Table

us_pop.where('AGE',80)

us_pop.where('AGE',70)

Plot the whole population with including both genders (coded as GENDER = 0) and compare 2010 and 2020 US population as a function of age.

us_pop.where('GENDER',0).select('AGE','2010','2020').where('AGE',are.below(100)).plot('AGE')