Tasks for part 2:
- Exploratory Data Analysis
- Construction of Geolocation graphics, showing the exact location of each occurange
- Segmenting the data in yearly periods
- Calculating the distance between events based on Geolocation (use the geopy package)
- Statistical summary of the variables
- Analysis and interpretation of the variable's distributions
- Normality test to verify whether the samples were extracted from a normally distributed population
- Analysis and interpretation of the p-value
- Conclusions from the exploratory analysis
In [1]:
# Disabling the multiple messages generated by the new versions from Pandas and Matplotlib
import sys
import warnings
import matplotlib.cbook
if not sys.warnoptions:
warnings.simplefilter('ignore')
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings('ignore', category=FutureWarning)
warnings.filterwarnings('ignore', category=matplotlib.cbook.mplDeprecation)
# Data manipulation and visualization
import scipy
import numpy as np
import pandas as pd
import matplotlib as m
import matplotlib.pyplot as plt
from scipy import stats
# Graphics formatting
m.rcParams['axes.labelsize'] = 14
m.rcParams['xtick.labelsize'] = 12
m.rcParams['ytick.labelsize'] = 12
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 15,7
plt.style.use('fivethirtyeight')
%matplotlib inline
Exploratory Analysis¶
In [2]:
df_arrests = pd.read_csv('files/df_arrests.csv')
In [3]:
df_arrests.shape
Out[3]:
In [4]:
df_arrests.head()
Out[4]:
In [5]:
df_arrests.describe()
Out[5]:
Checking the range of arrest dates:
In [6]:
# Minimum value
df_arrests['Arrest Date'].min()
Out[6]:
In [7]:
# Maxium value
df_arrests['Arrest Date'].max()
Out[7]:
Considering that the law came into force in 2017, and that therefore there are three years for which there is complete data of the after effects, I'll be restricting the analysis to 3 years before and 3 years after. In other words, the follows years will be considered: 2014, 2015, 2016, 2017, 2018, 2019.
In [8]:
# Filtering the dataframe to the 2014-2019 range
df_arrests = df_arrests[(df_arrests['Arrest Date'] >= '2014-01-01') & (df_arrests['Arrest Date'] <= '2019-12-31')]
In [9]:
df_arrests.shape
Out[9]:
In [10]:
# Saving the updated dataset
df_arrests.to_csv('files/df_arrests_02.csv', index = False)
Geolocation Plots¶
Plot of Arrests¶
In [11]:
# Filtering the dataframe per year
df_arrests_2014 = df_arrests[(df_arrests['Arrest Date'] >= '2014-01-01') & (df_arrests['Arrest Date'] <= '2014-12-31')]
df_arrests_2015 = df_arrests[(df_arrests['Arrest Date'] >= '2015-01-01') & (df_arrests['Arrest Date'] <= '2015-12-31')]
df_arrests_2016 = df_arrests[(df_arrests['Arrest Date'] >= '2016-01-01') & (df_arrests['Arrest Date'] <= '2016-12-31')]
df_arrests_2017 = df_arrests[(df_arrests['Arrest Date'] >= '2017-01-01') & (df_arrests['Arrest Date'] <= '2017-12-31')]
df_arrests_2018 = df_arrests[(df_arrests['Arrest Date'] >= '2018-01-01') & (df_arrests['Arrest Date'] <= '2018-12-31')]
df_arrests_2019 = df_arrests[(df_arrests['Arrest Date'] >= '2019-01-01') & (df_arrests['Arrest Date'] <= '2019-12-31')]
In [270]:
# Function to create the arrest geolocation plots
def geo_plot_arrests(dataframe, title):
# Total
print(f'Tally of Marijuana Related Arrests: {len(dataframe)}')
# Plot area
fig, ax = plt.subplots(figsize = (20,10))
# Scatterplot
plt.scatter(dataframe['longitude'], dataframe['latitude'],c = 'red', s = 12, alpha = 0.4, label = 'Arrests')
# Labels and Legend
plt.xlabel('Longitude', fontsize = 20)
plt.ylabel('Latitude', fontsize = 20)
plt.xticks(fontsize = 15)
plt.yticks(fontsize = 15)
plt.legend(fontsize = 18)
plt.title(title, fontsize = 26)
plt.tight_layout()
In [271]:
# Total arrests in 2014
geo_plot_arrests(df_arrests_2014, 'Total Arrests in 2014')
In [14]:
# Total arrests in 2015
geo_plot_arrests(df_arrests_2015, 'Total Arrests in 2015')
In [15]:
# Total arrests in 2016
geo_plot_arrests(df_arrests_2016, 'Total Arrests in 2016')
In [16]:
# Total arrests in 2017
geo_plot_arrests(df_arrests_2017, 'Total Arrests in 2017')
In [17]:
# Total arrests in 2018
geo_plot_arrests(df_arrests_2018, 'Total Arrests in 2018')
In [18]:
# Total arrests in 2019
geo_plot_arrests(df_arrests_2019, 'Total Arrests in 2019')
The data and the plots show that there was a drop in arrests from 2014 to 2019, with a marked fall starting in 2017, the year the new marijuana laws came into force. Although correlation doesn't imply causation, this is nevertheless evidence that the law impacted marijuana arrests.
Plot of Arrests x Dispensaries¶
In [19]:
# Loading the dispensaries dataset
df_stores = pd.read_csv('files/df_stores.csv')
In [20]:
df_stores.shape
Out[20]:
In [21]:
df_stores.head()
Out[21]:
Closed dispensaries are not meaningful to the analysis and can skew results. Removing them:
In [22]:
# Checking for closed stores
df_stores['is_closed'].value_counts()
Out[22]:
In [23]:
# Checking the closed store's register
df_stores[df_stores['is_closed']]
Out[23]:
In [24]:
# Removing the register
df_stores.drop(df_stores.index[234], inplace = True)
In [25]:
# Checking for closed stores
df_stores['is_closed'].value_counts()
Out[25]:
For the rest of my analysis, only the name, latitude and longitude variables will be necessary. Dropping the other variables:
In [26]:
# Filtering the dataframe
df_stores = df_stores[['name', 'latitude', 'longitude']]
In [27]:
df_stores.head()
Out[27]:
In [28]:
# Statistical summary
df_stores.describe()
Out[28]:
In [29]:
# Saving the filtered dataframe
df_stores.to_csv('files/df_stores_02.csv', index = False)
In [274]:
# Function to plot arrest location x store location
def geo_plot_02(dataframe_arrests, year):
# Plot area
fig, ax = plt.subplots(figsize = (25,15))
# Scatterplot of Arrests
plt.scatter(dataframe_arrests['longitude'], dataframe_arrests['latitude'], c = 'red', s = 12, label = 'Arrests')
# Scatterplot of Dispensaries
ax.scatter(df_stores['longitude'], df_stores['latitude'], c = 'b', s = 12, label = 'Dispensaries')
# Labels and Legend
plt.xlabel('Longitude', fontsize = 20)
plt.ylabel('Latitude', fontsize = 20)
plt.xticks(fontsize = 15)
plt.yticks(fontsize = 15)
plt.legend(fontsize = 18)
plt.xlim(None, -118)
plt.title(f'{year}: Arrests x Dispensaries', fontsize = 30)
plt.tight_layout()
In [275]:
geo_plot_02(df_arrests_2014, 2014)
In [276]:
geo_plot_02(df_arrests_2015, 2015)
In [277]:
geo_plot_02(df_arrests_2016, 2016)
In [278]:
geo_plot_02(df_arrests_2017, 2017)
In [279]:
geo_plot_02(df_arrests_2018, 2018)
In [280]:
geo_plot_02(df_arrests_2019, 2019)
The arrests were clearly concentrated in areas close to dispensaries, both before and after legalization. Moreso, few if any stores changed places, and over the years more stores were opened. Therefore, considering that legalization didn't change the arrest locatations even though new stores were opened, it seems likely that arrest locations are not related to dispensary locations, but rather to other significant attributes of the city landscape (such as certain neighborhoods or popular areas).
Plot of Arrests x Schools¶
In [37]:
df_schools = pd.read_csv('files/df_schools.csv')
In [38]:
df_schools.shape
Out[38]:
In [39]:
df_schools.head()
Out[39]:
Geolocation data was not included in the school dataset, hence it will need to be obtained by other means. I'll use the school address paired with the google maps API to obtain the latitude and longitude of each school. Such data will also allow for the identification of duplicate entries (under different names) in the school list.
In [40]:
!pip install -q googlemaps
In [41]:
import googlemaps
In [42]:
# API Key
gmaps_key = googlemaps.Client(key = '')
In [43]:
# Lists to retrieve latitude and longitude
list_latitude = []
list_longitude = []
In [44]:
# Loop through the school addresses list
# This may take a while
for address in df_schools['complete_address']:
# Obtain the address' geocode
g = gmaps_key.geocode(address)
# Extract latitude and longitude
try:
lat = g[0]['geometry']['location']['lat']
lng = g[0]['geometry']['location']['lng']
# In case of error fill with NaN
except:
lat = np.NaN
lng = np.NaN
# Apprend latitude and longide to the lists
list_latitude.append(lat)
list_longitude.append(lng)
In [45]:
# Adding the lists as columns on the schools dataframe
df_schools['latitude'] = list_latitude
df_schools['longitude'] = list_longitude
In [46]:
# Checking for missing values
df_schools.isnull().sum()
Out[46]:
In [ ]:
# Using dropna() to avoid problems in case the API fails to fetch the geolocation of a school
df_schools = df_schools.dropna()
In [47]:
df_schools.shape
Out[47]:
In [48]:
# Creating a single column with all geolocation data
# This column can be used to filter duplicate entries
df_schools['coordinates'] = df_schools['latitude'].astype('str') + ', ' + df_schools['longitude'].astype(str)
In [49]:
df_schools.head()
Out[49]:
In [131]:
# Using the coordinates to remove duplicate entries
df_schools = df_schools[~df_schools.duplicated(subset = 'coordinates')]
In [132]:
# Reseting index as rows were removed
df_schools.reset_index(drop = True, inplace = True)
In [133]:
# Shape
df_schools.shape
Out[133]:
The shape shrank from 939 to 906, therefore there were indeed duplicate entries. Most likely schools which operated on the same location two different programs as two different institutions.
In [134]:
# Statistical summary
df_schools.describe()
Out[134]:
In [135]:
# The Unnamed: 0 column was the old index which became a column when the csv was imported
# It's not necessary. Removing it
df_schools.drop(labels = 'Unnamed: 0', axis = 1, inplace = True)
In [136]:
df_schools.head()
Out[136]:
In [137]:
# Saving the dataframe
df_schools.to_csv('files/df_schools_02.csv', index = False)
In [138]:
# Function to plot arrest location x school location
def geo_plot_03(dataframe_arrests, year):
# Plot area
fig, ax = plt.subplots(figsize = (25,15))
# Scatterplot of Arrests
plt.scatter(dataframe_arrests['longitude'], dataframe_arrests['latitude'], c = 'red', s = 12, label = 'Arrests')
# Scatterplot of Dispensaries
ax.scatter(df_schools['longitude'], df_schools['latitude'], c = 'g', s = 12, label = 'Schools')
# Labels and Legend
plt.xlabel('Longitude', fontsize = 20)
plt.ylabel('Latitude', fontsize = 20)
plt.xticks(fontsize = 15)
plt.yticks(fontsize = 15)
plt.legend(fontsize = 18)
plt.title(f'{year}: Arrests x Schools', fontsize = 30)
plt.tight_layout()
In [139]:
geo_plot_03(df_arrests_2014, 2014)
In [140]:
geo_plot_03(df_arrests_2015, 2015)
In [141]:
geo_plot_03(df_arrests_2016, 2016)
In [142]:
geo_plot_03(df_arrests_2017, 2017)
In [143]:
geo_plot_03(df_arrests_2018, 2018)
In [144]:
geo_plot_03(df_arrests_2019, 2019)
There are schools everywhere, so it's hard to assess visually whether there is a higher incidence of marijuana arrests near schools.
As instructed, my next step will be using the geopy package to calculate the distance between schools and arrests.
Distance from Arrests to Dispensaries and Schools¶
Distance Between Arrest Locations and Dispensaries¶
Here I'm looking at how many stores where in the area around each arrest
In [145]:
# Creating a dataframe to calculate distances - stores
df_stores_dist = df_stores[['name', 'latitude', 'longitude']]
In [146]:
df_stores_dist.shape
Out[146]:
In [147]:
df_stores_dist.head()
Out[147]:
In [148]:
# Creating a dataframe to calculate distances - arrests
df_arrests_dist = df_arrests[['Report ID', 'latitude', 'longitude']]
In [149]:
# The index is skipping rows, reseting it
df_arrests_dist.reset_index(drop = True, inplace = True)
In [150]:
df_arrests_dist.shape
Out[150]:
In [151]:
df_arrests_dist.head(3)
Out[151]:
In [152]:
# Loading the geopy package which can be used to calculate distance between coordinates
# The function which does this is geodesic()
import geopy
from geopy.distance import geodesic
In [153]:
%%time
# Loop through all stores and all arrests and calculate the distance
# This takes a while (more than 30 minutes)
for store in range(len(df_stores_dist)):
# Get the data from that store
store_name = df_stores_dist.iloc[store]['name']
store_lat = df_stores_dist.iloc[store]['latitude']
store_long = df_stores_dist.iloc[store]['longitude']
# create new column on arrest_dist with store name
df_arrests_dist[store_name] = np.NaN
# Create an empty list to be filled with
# The distance between arrest and that given store
distance_list = []
# Loop though all arrests
for arrest in range(len(df_arrests_dist)):
# Get the geolocation from arrest
arrest_lat = df_arrests_dist.iloc[arrest]['latitude']
arrest_long = df_arrests_dist.iloc[arrest]['longitude']
# Calculate distance from arrest to store
distance = geodesic((store_lat, store_long), (arrest_lat, arrest_long)).miles
# Add to the list containing all distances from a given store
distance_list.append(distance)
# Fill the store name column with the distances list
df_arrests_dist[store_name] = distance_list
In [269]:
df_arrests_dist.to_csv('files/distances_dispensaries.csv')
In [154]:
df_arrests_dist
Out[154]:
In [155]:
# Creating a new dataframe with only distance data
df_dist_arrest_store = df_arrests_dist.drop(['Report ID', 'latitude', 'longitude'], axis = 1)
In [156]:
df_dist_arrest_store.head(3)
Out[156]:
In [157]:
# For each arrest count how many stores were in a 0.5 mile range
df_arrests_dist['store_0.5_mile'] = (df_dist_arrest_store < 0.5).sum(axis = 1)
In [158]:
# For each arrest count how many stores were in a 1 mile range
df_arrests_dist['store_1_mile'] = (df_dist_arrest_store < 1).sum(axis = 1)
In [159]:
# Problem prevention: Checking df_arrests and df_arrests_dist
# To see if the 'Report ID' columns still match
len(df_arrests['Report ID'].unique()) == len(df_arrests_dist['Report ID'].unique())
Out[159]:
In [160]:
# Adding the columns with the calculated distances back to df_arrests
df_arrests = df_arrests.merge(df_arrests_dist[['Report ID', 'store_0.5_mile', 'store_1_mile']],
left_on = 'Report ID',
right_on = 'Report ID')
In [161]:
df_arrests.head(2)
Out[161]:
In [162]:
# Dropping the 'Unnamed: 0' column
df_arrests = df_arrests.drop('Unnamed: 0', 1)
In [163]:
# Saving df_arrests to disk
df_arrests.to_csv('files/df_arrests_03.csv', index = False)
Distance Between Arrests and Schools¶
Here I'm looking at how many arrests happened near each school
AND at how many schools were near each arrest
In [164]:
# Creating a dataframe to calculate distances - stores
df_schools_dist = df_schools[['School', 'latitude', 'longitude']].dropna()
In [165]:
df_schools_dist.shape
Out[165]:
In [166]:
df_schools.isnull().sum()
Out[166]:
In [167]:
df_schools_dist
Out[167]:
In [168]:
# Dataframe for distance between arrests and schools
df_dist_arrest_school = df_arrests[['Report ID', 'latitude', 'longitude']]
In [169]:
# The index is skipping rows, reseting it
df_dist_arrest_school.reset_index(drop = True, inplace = True)
In [170]:
df_dist_arrest_school
Out[170]:
In [171]:
%%time
# Loop through all stores and all arrests and calculate the distance
# This takes a long time! (2 hours or so)
for store in range(len(df_schools_dist)):
# Get the data from that store
store_name = df_schools_dist.iloc[store]['School']
store_lat = df_schools_dist.iloc[store]['latitude']
store_long = df_schools_dist.iloc[store]['longitude']
# create new column on arrest_dist with store name
df_dist_arrest_school[store_name] = np.NaN
# Create an empty list to be filled with
# The distance between arrest and that given store
distance_list = []
# Loop though all arrests
for arrest in range(len(df_arrests_dist)):
# Get the geolocation from arrest
arrest_lat = df_arrests_dist.iloc[arrest]['latitude']
arrest_long = df_arrests_dist.iloc[arrest]['longitude']
# Calculate distance from arrest to store
distance = geodesic((store_lat, store_long), (arrest_lat, arrest_long)).miles
# Add to the list containing all distances from a given store
distance_list.append(distance)
# Fill the store name column with the distances list
df_dist_arrest_school[store_name] = distance_list
In [268]:
df_dist_arrest_school.to_csv('files/distances_schools.csv')
In [172]:
df_dist_arrest_school.shape
Out[172]:
In [285]:
df_dist_arrest_school.head(2)
Out[285]:
In [174]:
df_dist_arrest_school_2 = df_dist_arrest_school.drop(['Report ID', 'latitude', 'longitude'], axis = 1)
In [284]:
df_dist_arrest_school_2.head(2)
Out[284]:
In [178]:
# For each ARREST count how many SCHOOLS were in a 0.5 mile range
df_dist_arrest_school['school_0.5_mile'] = (df_dist_arrest_school_2 < 0.5).sum(axis = 1)
In [179]:
# For each ARREST count how many SCHOOLS were in a 1 mile range
df_dist_arrest_school['school_1_mile'] = (df_dist_arrest_school_2 < 1).sum(axis = 1)
In [283]:
df_dist_arrest_school.head(2)
Out[283]:
In [181]:
# Now transposing the original distances dataframe in order to count the number of arrests near each school
df_dist_school_arrest = df_dist_arrest_school_2.T
In [184]:
# For each SCHOOL count how many ARREST happened in a 0.5 mile range
df_dist_school_arrest['arrest_0.5_mile'] = (df_dist_school_arrest < 0.5).sum(axis = 1)
In [185]:
# For each SCHOOL count how many ARREST happened in a 1 mile range
df_dist_school_arrest['arrest_1_mile'] = (df_dist_school_arrest < 1).sum(axis = 1)
In [282]:
df_dist_school_arrest.head(2)
Out[282]:
In [193]:
df_dist_school_arrest.reset_index(inplace = True)
In [205]:
df_dist_school_arrest.rename(columns = {'index': 'School'}, inplace = True)
In [206]:
# Problem prevention: Checking df_arrests and df_arrests_dist
# To see if the 'Report ID' columns still match
len(df_schools['School'].unique()) == len(df_dist_school_arrest['School'].unique())
Out[206]:
In [209]:
# Adding two columns to the schools dataframe
# With the tally of how many arrests happened nearby
df_schools = df_schools.merge(df_dist_school_arrest[['School', 'arrest_0.5_mile', 'arrest_1_mile']],
left_on = 'School',
right_on = 'School')
In [210]:
df_schools.head(2)
Out[210]:
In [211]:
df_arrests.head(2)
Out[211]:
In [213]:
# Adding two columns to the arrests dataframe
# With the tally of how many schools were nearby
df_arrests['school_0.5_mile'] = (df_dist_arrest_school_2 < 0.5).sum(axis =1)
df_arrests['school_1_mile'] = (df_dist_arrest_school_2 < 1).sum(axis =1)
In [286]:
df_arrests.head(2)
Out[286]:
In [215]:
df_arrests.columns
Out[215]:
In [216]:
# Saving to disk all dataframes
df_arrests.to_csv('files/df_arrests_04.csv', index = False)
df_schools.to_csv('files/df_schools_03.csv' ,index = False)
Arrest Time Series¶
In [217]:
# Range of dates for indexing
index_dates = pd.date_range('2014-01-01', '2019-12-31')
In [218]:
df_arrests
Out[218]:
In [219]:
# Dataframe with arrests grouped per day (categorial variables will disappear)
df_arrests_g = df_arrests.groupby('Arrest Date').sum()
In [220]:
# Tally of arrest near schools per day
df_arrests_g['school_0.5_mile'].head()
Out[220]:
In [221]:
# Tally of arrest near schools per day
df_arrests_g['school_1_mile'].head()
Out[221]:
In [222]:
# Making dataframes
df_arrests_half_mile_schools = pd.DataFrame(df_arrests_g['school_0.5_mile'])
df_arrests_one_mile_schools = pd.DataFrame(df_arrests_g['school_1_mile'])
In [225]:
df_arrests_half_mile_schools.head()
Out[225]:
In [226]:
df_arrests_one_mile_schools.head()
Out[226]:
The dataframe index is of type: Index
To be able to tally the monthly totals, it must be of type: DateTimeIndex
In [227]:
# Checking index type
df_arrests_half_mile_schools.index
Out[227]:
In [228]:
# Converting index type
df_arrests_half_mile_schools.index = pd.to_datetime(df_arrests_half_mile_schools.index)
df_arrests_one_mile_schools.index = pd.to_datetime(df_arrests_one_mile_schools.index)
In [229]:
# Using resampling to tally monthly data
df_arrests_half_mile_schools.resample('M').sum().head()
Out[229]:
Timeseries of Arrests Near Schools¶
In [230]:
# Number of arrests with a school in a 0.5-mile area
# Figure
plt.figure(figsize = (20,10))
# Plot data
plt.plot(df_arrests_half_mile_schools.resample('M').sum(), color = 'blue', linewidth = 3)
# Labels and Legend
plt.xticks(fontsize = 15)
plt.yticks(fontsize = 15)
plt.title('\nNumber of Arrests with Schools on a 0.5-Mile Range\n', fontsize = 30)
plt.xlabel('\nMonth of Year')
plt.ylabel('\nNumber of Arrests')
plt.tight_layout()
In [231]:
# Number of arrests with a school in a 0.5-mile area
# Figure
plt.figure(figsize = (20,10))
# Plot data
plt.plot(df_arrests_one_mile_schools.resample('M').sum(), color = 'blue', linewidth = 3)
# Labels and Legend
plt.xticks(fontsize = 15)
plt.yticks(fontsize = 15)
plt.title('\nNumber of Arrests with Schools on a 1-Mile Range\n', fontsize = 30)
plt.xlabel('\nMonth of Year')
plt.ylabel('\nNumber of Arrests')
plt.tight_layout()
Something changed in 2017 that caused arrests near schools to drop. Was it marijuana legalization? I cannot be affirmed with total certainty, but everything thus far points on that direction. One important question is: Have the 'crimes' been reduced, or was this simply a change in how they are accounted for (what used to be a crime no longer is)?
Top Reasons for Arrest¶
In [232]:
# Function to plot the tally of arrests per crime on each year
def plot_total_crime_type(dataframe, year):
# Filtering the dataframe by type of crime
# Counting the number of occurances per type
# Rearrange in ascending order
# Create a bar plot
ax = dataframe['Charge Description'].value_counts(ascending = True).tail().plot(kind = 'bar',
figsize = (16,10),
color = 'magenta',
alpha = 0.7,
rot = 30)
ax.set_title(f'\nTally of Arrests per Crime Type in {year}\n', fontsize = 30)
ax.set_ylabel('\nCrime Type\n', fontsize = 18);
ax.set_xlabel('\nCrime Tally\n', fontsize = 18)
In [233]:
# Before plotting the data there is an issue to fix on the dataset
# There are two entries for "possess 28.5 grams or less of marijuana"
# With the second one having an extra "**"
# Merging both those
df_arrests['Charge Description'] = df_arrests['Charge Description'].map(lambda x: 'possess 28.5 grams or less of marijuana' if x == 'possess 28.5 grams or less of marijuana**' else x)
In [234]:
# Creating new dataframes per year
df_arrests_2014 = df_arrests[(df_arrests['Arrest Date'] <= '2014-12-31') & (df_arrests['Arrest Date'] >= '2014-01-01')]
df_arrests_2015 = df_arrests[(df_arrests['Arrest Date'] <= '2015-12-31') & (df_arrests['Arrest Date'] >= '2015-01-01')]
df_arrests_2016 = df_arrests[(df_arrests['Arrest Date'] <= '2016-12-31') & (df_arrests['Arrest Date'] >= '2016-01-01')]
df_arrests_2017 = df_arrests[(df_arrests['Arrest Date'] <= '2017-12-31') & (df_arrests['Arrest Date'] >= '2017-01-01')]
df_arrests_2018 = df_arrests[(df_arrests['Arrest Date'] <= '2018-12-31') & (df_arrests['Arrest Date'] >= '2018-01-01')]
df_arrests_2019 = df_arrests[(df_arrests['Arrest Date'] <= '2019-12-31') & (df_arrests['Arrest Date'] >= '2019-01-01')]
In [235]:
# Total Arrests per Crime in 2014
plot_total_crime_type(df_arrests_2014, 2014)
In [241]:
# Total Arrests per Crime in 2015
plot_total_crime_type(df_arrests_2015, 2015)
In [242]:
# Total Arrests per Crime in 2016
plot_total_crime_type(df_arrests_2016, 2016)
In [243]:
# Total Arrests per Crime in 2017
plot_total_crime_type(df_arrests_2017, 2017)
In [244]:
# Total Arrests per Crime in 2018
plot_total_crime_type(df_arrests_2018, 2018)
In [245]:
# Total Arrests per Crime in 2019
plot_total_crime_type(df_arrests_2019, 2019)
Statistical Modeling¶
In [246]:
df_arrests.head()
Out[246]:
In [247]:
df_arrests.describe().T
Out[247]:
In [248]:
df_arrests.dtypes
Out[248]:
In [253]:
df_arrests.nunique()
Out[253]:
In [254]:
df_arrests_vars = df_arrests.describe().columns
In [258]:
# Histograms
# Figure
fig = plt.figure(figsize = (32,16))
# Plot a histogram of each variable
for i, col in enumerate(df_arrests_vars):
fig.add_subplot(4,4,1+i)
col_data = df_arrests[col]
plt.hist(col_data, color = 'khaki')
plt.title(col)
Looks like none of the variables follows a normal distribution. This can be later checked with am appropriate test!
In [260]:
# Now moving to the schools dataframe
df_schools.describe()
Out[260]:
In [261]:
df_schools_vars = ['arrest_0.5_mile', 'arrest_1_mile']
In [263]:
# Histograms
# Figure
fig = plt.figure(figsize = (16,8))
# Plot a histogram of each variable
for i, col in enumerate(df_schools_vars):
fig.add_subplot(2,4,1+i)
col_data = df_schools[col]
plt.hist(col_data, color = 'green')
plt.title(col)
Again doesn't seem to be normally distributed.
Normality Test¶
Quick Review: The p-value is the probability of obtaining a test statistic at least as extreme as the one observed when the null hypothesis is true. The null hypothesis (H0) affirms that the population is normally distributed, against the alternative hypothesis (H1) that it is not normally distributed.
In [264]:
# Function to test variable normality
def test_normality(dataframe, list_of_cols, sig_level = 0.05):
# List columns on dataframe
total_variables = len(list_of_cols)
# Start the counter of varibles which are 'non normal'
not_normal_variables = 0
# Loop through each column
for col in list_of_cols:
# Run the normality test (from statsmodels)
p_val = stats.normaltest(dataframe[col])[1]
# Significance Level
sig_lvl = 0.05
# Check p-value
if p_val < sig_lvl:
print(f'{col}')
print(f'P-value: {p_val}\nSignificance Level: {sig_lvl}\n')
print("The p-value is lower than the Significance Level, hence the null hypothesis is rejected and we conclude that the variable is not normally distributed.\n")
not_normal_variables += 1
elif p_val > sig_lvl:
print(f'{col}')
print(f'P-value: {p_val}\nSignificance Level: {sig_lvl}\n')
print("The p-value is greater than the Significance Level, hence we fail to reject the null hypothesis that the variable is normally distributed.\n")
if not_normal_variables > 0:
print(f'{not_normal_variables} variables failed the normality test')
In [265]:
# Test normality for arrests
test_normality(df_arrests, df_arrests_vars)
In [266]:
# Test normality for schools
test_normality(df_schools, df_schools_vars)
Conclusion: None of the variables is normally distributed. This means that much caution is needed when creating the predictive models.
Final Conclusion¶
Clearly there seems to be a pattern between number of arrests and marijuana legalization. The proximity between arrests and dispensaries or schools doesn't seem to change from year to year, which seems to imply that there is no correlation between them. The important question to answer now is whether is there is some other factor which could better explain the drop in marijuana related crimes.
