## Adding colour map to stacked bar plot

We start with the bar plot I created here. Which gave us this…

Now we want to give it some nice colours.

This we can do by importing the colour map from Matplotlib and extracting a range of colours. I like the Viridis colormap so I’ll use that. Also, we have 5 variables in our stacked bar plot, so we only want 5 colours from it.

```from matplotlib import cm

# get the colormap and extract 5 colours
viridis = cm.get_cmap('viridis', 5)```

You can get the individual values like this…

`print(viridis.colors)`

Which should give you something like this

```[[0.267004 0.004874 0.329415 1.      ]
[0.229739 0.322361 0.545706 1.      ]
[0.127568 0.566949 0.550556 1.      ]
[0.369214 0.788888 0.382914 1.      ]
[0.993248 0.906157 0.143936 1.      ]]```

Now we can allocate each of these values to out plot. Lets do the first one as an example…

```p1 = plt.bar(xMain,
dataVar1,
color=viridis.colors[0])```

Once you do the same for the other 4, you should get something that looks like this.

So much nicer!

The complete code looks like this:

```#!/usr/local/bin/python3
# -*- coding: utf-8 -*-
#
# make a stackbar for data

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
from matplotlib import cm

# open the file

# get the data you want to plot
dataVar1 = df['column1']
dataVar2 = df['column2']
dataVar3 = df['column3']
dataVar4 = df['column4']
dataVar5 = df['column5']

# number of entries (x axis)
xMain = np.arange(30)

# color map
viridis = cm.get_cmap('viridis', 5)

# plot each data
p1 = plt.bar(xMain,
dataVar1,
color=viridis.colors[0])

p2 = plt.bar(xMain,
dataVar2,
bottom=dataVar1,
color=viridis.colors[1])

p3 = plt.bar(xMain,
dataVar3,
bottom=dataVar1+dataVar2,
color=viridis.colors[2])

p4 = plt.bar(xMain,
dataVar4,
bottom=dataVar1+dataVar2+dataVar3,
color=viridis.colors[3])

p5 = plt.bar(xMain,
dataVar4,
bottom=dataVar1+dataVar2+dataVar3+dataVar4,
color=viridis.colors[4])

# show the graph
plt.savefig('output_fig.png')```

## Stacked bar plot using matplotlib

How to create a simple stacked bar chart.

```import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# open data file

# get the data you want to plot
dataVar1 = df['column1']
dataVar2 = df['column2']
dataVar3 = df['column3']
dataVar4 = df['column4']
dataVar5 = df['column5']

# number of entries (x axis)
xMain = np.arange(30)

# plot each data
p1 = plt.bar(xMain,
dataVar1)

p2 = plt.bar(xMain,
dataVar2,
bottom=dataVar1)

p3 = plt.bar(xMain,
dataVar3,
bottom=dataVar1+dataVar2)

p4 = plt.bar(xMain,
dataVar4,
bottom=dataVar1+dataVar2+dataVar3)

p5 = plt.bar(xMain,
dataVar4,
bottom=dataVar1+dataVar2+dataVar3+dataVar4)

# save the graph
plt.savefig('output_fig.png')
```

The `botttom=dataVarn` option is needed to stack the data, otherwise it will draw the next set of data at y=0.

If you do it right, you’ll get something like this.

# Stoping mongo

If its already running, stop it

`brew services stop mongo`

If all goes well, it should return something like this

```Stopping `mongodb`... (might take a while)
==> Successfully stopped `mongodb` (label: homebrew.mxcl.mongodb)```

# Locate the mongo config file and open it

should be here…

`/usr/local/etc/mongo.conf`

…which I will edit using Vim

`sudo vim /usr/local/etc/mongo.conf`

# Edit the storage path

The file should look something like this….

```systemLog:
destination: file
path: /usr/local/var/log/mongodb/mongo.log
logAppend: true
storage:
dbPath: /usr/local/var/mongodb
net:
bindIp: 127.0.0.1```

Line 6 is the one we are after, change that to the location you want.

`dbPath: /my/new/path/is/here`

Save the file and close it.

# Start mongo again

`brew services start mongo`

If everything goes well, Mongo should have added a whole bunch of files to the new directory.

## CSV files into SQLite database

This bit of code will transfer a whole bunch of csv files in the ‘logs/’ directory into a single SQLite db file called `repo_data.db`.

```import pandas as pd
import re
import os
import sqlite3 as sql

# create list of all the different files
list_files = [x for x in os.listdir('logs') if x.endswith('.csv')]

# create a new file with a connection
db_con = sql.connect('repo_data.db')

# for 8-bit string instead of unicode...why?
db_con.text_factory = str

# now go through the things and create a table for each
for str_raw_name in list_files:

# clean name and lower
str_name = re.sub('.csv', '', str_raw_name).lower()

# open the file and create

# pass on to the database
df.to_sql(str_name, con = db_con, index=False)

print 'DONE', str_name```

NOTE: I kept getting an error about some 8-bit blah blah blah text thing. I googled it and fixed it (line 13), but I still have no idea what it means.

## Git log format for project

Formatting the log file from a git repo into a nice table.

You will have to `cd` into the repo’s folder…

`git log --full-history --pretty=format:%H,%an,%ae,%ad,%f --date=short > ../outputRawData.csv`

where `%f` is a sanitised version of the commit’s subject.

## Create Word document with Python

How to create a simple word document using a python script.

`pip3 install python-docx`

yes…I’m using Python3 now.

Then, this is how simple the rest is…

```from docx import Document

str_title = 'This is the document Title'

# create the document
doc_output = Document()

# save the file
doc_output.save('testing.docx')```

## Using the .isin() function in Pandas

Searching and selecting data from a dataframe using the `.isin()` function.

The `.isin()` function is a powerful tool that can help you search search for a number of values in a data frame.

This is how it’s done.

We start by creating a simple data frame

```import pandas as pd

df = pd.DataFrame({'col_1':[1,2,3,4],'col_2':[2,3,4,1]})```

The data frame should look something like this

```col_1  col_2
0      1      2
1      2      3
2      3      4
3      4      1```

Now, we will use the `.isin()` function to select all the rows that have either the number 1 or 4 in `col_1`, and we’ll put them in a new data frame called `df_14`.

We do this by using a list of the values and placing that list in the `.isin()` function.

either like this, if we have a short list…

`df_14 = df[df['col_1'].isin([1,4])]`

…or like this, when we have a longer list.

```list_numbers = [1,4]
df_14 = df[df['col_1'].isin(list_numbers)]```

The new data frame should look like this

```col_1  col_2
0      1      2
3      4      1```

Simple.

But what if we want all the values that DO NOT match those in the list?

We can do this by adding `==False` to the function. Like this

`df_not_14 = df[df['col_1'].isin([1,4])==False]`

and this new data frame would look like this

```col_1  col_2
1      2      3
2      3      4```

## Finding and deleting empty file

A short Python script to find and delete all empty files.

```#!/usr/local/bin/python
# -*- coding: utf-8 -*-
#
# search and delete empty files

import os

# this is the directory
str_directory = 'myfolder'

# get list of all files in the directory and remove possible hidden files
list_files = [x for x in os.listdir(str_directory) if x[0]!='.']

# now loop through the files and remove empty ones
for each_file in list_files:
file_path = '%s/%s' % (str_directory, each_file)

# check size and delete if 0
if os.path.getsize(file_path)==0:
os.remove(file_path)
else:
pass```

## From csv to json

EDIT: The original script is for idiots…smart people use the `to_dict()` function, then save as a JSON dump.

```import json

with open(‘path/to/file.json’, ‘w’) as fw:
```

This is the stupid version …

```#!/usr/local/bin/python
# -*- coding: utf-8 -*-

import json
import pandas as pd

# this is the file that will be open
str_file_path = 'sample_pages_data.csv'

# get file name for output
str_name = str_file_path[:-4]

# open the dataframe

# list of columns
list_cols = df_data.columns

# the output list
list_output = []

# iterate through each row
for df_row in df_data.iterrows():

# dictionary for each entry
dic_data = {}

# loop through each columns
for str_column in list_cols:

dic_data[str_column] = df_row[1][str_column]

# append dictionary to list_output
list_output.append(dic_data)

# from dictionary to json and save output file
json_data = json.dumps(list_output, indent=2)

# save json file
with open('%s.json' % str_name, 'w') as fw:
fw.write(json_data)```

## Dealing with dates in Pandas

A few lines on how to deal with dates and date formating issues in Pandas.

First thing first, import the pandas module and read the csv file

```>>> import pandas as pd

So, we have a simple data frame like this…

```>>> df
start         end
0  2001-06-01  2004-02-01
1  2001-11-01  2003-12-01
2  2005-04-01  2007-03-01
3  2005-05-01  2007-03-01```

…and we want to calculate the amount of time between the start and the end.

The main problems is that the dates are read as string (when read from a file), and therefore there is very little we can do with this right now.

To change the column to dates, we can use the to_datetime function in pandas. We also don’t want to be completely destructive and risk messing up the data, so we put the newly formatted dates into new columns (`start_d` and `end_d`), like this…

```>>> df['start_d'] = pd.to_datetime(df['start'])
>>> df['end_d'] = pd.to_datetime(df['end'])```

Now, our data frame should look a bit like this

```>>> df
start         end    start_d      end_d
0  2001-06-01  2004-02-01 2001-06-01 2004-02-01
1  2001-11-01  2003-12-01 2001-11-01 2003-12-01
2  2005-04-01  2007-03-01 2005-04-01 2007-03-01
3  2005-05-01  2007-03-01 2005-05-01 2007-03-01```

to calculate the length of time between start and end, we simply subtract `start_d` from `end_d`, like this

`>>> df['len'] = df['end_d'] - df['start_d']`

which will result in the difference being calculated in days, leaving the data frame looking like this

```        start         end    start_d      end_d      len
0  2001-06-01  2004-02-01 2001-06-01 2004-02-01 975 days
1  2001-11-01  2003-12-01 2001-11-01 2003-12-01 760 days
2  2005-04-01  2007-03-01 2005-04-01 2007-03-01 699 days
3  2005-05-01  2007-03-01 2005-05-01 2007-03-01 669 days```