My Motivation¶
- I like Python
- I like plotting data
- I like sharing
- I think Pandas is awesome and you should use it too
Motto: »Pandas as early as possible!«
Tutorial Setup¶
- 60 minutes (we might do this again for some advanced stuff if you want to)
- Well, as it turns out, 60 minutes weren't nearly enought
- We ended up spending nearly 2 hours on it, and we needed to rush quickly through the material
- Alternating between lecture and hands-on
- Please give status of hands-ons via pollev.com/aherten538
- Please open Jupyter Notebook of this session
- … either on your local machine (
pip install --user pandas seaborn) - … or on the JSC Jupyter service at https://jupyter-jsc.fz-juelich.de/
Pandas and seaborn should already be there!
- … either on your local machine (
- Tell me when you're done on pollev.com/aherten538
About Pandas¶
- Python package (Python 2, Python 3)
- For data analysis
- With data structures (multi-dimensional table; time series), operations
- Name from »Panel Data« (multi-dimensional time series in economics)
- Since 2008
- https://pandas.pydata.org/
- Install via PyPI:
pip install pandas
Pandas Cohabitation¶
- Pandas works great together with other established Python tools
- Jupyter Notebooks
- Plotting with
matplotlib - Modelling with
statsmodels,scikit-learn - Nicer plots with
seaborn,altair,plotly
First Steps¶
In [1]:
import pandas
In [2]:
import pandas as pd
In [3]:
pd.__version__
Out[3]:
In [4]:
%pdoc pd
In [5]:
ages = [41, 56, 56, 57, 39, 59, 43, 56, 38, 60]
In [6]:
pd.DataFrame(ages)
Out[6]:
In [7]:
df_ages = pd.DataFrame(ages)
df_ages.head(3)
Out[7]:
- Let's add names to ages; put everything into a
dict()
In [8]:
data = {
"Names": ["Liu", "Rowland", "Rivers", "Waters", "Rice", "Fields", "Kerr", "Romero", "Davis", "Hall"],
"Ages": ages
}
print(data)
In [9]:
df_sample = pd.DataFrame(data)
df_sample.head(4)
Out[9]:
- Two columns now; one for names, one for ages
In [10]:
df_sample.columns
Out[10]:
DataFramealways have indexes; auto-generated or custom
In [11]:
df_sample.index
Out[11]:
- Make
Namesbe index with.set_index() inplace=Truewill modifiy the parent frame (I don't like it)
In [12]:
df_sample.set_index("Names", inplace=True)
df_sample
Out[12]:
- Some more operations
In [13]:
df_sample.describe()
Out[13]:
In [14]:
df_sample.T
Out[14]:
In [15]:
df_sample.T.columns
Out[15]:
- Also: Arithmetic operations
In [16]:
df_sample.multiply(2).head(3)
Out[16]:
In [17]:
df_sample.reset_index().multiply(2).head(3)
Out[17]:
In [18]:
(df_sample / 2).head(3)
Out[18]:
In [19]:
(df_sample * df_sample).head(3)
Out[19]:
Logical operations allowed as well
In [20]:
df_sample > 40
Out[20]:
Task 1¶
- Create data frame with
- 10 names of dinosaurs,
- their favourite prime number,
- and their favourite color
- Play around with the frame
- Tell me on poll when you're done: pollev.com/aherten538
In [21]:
happy_dinos = {
"Dinosaur Name": [],
"Favourite Prime": [],
"Favourite Color": []
}
#df_dinos =
In [22]:
happy_dinos = {
"Dinosaur Name": ["Aegyptosaurus", "Tyrannosaurus", "Panoplosaurus", "Isisaurus", "Triceratops", "Velociraptor"],
"Favourite Prime": ["4", "8", "15", "16", "23", "42"],
"Favourite Color": ["blue", "white", "blue", "purple", "violet", "gray"]
}
df_dinos = pd.DataFrame(happy_dinos).set_index("Dinosaur Name")
df_dinos.T
Out[22]:
Some more DataFrame examples
In [24]:
df_demo = pd.DataFrame({
"A": 1.2,
"B": pd.Timestamp('20180226'),
"C": [(-1)**i * np.sqrt(i) + np.e * (-1)**(i-1) for i in range(5)],
"D": pd.Categorical(["This", "column", "has", "entries", "entries"]),
"E": "Same"
})
df_demo
Out[24]:
In [25]:
df_demo.sort_values("C")
Out[25]:
In [26]:
df_demo.round(2).tail(2)
Out[26]:
In [27]:
df_demo.round(2).sum()
Out[27]:
In [28]:
print(df_demo.round(2).to_latex())
Reading External Data¶
(Links to documentation)
Example:
{
"Character": ["Sawyer", "…", "Walt"],
"Actor": ["Josh Holloway", "…", "Malcolm David Kelley"],
"Main Cast": [true, "…", false]
}
In [29]:
pd.read_json("lost.json").set_index("Character").sort_index()
Out[29]:
Task 2¶
- Read in
nest-data.csvtoDataFrame; call itdf
Data was produced with JUBE, Pandas works very well together with JUBE - Get to know it and play a bit with it
- Tell me when you're done: pollev.com/aherten538
In [30]:
!cat nest-data.csv | head -3
In [31]:
df = pd.read_csv("nest-data.csv")
df.head()
Out[31]:
Read CSV Options¶
- See also full API documentation
- Important parameters
sep: Set separator (for example:instead of,)header: Specify info about headers for columns; able to use multi-index for columns!names: Alternative toheader– provide your own column titlesusecols: Don't read whole set of columns, but only these; works with any list (range(0:20:2))…skiprows: Don't read in these rowsna_values: What string(s) to recognize asN/Avalues (which will be ignored during operations on data frame)parse_dates: Try to parse dates in CSV; different behaviours as to provided data structure; optionally used together withdate_parsercompression: Treat input file as compressed file ("infer", "gzip", "zip", …)decimal: Decimal point divider – for German data…
pandas.read_csv(filepath_or_buffer, sep=', ', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=0, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, iterator=False, chunksize=None, compression='infer', thousands=None, decimal=b'.', lineterminator=None, quotechar='"', quoting=0, doublequote=True, escapechar=None, comment=None, encoding=None, dialect=None, tupleize_cols=None, error_bad_lines=True, warn_bad_lines=True, delim_whitespace=False, low_memory=True, memory_map=False, float_precision=None)
In [32]:
df_demo.head(3)
Out[32]:
In [33]:
df_demo["C"]
Out[33]:
- Select more than one column by providing list
[]to slice operator[] - You usually end up forgetting one of the brackets…
- Example: Select list of columns
AandC,["A", "C"]fromdf_demo
In [34]:
df_demo[["A", "C"]]
Out[34]:
In [35]:
df_demo[1:3]
Out[35]:
- Get a certain range as per the current sort structure
In [36]:
df_demo.iloc[1:3]
Out[36]:
In [37]:
df_demo.iloc[1:6:2]
Out[37]:
- Attention:
.iloc[]location might change after re-sorting!
In [38]:
df_demo.sort_values("C").iloc[1:3]
Out[38]:
- One more row-slicing option:
.loc[] - See the difference with a proper index (and not the auto-generated default index from before)
In [39]:
df_demo_indexed = df_demo.set_index("D")
df_demo_indexed
Out[39]:
In [40]:
df_demo_indexed.loc["entries"]
Out[40]:
Advanced Slicing: Logical Slicing¶
In [41]:
df_demo[df_demo["C"] > 0]
Out[41]:
In [42]:
df_demo[(df_demo["C"] < 0) & (df_demo["D"] == "entries")]
Out[42]:
Adding to Existing Data Frame¶
- Add new columns with
frame["new col"] = somethingor.insert() - Add new rows with
frame.append() - Combine data frames
- Concat: Combine several data frames along an axis
- Merge: Combine data frames on basis of common columns; database-style
- (Join)
- See user guide on merging
In [43]:
df_demo.head(3)
Out[43]:
In [44]:
df_demo["F"] = df_demo["C"] - df_demo["A"]
df_demo.head(3)
Out[44]:
In [45]:
df_demo.insert(df_demo.shape[1], "G", df_demo["C"] ** 2)
In [46]:
df_demo.tail(3)
Out[46]:
In [47]:
df_demo.append(
{"A": 1.3, "B": pd.Timestamp("2018-02-27"), "C": -0.777, "D": "has it?", "E": "Same", "F": 23},
ignore_index=True
)
Out[47]:
Combining Frames¶
- First, create some simpler data frame to show
.concat()and.merge()
In [48]:
df_1 = pd.DataFrame({"Key": ["First", "Second"], "Value": [1, 1]})
df_1
Out[48]:
In [49]:
df_2 = pd.DataFrame({"Key": ["First", "Second"], "Value": [2, 2]})
df_2
Out[49]:
- Concatenate list of data frame vertically (
axis=0)
In [50]:
pd.concat([df_1, df_2])
Out[50]:
- Same, but re-index
In [51]:
pd.concat([df_1, df_2], ignore_index=True)
Out[51]:
- Concat, but horizontally
In [52]:
pd.concat([df_1, df_2], axis=1)
Out[52]:
- Merge on common column
In [53]:
pd.merge(df_1, df_2, on="Key")
Out[53]:
Task 3¶
- Add a column to the Nest data frame called
Virtual Processeswhich is the total number of threads across all nodes (i.e. the product of threads per task and tasks per node and nodes) - Remember to tell me when you're done: pollev.com/aherten538
In [54]:
df["Virtual Processes"] = df["Nodes"] * df["Tasks/Node"] * df["Threads/Task"]
df.head()
Out[54]:
In [55]:
df.columns
Out[55]:
Aside: Plotting without Pandas¶
Matplotlib 101¶
- Matplotlib: de-facto standard for plotting in Python
- Main interface:
pyplot; provides MATLAB-like interface - Better: Use object-oriented API with
FigureandAxis - Great integration into Jupyter Notebooks
- Since v. 3: Only support for Python 3
- → https://matplotlib.org/
In [56]:
import matplotlib.pyplot as plt
%matplotlib inline
In [57]:
x = np.linspace(0, 2*np.pi, 400)
y = np.sin(x**2)
In [58]:
fig, ax = plt.subplots()
ax.plot(x, y)
ax.set_title('Use like this')
ax.set_xlabel("Numbers again");
ax.set_ylabel("$\sqrt{x}$");
- Plot multiple lines into one canvas
- Call
ax.plot()multiple times
In [59]:
y2 = y/np.exp(y*1.5)
In [60]:
fig, ax = plt.subplots()
ax.plot(x, y, label="y")
ax.plot(x, y2, label="y2")
ax.legend()
ax.set_title("This plot makes no sense");
Task 4¶
- Sort the data frame by the virtual proccesses
- Plot
"Presim. Time / s"and"Sim. Time / s"of our data framedfas a function of the virtual processes - Use a dashed, red line for
"Presim. Time / s", a blue line for"Sim. Time / s"(see API description) - Don't forget to label your axes and to add a legend
- Submit when you're done: pollev.com/aherten538
In [61]:
df.sort_values(["Virtual Processes", "Nodes", "Tasks/Node", "Threads/Task"], inplace=True)
In [62]:
fig, ax = plt.subplots()
ax.plot(df["Virtual Processes"], df["Presim. Time / s"], linestyle="dashed", color="red")
ax.plot(df["Virtual Processes"], df["Sim. Time / s"], "-b")
ax.set_xlabel("Virtual Process")
ax.set_ylabel("Time / s")
ax.legend();
Plotting with Pandas¶
- Each data frame hast a
.plot()function (see API) - Plots with Matplotlib
- Important API options:
kind:line(default),bar[h],hist,box,kde,scatter,hexbinsubplots: Make a sub-plot for each column (good together withsharex,sharey)figsizegrid: Add a grid to plot (use Matplotlib options)style: Line style per column (accepts list or dict)logx,logy,loglog: Logarithmic plotsxticks,yticks: Use values for ticksxlim,ylim: Limits of axesyerr,xerr: Add uncertainty to data pointsstacked: Stack a bar plotsecondary_y: Use a secondaryyaxis for this plot- Labeling
title: Add title to plot (Use a list of strings ifsubplots=True)legend: Add a legendtable: Iftrue, add table of data under plot
**kwds: Every non-parsed keyword is passed through to Matplotlib's plotting methods
- Either slice and plot…
In [63]:
df_demo["C"].plot(figsize=(10, 2));
- … or plot and select
In [64]:
df_demo.plot(y="C", figsize=(10, 2));
- I prefer slicing first, as it allows for further operations on the sliced data frame
In [65]:
df_demo["C"].plot(kind="bar");
- There are pseudo-sub-functions for each of the plot
kinds - I prefer to just call
.plot(kind="smthng")
In [66]:
df_demo["C"].plot.bar();
In [67]:
df_demo["C"].plot(kind="bar", legend=True, figsize=(12, 4), ylim=(-1, 3), title="This is a C plot");
Task 5¶
Use the NEST data frame df to:
- Make the virtual processes the index of the data frame (
.set_index()) - Plot
"Presim. Program / s"and"Sim. Time / s" individually - Plot them onto one common canvas!
- Make them have the same line colors and styles as before
Add a legend, add missing labels
Done? Tell me! pollev.com/aherten538
In [68]:
df.set_index("Virtual Processes", inplace=True)
In [69]:
df["Presim. Time / s"].plot(figsize=(10, 3));
In [70]:
df["Sim. Time / s"].plot(figsize=(10, 3));
In [71]:
df["Presim. Time / s"].plot();
df["Sim. Time / s"].plot();
In [72]:
ax = df[["Presim. Time / s", "Sim. Time / s"]].plot();
ax.set_ylabel("Time / s");
In [73]:
df[["Presim. Time / s", "Sim. Time / s"]].plot();
- That's why I think Pandas is great!
- It has great defaults to quickly plot data
- Plotting functionality is very versatile
- Before plotting, data can be massaged within data frames, if needed
In [74]:
df_demo[["A", "C", "F"]].plot(kind="bar", stacked=True);
In [75]:
df_demo[df_demo["F"] < 0][["A", "C", "F"]].plot(kind="bar", stacked=True);
In [76]:
df_demo[df_demo["F"] < 0][["A", "C", "F"]]\
.plot(kind="barh", subplots=True, sharex=True, title="Subplots", figsize=(12, 4));
In [77]:
df_demo[df_demo["F"] < 0][["A", "F"]]\
.plot(
style=["-*r", "--ob"],
secondary_y="A",
figsize=(12, 6),
table=True
);
In [78]:
df_demo[df_demo["F"] < 0][["A", "F"]]\
.plot(
style=["-*r", "--ob"],
secondary_y="A",
figsize=(12, 6),
yerr={
"A": df_demo[df_demo["F"] < 0]["C"],
"F": 0.2
},
capsize=4,
title="Bug: style is ignored with yerr",
marker="P"
);
Combine Pandas with Matplotlib¶
- Pandas shortcuts very handy
- But sometimes, one needs to access underlying Matplotlib functionality
- No problemo!
- Option 1: Pandas always returns axis
- Use this to manipulate the canvas
- Get underlying
figurewithax.get_figure()(forfig.savefig())
- Option 2: Create figure and axes with Matplotlib, use when drawing
.plot(): Useaxoption
Option 1: Pandas Returns Axis¶
In [79]:
ax = df_demo["C"].plot(figsize=(10, 4))
ax.set_title("Hello there!");
fig = ax.get_figure()
fig.suptitle("This title is super!");
Option 2: Draw on Matplotlib Axes¶
In [80]:
fig, ax = plt.subplots(figsize=(10, 4))
df_demo["C"].plot(ax=ax)
ax.set_title("Hello there!");
fig.suptitle("This title is super!");
- We can also get fancy!
In [81]:
fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True, figsize=(12, 4))
for ax, column, color in zip([ax1, ax2], ["C", "F"], ["blue", "#b2e123"]):
df_demo[column].plot(ax=ax, legend=True, color=color)
Aside: Seaborn¶
- Python package on top of Matplotlib
- Powerful API shortcuts for plotting of statistical data
- Manipulate color palettes
- Works well together with Pandas
- Also: New, well-looking defaults for Matplotlib (IMHO)
- → https://seaborn.pydata.org/
In [82]:
import seaborn as sns
sns.set()
In [83]:
df_demo[["A", "C"]].plot();
Seaborn Color Palette Example¶
In [84]:
sns.palplot(sns.color_palette())
In [85]:
sns.palplot(sns.color_palette("hls", 10))
In [86]:
sns.palplot(sns.color_palette("hsv", 20))
In [87]:
sns.palplot(sns.color_palette("Paired", 10))
In [129]:
sns.palplot(sns.color_palette("cubehelix", 8))
In [131]:
sns.palplot(sns.color_palette("colorblind", 10))
Seaborn Plot Examples¶
- Most of the time, I use a regression plot from Seaborn
In [89]:
with sns.color_palette("hls", 2):
sns.regplot(x="C", y="F", data=df_demo);
sns.regplot(x="C", y="G", data=df_demo);
- A joint plot combines two plots relating to distribution of values into one
- Very handy for showing a fuller picture of two-dimensionally scattered variables
In [90]:
x, y = np.random.multivariate_normal([0, 0], [[1, -.5], [-.5, 1]], size=300).T
In [91]:
sns.jointplot(x=x, y=y, kind="reg");
Task 6¶
- To your
dfNEST data frame, add a column with the unaccounted time (Unaccounted Time / s), which is the difference of program runtime, average neuron build time, minimal edge build time, minimal initialization time, presimulation time, and simulation time.
(I know this is technically not super correct, but it will do for our example.) - Plot a stacked bar plot of all these columns (except for program runtime) over the virtual processes
- Remember: pollev.com/aherten538
In [92]:
cols = [
'Avg. Neuron Build Time / s',
'Min. Edge Build Time / s',
'Min. Init. Time / s',
'Presim. Time / s',
'Sim. Time / s'
]
df["Unaccounted Time / s"] = df['Runtime Program / s']
for entry in cols:
df["Unaccounted Time / s"] = df["Unaccounted Time / s"] - df[entry]
In [93]:
df[["Runtime Program / s", "Unaccounted Time / s", *cols]].head(2)
Out[93]:
In [94]:
df[["Unaccounted Time / s", *cols]].plot(kind="bar", stacked=True, figsize=(12, 4));
- Make it relative to the total program run time
- Slight complication: Our virtual processes as indexes are not unique; we need to find new unique indexes
- Let's use a multi index!
In [95]:
df_multind = df.set_index(["Nodes", "Tasks/Node", "Threads/Task"])
df_multind.head()
Out[95]:
In [96]:
df_multind[["Unaccounted Time / s", *cols]]\
.divide(df_multind["Runtime Program / s"], axis="index")\
.plot(kind="bar", stacked=True, figsize=(14, 6), title="Relative Time Distribution");
In [97]:
df.groupby("Nodes").mean()
Out[97]:
Pivoting¶
- Combine categorically-similar columns
- Creates hierarchical index
- Respected during plotting!
- A pivot table has three layers; if confused, think about these questions
index: »What's on thexaxis?«values: »What value do I want to plot?«columns: »What categories do I want [to be in the legend]?«
- All can be populated from base data frame
- Might be aggregated, if needed
In [98]:
df_demo["H"] = [(-1)**n for n in range(5)]
In [99]:
df_pivot = df_demo.pivot_table(
index="F",
values="G",
columns="H"
)
df_pivot
Out[99]:
In [100]:
df_pivot.plot();
Task 7¶
- Create a pivot table based on the NEST
dfdata frame - Let the
xaxis show the number of nodes; display the values of the simulation time"Sim. Time / s"for the tasks per node and threas per task configurations - Please plot a bar plot
- Done? pollev.com/aherten538
In [101]:
df.pivot_table(
index=["Nodes"],
columns=["Tasks/Node", "Threads/Task"],
values="Sim. Time / s",
).plot(kind="bar", figsize=(12, 4));
The End¶
- Pandas works on data frames
- Slice frames to your likings
- Plot frames
- Together with Matplotlib, Seaborn, others
- Pivot tables are next level greatness
- Remember: Pandas as early as possible!
- Thanks for being here! 😍
Tell me what you think about this tutorial! a.herten@fz-juelich.de
Next slide: Further reading
Further Reading¶
- Pandas User Guide
- Matplotlib and LaTeX Plots
- towardsdatascience.com:
Poll Results¶
