Home » Content » Data Science
Category:

Data Science

ContentData ScienceFeaturedProgrammingTips & Tutorials

Data science tip: store constants in their own file

by Jack Simpson
written by Jack Simpson

Sometimes when you work on large projects, you can end up with code that looks like this:

if num_values < 15:
    do_thing()

Now, what does the 15 mean? Well, I’m sure you would have known when you first wrote the code, but what happens in 3 months when you try to modify it or pass it on to someone else in your team? Also, what happens if you determine that the number should actually be 16? Now you have to go through all your files and swap numbers in all the right places (and if you miss any you can introduce subtle bugs).

So what’s the solution? Well, what you really need is a single source of truth for your clearly defined constants, which you can:

  • Easily look up
  • Modify/update
  • Import into other code

Python makes this really easy – all you have to do is create a “constants.py” file in your project directory (I suppose you could call it whatever you like, but that’s beside the point). From here, you can import your variables as if you were importing a library:

from constants import NOM_FREQ_HZ, VALUES_IN_MIN

A screenshot of this file from a recent project looks a little like this:

constants code file

This approach helped me write much more readable code, when working with an extremely complicated dataset where numbers were used to represent different value categories.

Obviously for smaller scripts it might not be worth setting up a dedicated file, but for this project, my constants file ended up being a couple of hundred lines long, and is far easier to maintain than if they were defined multiple times across all the files in the project.

0 comment
0 FacebookTwitterPinterestEmail
ContentData ScienceEnergyLinksMachine LearningResearch

Excellent seminar on the applications of machine learning in the energy sector

by Jack Simpson
written by Jack Simpson

If you’ve ever wanted to see the impact that machine learning is having in the energy sector, then I recommend watching this seminar released by the National Renewable Energy Laboratory (NREL).

Each talk describes an application of machine learning in the industry at different levels, from the big (weather and climate modelling) through to the small (optimising the aerodynamics of turbine blades).

Some of the topics discussed include:

  • How researchers at NREL are using generative adversarial networks (GANs) to assist them with weather and climate modelling
  • How you can represent a wind farm as a graph neural network (GNN) with directed edges (this is brilliant!)
  • How hard it is to acquire enough data to train models for wind farms (this is why they mention having success with ensemble-based modelling approaches)
  • How they’ve been creating simulations to augment their wind farm datasets
  • A few key points which I agree with from personal experience
    • Features matter more than models – having enough input data, processed in the right way often matters more than the specific machine learning algorithm you’re using
    • Training models is expensive and time consuming, but once that stage is done, you can run them cheaply and quickly in production
Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
julia programming
ContentData ScienceLinksProgrammingResources

Learning to program in Julia (for energy modelling)

by Jack Simpson
written by Jack Simpson

Most people know that I’m a huge fan of the Python programming language – while that isn’t going to change, a recent encounter with some researchers at CSIRO has convinced me that I should pick up Julia for some of the energy modelling and optimisation work that I do.

I’ve known for a while that Julia was a language with a lot of benefits (as fast as a lower-level language but with the productivity benefits of a higher-level language). However, if you understand how to write efficient vectorised code in Python (using NumPy and Pandas), then except for some use-cases, you don’t really get that much of a boost out of switching to Julia.

So what changed my mind? Well, for the past few years, the National Renewable Energy Laboratory (NREL) have been working on a number of amazing open-source energy modelling packages for the Julia programming language. I’ve now updated my electricity modelling resources page with the links to some material about these packages.

Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
ContentData ScienceFeaturedLinksMachine LearningResourcesStatistics

Simple explanations of data science and statistics concepts

by Jack Simpson
written by Jack Simpson

One of my favourite data science resources is the mini-episode series of the Data Skeptic podcast. These short episodes would feature the host explaining a data science concept to a non-expert in plain English.

I wanted to share a few of these with some colleagues from work and thought I’d catalogue them here.

Continue Reading
1 comment
0 FacebookTwitterPinterestEmail
Pandas Query
ContentData ScienceFeaturedProgrammingTips & Tutorials

How to improve the performance of Pandas with eval and query

by Jack Simpson
written by Jack Simpson

I should mention up-front that the techniques described in this post are really only worthwhile once you have a dataset in the millions of rows or above. Once your data hits this size, it is worth paying the initial optimisation overhead as it will save you memory and be faster overall.

Pandas’ eval and query is built on Python’s Numexpr library, and provides an optimised way to run a calculation or filter on a Pandas dataframe. For example, the code below shows you the traditional way of doing these things in Pandas:

start = '2020-02-10 08:20:00'
end = '2020-02-10 08:30:00'
duids = ['LYA4', 'BW02']

# traditional vectorized calculation
map_gen_df['DIST'] = np.sqrt(lya1_df['SEC_DIFF'].pow(2) + lya1_df['VALUE_DIFF'].pow(2))

# traditional filter
event_duid_df = map_gen_df[(map_gen_df['MMSNAME'].isin(duids))&(map_gen_df['TIMESTAMP_MIN']>=start)&(map_gen_df['TIMESTAMP_MIN']<=end)]
Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Conditional Code
ContentData ScienceFeaturedProgrammingTips & Tutorials

How to vectorize conditional calculations in Python

by Jack Simpson
written by Jack Simpson

Pandas and NumPy are fantastic libraries that enable you to take advantage of vectorization to write extremely efficient Python code. However, what happens when the calculation you wish to run changes based on the value in another column of your dataset?

For example, take a look at the dataset in the table below (along with the code to generate it):

GroupValue
A1
A1
B1
C1
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Group':['A','A','B','C'],
    'Value':[1,1,1,1]
})

Imagine I wish to create a third column (‘Result’) based on the following logic:

  • Multiply Value by 2 if Group == ‘A’
  • Multiply Value by 3 if Group == ‘B’
  • Multiply Value by 4 if Group == ‘C’
  • Fill with a missing value (nan) if none of the above is true
Continue Reading
0 comment
1 FacebookTwitterPinterestEmail
Checking Analysis
ContentData ScienceFeaturedTips & Tutorials

How to verify the analysis of your data scientist

by Jack Simpson
written by Jack Simpson

Purpose

I wrote this article for non-technical consultants at my firm to provide guidance on how to interact with a data scientist on a project and have confidence in their outputs.

The goal is to provide some advice for the kinds of checks you should be running to make sure no mistakes have crept into the analysis. I will be following up this post with an article I wrote which provides more of a general overview of the data science workflow for managers.

Checking data & outputs

While you should be doing spot and sense checks of the numbers you are given, the level of detail that you go into will come down to how much you trust the competency of your data scientist. Remember, ultimately you are the one who is going to have to stand in front of the client and stake your reputation on the numbers you present. A lot of tech/analytics people are often so focussed on finishing the analysis and getting you those numbers as quickly as possible that once they get a number they don’t always spend the extra time to check that that number makes sense. I have written a guide below for how to conduct the checks that could be done.

High Level Checks

Spot checks

  1. Pick a few cases (e.g. rows from your raw dataset) and make sure they end up matched with the correct data in the outputs
  2. As the consultant, you often have more domain expertise than your data scientist – try to look for edge cases in the data that can be used to test our assumptions about the analysis

Sense checks

  1. Do a ‘back of the envelope’ calculation to make sure that the numbers you’re given are within reason. If they are not then either:
    • Something is wrong with the analysis
    • Something is wrong with the raw data
    • You’ve found something novel that may be of value
  2. If it is difficult to come up with a meaningful calculation, e.g. if and the data is too big to sum yourself in Excel, then you should ask the data scientist to do some simple sums of columns or groups in the data. 
  3. Errors need to be traced back to their root cause (e.g. sometimes a major problem in the outputs can be a simple fix like correcting the data type of a column right back at the beginning of the analysis pipeline)

Granular Checks

Checks for individual datasets once they are read into Python/R

  1. Are all the expected columns/rows there? Keep track of those numbers.
  2. What data types are the columns (e.g. datetime, integer, decimal, etc.)? This is more important than you would think, and an easy way for subtle errors to creep into the analysis.
  3. Are there missing values in the places we expected? Are there more missing values than we expected?
  4. Are some rows or values duplicated in a way we did not expect?
  5. Are there unexpected unique values that could have crept into some columns?

Checks when combining datasets

  1. Frequently, combining multiple datasets will be a multistage process
  2. You want to make sure that the rows from your different raw datasets matches up to the same row in the outputs correctly
  3. Sometimes you may have a single row from one dataset match with multiple from another – ensure this matching process is correct.
  4. Do the number of rows and columns in your newly combined outputs make sense given the number of rows and columns in the raw data?
  5. Check that some rows which are meant to be missing after you merge datasets are actually gone
  6. Check that all the rows that are meant to remain are still there too
  7. Is it possible that the ID columns you’re combining data on may have missing values? You need to come up with a way to resolve this.
0 comment
1 FacebookTwitterPinterestEmail
BloggingContentData ScienceResearch

Honeybees and missing data part 3: Ant search optimisation

by Jack Simpson
written by Jack Simpson

Earlier in the year I had to reset the observation hive and sadly that process resulted in the death of a fair few bees who ended up on the ground. Now, I knew that there were large carnivorous bull ants in the area and I completely expected them to clear away the bodies within a few hours. Unexpectedly though, none of the bodies were removed.

After a couple of days, I took a closer look and noticed something interesting – the bull ants were missing, but the tiny ants* in the area had found them! I’ve circled a few of them in the image below.

Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
BloggingContentData ScienceResearch

Honeybees and missing data part 2: Where do bees like to live?

by Jack Simpson
written by Jack Simpson

This article relates to where bees prefer to build their hives. Back in the 1970s, there was a bunch of research conducted at Cornell surveying where bees build hives in the wild, and all the evidence seemed to indicate that honeybees preferred to build their hives relatively close to the ground. This finding seemed rather odd, since logically a hive built higher up would be better protected from predators.

Eventually they figured out that the answer was contained in the missing data. What had happened is that the hives which were closer to the ground were far easier to spot by the researchers! Bees do actually prefer to build a hive as high up as possible (and will do so in the majority of cases). In this case, the missing data didn’t just give the scientists more details, it changed the story entirely!

I thought this story was a great lesson for how researchers and organisations have to be careful with how they collect their data, and which data they base their decisions on, as collecting data just because it is easier to get may not just be a waste of money, it may lead you in entirely the wrong direction.

Part 1 is available here.

1 comment
0 FacebookTwitterPinterestEmail
BloggingContentData ScienceResearch

​Honeybees and missing data part 1: Tale of the missing bees

by Jack Simpson
written by Jack Simpson

​Sometimes missing data can tell you its own story, while other times missing data can lead you in completely the wrong direction. As I’ve been working on a PhD studying honeybee behaviour, I thought I’d share a couple of articles about bees and how important missing data can be.

This first article is the story of how I was tearing my hair out in frustration trying to get tags to stop coming off my bees when something else entirely was going on.

Continue Reading
1 comment
0 FacebookTwitterPinterestEmail
Newer Posts
Older Posts