Category:

Tips & Tutorials

Data science tip: store constants in their own file

by Jack Simpson July 7, 2021

written by Jack Simpson

Sometimes when you work on large projects, you can end up with code that looks like this:

if num_values < 15:
    do_thing()

Now, what does the 15 mean? Well, I’m sure you would have known when you first wrote the code, but what happens in 3 months when you try to modify it or pass it on to someone else in your team? Also, what happens if you determine that the number should actually be 16? Now you have to go through all your files and swap numbers in all the right places (and if you miss any you can introduce subtle bugs).

So what’s the solution? Well, what you really need is a single source of truth for your clearly defined constants, which you can:

Easily look up
Modify/update
Import into other code

Python makes this really easy – all you have to do is create a “constants.py” file in your project directory (I suppose you could call it whatever you like, but that’s beside the point). From here, you can import your variables as if you were importing a library:

from constants import NOM_FREQ_HZ, VALUES_IN_MIN

A screenshot of this file from a recent project looks a little like this:

This approach helped me write much more readable code, when working with an extremely complicated dataset where numbers were used to represent different value categories.

Obviously for smaller scripts it might not be worth setting up a dedicated file, but for this project, my constants file ended up being a couple of hundred lines long, and is far easier to maintain than if they were defined multiple times across all the files in the project.

July 7, 2021 0 comment

Content Energy Explanations Featured Tips & Tutorials

Summary of reliability forecasting in the NEM

by Jack Simpson July 6, 2021

written by Jack Simpson

I’ve written this article as a guide to the different reliability forecasts that AEMO undertakes in the National Electricity Market (NEM). These forecasts help AEMO identify supply shortfalls, and plan for the future. For example, AEMO’s annual Summer Readiness Plan draws on the outputs of most of the models described here.

This article is under active revision. My plan is to continue to tweak and expand on this content as I continue to learn more about these models. At the moment, this article covers the following topics:

LT PASA
MT PASA
ST PASA
EAAP

Long Term Projected Assessment of System Adequacy (LT PASA)

Purpose

Published annually looking out 10 years as part of the Electricity Statement of Opportunities (ESOO)
The forecast is used to identify reliability gaps in the first five years (which helps inform the RRO)
The final five years indicates whether there will be any future gaps in reliability
Informs the market on the need for new investment
Helps AEMO determine whether it needs to procure additional reserves

Inputs

Demand traces
- Two scenarios: Probability of Exceedance (PoE) 10 and 50 (expect to exceed every 10 or 2 years)
Weather traces – 8 historic weather patterns:Wind/solar generation traces
- Rooftop solar generation traces
- Generator ratings and forced outages
Demand-side participation (DSP)
Transmission constraint equations (e.g. transmission line thermal ratings)

Outputs

ESOO report flagging breaches to the reliability standard
The ESOO Plexos model and input traces is available for download from AEMO’s site

July 6, 2021 1 comment

Blogging Content Featured Tips & Tutorials

What I learned Visiting Silicon Valley

by Jack Simpson April 12, 2021

written by Jack Simpson

Back in 2017, I had the amazing opportunity to be one of 20 young Australians selected by Startup Catalyst to receive a scholarship for their Future Founders Mission to Silicon Valley. On this trip I had the opportunity to visit dozens of tech companies including Google, Facebook, Twitter, etc and learn from senior people within these organisations. At the time I was constantly jotting down little notes, and I thought I’d take some of them and write them up as a post now.

Ideas

You’ll have more tries if you pick a market right. Really challenge yourself to imagine if there’s a lot of customers who will be willing to pay money. You can get things wrong but if you guess right about the market size then there will be acquisition options. Be realistic about growth and market size
Customer discovery is important: make sure you meet a need and the market is big enough to support your business.
Don’t get too emotionally attached to your idea/business and let it become your identity – being too attached can lead to you making bad decisions.
For every yes you get, you’ll hear no at least 100 times. When getting started, people will reject you many times. You need to learn not to take it personally, just keep trying.
Execution is important but so are exploration and discovery.
If you just have one machine learning algorithm you don’t have much, people with more money or data can kill you. It’s how you use machine learning to make your product better, stickier etc or if you can build up more data than anyone else and sell that data. It’s more how you can use machine learning to make your product better than machine learning in itself

April 12, 2021 0 comment

Content Data Science Featured Programming Tips & Tutorials

How to improve the performance of Pandas with eval and query

by Jack Simpson April 5, 2021

written by Jack Simpson

I should mention up-front that the techniques described in this post are really only worthwhile once you have a dataset in the millions of rows or above. Once your data hits this size, it is worth paying the initial optimisation overhead as it will save you memory and be faster overall.

Pandas’ eval and query is built on Python’s Numexpr library, and provides an optimised way to run a calculation or filter on a Pandas dataframe. For example, the code below shows you the traditional way of doing these things in Pandas:

start = '2020-02-10 08:20:00'
end = '2020-02-10 08:30:00'
duids = ['LYA4', 'BW02']

# traditional vectorized calculation
map_gen_df['DIST'] = np.sqrt(lya1_df['SEC_DIFF'].pow(2) + lya1_df['VALUE_DIFF'].pow(2))

# traditional filter
event_duid_df = map_gen_df[(map_gen_df['MMSNAME'].isin(duids))&(map_gen_df['TIMESTAMP_MIN']>=start)&(map_gen_df['TIMESTAMP_MIN']<=end)]

April 5, 2021 0 comment

Content Data Science Featured Programming Tips & Tutorials

How to vectorize conditional calculations in Python

by Jack Simpson April 4, 2021

written by Jack Simpson

Pandas and NumPy are fantastic libraries that enable you to take advantage of vectorization to write extremely efficient Python code. However, what happens when the calculation you wish to run changes based on the value in another column of your dataset?

For example, take a look at the dataset in the table below (along with the code to generate it):

Group	Value
A	1
A	1
B	1
C	1

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Group':['A','A','B','C'],
    'Value':[1,1,1,1]
})

Imagine I wish to create a third column (‘Result’) based on the following logic:

Multiply Value by 2 if Group == ‘A’
Multiply Value by 3 if Group == ‘B’
Multiply Value by 4 if Group == ‘C’
Fill with a missing value (nan) if none of the above is true

April 4, 2021 0 comment

Content Energy Explanations Featured Tips & Tutorials

How to build a simple energy market dispatch engine

by Jack Simpson March 29, 2021

written by Jack Simpson

This article will walk you through the construction of an extremely simple dispatch engine programmed in GMPL. While my preferred language is Python, GLPK lets us focus exclusively on the equations behind the dispatch engine, enabling us to understand the basics of how dispatch works in the energy market.

Background

So what is a dispatch engine? It is a program that allows us to figure out the optimial volume of power that different power stations should dispatch into the grid. As power is instantly produced and consumed, the total volume of power generated must equal demand at any time. Given that power stations sell different volumes of power at varying prices, the purpose of a dispatch engine is to identify how much each power station should generate to meet demand, while minimizing the total cost to supply that power.

Another important detail is that all power stations are ultimately paid the same price for the power they generate within a given interval – the most expensive power station called on to dispatch in order to satisfy demand effectively becomes the ‘price setter’.

Our dispatch engine is based on the technique of linear optimization – you can find out more background on the technique here if you’re interested.

Installing

Install GLPK (GNU Linear Programming Kit) – you can do this with the following command once you have Anaconda installed:

conda install -c conda-forge glpk

Parameters

We’re going to start by setting out all the parameters for our optimization:

We’re simulating a single region with a demand target of 150 MW
We’ll have 3 power stations – 2 coal, and 1 gas
- Coal 1 sells power at $10/MWh and has a max output of 100 MW
- Coal 2 sells power at $20/MWh and has a max output of 100 MW
- Gas 1 sells power at $100/MWh and has a max output of 50 MW

March 29, 2021 0 comment

Content Data Science Featured Tips & Tutorials

How to verify the analysis of your data scientist

by Jack Simpson November 16, 2019

written by Jack Simpson

Purpose

I wrote this article for non-technical consultants at my firm to provide guidance on how to interact with a data scientist on a project and have confidence in their outputs.

The goal is to provide some advice for the kinds of checks you should be running to make sure no mistakes have crept into the analysis. I will be following up this post with an article I wrote which provides more of a general overview of the data science workflow for managers.

Checking data & outputs

While you should be doing spot and sense checks of the numbers you are given, the level of detail that you go into will come down to how much you trust the competency of your data scientist. Remember, ultimately you are the one who is going to have to stand in front of the client and stake your reputation on the numbers you present. A lot of tech/analytics people are often so focussed on finishing the analysis and getting you those numbers as quickly as possible that once they get a number they don’t always spend the extra time to check that that number makes sense. I have written a guide below for how to conduct the checks that could be done.

High Level Checks

Spot checks

Pick a few cases (e.g. rows from your raw dataset) and make sure they end up matched with the correct data in the outputs
As the consultant, you often have more domain expertise than your data scientist – try to look for edge cases in the data that can be used to test our assumptions about the analysis

Sense checks

Do a ‘back of the envelope’ calculation to make sure that the numbers you’re given are within reason. If they are not then either:
- Something is wrong with the analysis
- Something is wrong with the raw data
- You’ve found something novel that may be of value
If it is difficult to come up with a meaningful calculation, e.g. if and the data is too big to sum yourself in Excel, then you should ask the data scientist to do some simple sums of columns or groups in the data.
Errors need to be traced back to their root cause (e.g. sometimes a major problem in the outputs can be a simple fix like correcting the data type of a column right back at the beginning of the analysis pipeline)

Granular Checks

Checks for individual datasets once they are read into Python/R

Are all the expected columns/rows there? Keep track of those numbers.
What data types are the columns (e.g. datetime, integer, decimal, etc.)? This is more important than you would think, and an easy way for subtle errors to creep into the analysis.
Are there missing values in the places we expected? Are there more missing values than we expected?
Are some rows or values duplicated in a way we did not expect?
Are there unexpected unique values that could have crept into some columns?

Checks when combining datasets

Frequently, combining multiple datasets will be a multistage process
You want to make sure that the rows from your different raw datasets matches up to the same row in the outputs correctly
Sometimes you may have a single row from one dataset match with multiple from another – ensure this matching process is correct.
Do the number of rows and columns in your newly combined outputs make sense given the number of rows and columns in the raw data?
Check that some rows which are meant to be missing after you merge datasets are actually gone
Check that all the rows that are meant to remain are still there too
Is it possible that the ID columns you’re combining data on may have missing values? You need to come up with a way to resolve this.

November 16, 2019 0 comment

Content Data Science Tips & Tutorials

Best practices for data science with the Jupyter Notebook

by Jack Simpson December 18, 2016

written by Jack Simpson

I recently listened to a really interesting talk by Jonathan Whitmore where he discussed the approach his company has to working with data using the Jupyter Notebook. I’d recommend watching it, but I’ve made a brief summary below for my own future reference.

December 18, 2016 0 comment

Content Data Science Machine Learning Tips & Tutorials

Working with Imbalanced Classes

by Jack Simpson December 11, 2016

written by Jack Simpson

I wrote a few quick bullet points down from the article “8 Tactics to Combat Imbalanced Classes in Your Machine Learning Dataset” for future reference.

Tactics

Imbalanced datasets occur when you have a class that occurs much more infrequently than the others.
If a model ignores the class, it can still achieve a high classification accuracy, but that’s not the result we want
Make sure you use a confusion matrix to ensure that you’re getting acceptable accuracy for all your classes
One potential solution to this problem is to collect more data
Resampling your dataset (can be random or non-random – stratified):
- Add copies of underrepresented classes (oversampling/sampling with replacement). Useful if you don’t have much data – 10s of thousands or less.
- Delete instances of classes that occur frequently (undersampling). Handy to use if you have a lot of data – 10s-100s of thousands of instances
Try different algorithms: decision trees can perform well on imbalanced datasets
Penalised models: Extra penalties for misclassifying minority class. Examples of these algorithms could include penalized-SVM and penalized-LDA.
There are areas of research dedicated to imbalanced datasets: can look into anomaly detection and change detection.

December 11, 2016 0 comment

Content Data Science Machine Learning Tips & Tutorials

Assessing machine learning algorithm performance

by Jack Simpson December 11, 2016

written by Jack Simpson

I wrote a few quick bullet points down from the article “How To Implement Machine Learning Algorithm Performance Metrics From Scratch With Python“.

Metrics

Classification accuracy
- Test how well predictions of a model do overall
- accuracy = correct predictions / total predictions
Confusion matrix
- Use to identify how well your predictions did with different classes
- Very useful if you have an imbalanced dataset
- I wrote an extremely hacked together confusion matrix for my tag identification software. I had 4 classes (U, C, R, Q) and the confusion matrix shows you what your model predicted against what the real category was.

	U	C	R	Q
U	175	17	67	1
C	11	335	14	0
R	26	8	298	0
Q	6	0	3	93

Mean absolute error for regression
- Positive values – the average of how much your predicted value differ from the real value
Root mean squared error for regression
- Square root of the mean of squared differences between the actual and predicted value
- Squaring the values gives you positive numbers and finding the root lets you compare the values to the original units.

December 11, 2016 0 comment

Newer Posts

Older Posts