I was recently interviewed on the Talk Python podcast about my background in analytics and how I’m applying it in the energy sector: Python in the Electrical Energy Sector
AEMO’s Quarterly Energy Dynamics (QED) report just came out today for Q1 2021. Of particular interest is the South Australian region which has been experiencing incredibly low prices thanks to the high levels of renewables in the market.
At the same time, these low prices have created system security challenges for the market operator. In fact over the past quarter, AEMO has had to intervene in the market a record 70% of the time to ensure that at least one gas powered generator remained running.
Back in 2017, I had the amazing opportunity to be one of 20 young Australians selected by Startup Catalyst to receive a scholarship for their Future Founders Mission to Silicon Valley. On this trip I had the opportunity to visit dozens of tech companies including Google, Facebook, Twitter, etc and learn from senior people within these organisations. At the time I was constantly jotting down little notes, and I thought I’d take some of them and write them up as a post now.
Topics
Ideas
- You’ll have more tries if you pick a market right. Really challenge yourself to imagine if there’s a lot of customers who will be willing to pay money. You can get things wrong but if you guess right about the market size then there will be acquisition options. Be realistic about growth and market size
- Customer discovery is important: make sure you meet a need and the market is big enough to support your business.
- Don’t get too emotionally attached to your idea/business and let it become your identity – being too attached can lead to you making bad decisions.
- For every yes you get, you’ll hear no at least 100 times. When getting started, people will reject you many times. You need to learn not to take it personally, just keep trying.
- Execution is important but so are exploration and discovery.
- If you just have one machine learning algorithm you don’t have much, people with more money or data can kill you. It’s how you use machine learning to make your product better, stickier etc or if you can build up more data than anyone else and sell that data. It’s more how you can use machine learning to make your product better than machine learning in itself
In July 2020, Melbourne entered its second COVID-19 lockdown – a lockdown which would not be lifted until late October. I thought it would be rather interesting to visualise the impact of this event on the number of PV system installations in each postcode of the city. These figures are based on the data provided by the Clean Energy Regulator which I merged with the spatial postcode data provided by the Australian Bureau of Statistics.
Pre-lockdown
Here’s how things looked in the months leading up to the lockdown – you can see how a few postcodes (in particular 3064 which appears to encompass Roxburgh Park) were having so many PV systems installed, I actually had to cap the maximum value of my heatmap at 300 so you could compare it to other postcodes in the figure.
I should mention up-front that the techniques described in this post are really only worthwhile once you have a dataset in the millions of rows or above. Once your data hits this size, it is worth paying the initial optimisation overhead as it will save you memory and be faster overall.
Pandas’ eval and query is built on Python’s Numexpr library, and provides an optimised way to run a calculation or filter on a Pandas dataframe. For example, the code below shows you the traditional way of doing these things in Pandas:
start = '2020-02-10 08:20:00'
end = '2020-02-10 08:30:00'
duids = ['LYA4', 'BW02']
# traditional vectorized calculation
map_gen_df['DIST'] = np.sqrt(lya1_df['SEC_DIFF'].pow(2) + lya1_df['VALUE_DIFF'].pow(2))
# traditional filter
event_duid_df = map_gen_df[(map_gen_df['MMSNAME'].isin(duids))&(map_gen_df['TIMESTAMP_MIN']>=start)&(map_gen_df['TIMESTAMP_MIN']<=end)]
Pandas and NumPy are fantastic libraries that enable you to take advantage of vectorization to write extremely efficient Python code. However, what happens when the calculation you wish to run changes based on the value in another column of your dataset?
For example, take a look at the dataset in the table below (along with the code to generate it):
| Group | Value |
| A | 1 |
| A | 1 |
| B | 1 |
| C | 1 |
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Group':['A','A','B','C'],
'Value':[1,1,1,1]
})Imagine I wish to create a third column (‘Result’) based on the following logic:
- Multiply Value by 2 if Group == ‘A’
- Multiply Value by 3 if Group == ‘B’
- Multiply Value by 4 if Group == ‘C’
- Fill with a missing value (nan) if none of the above is true
This article will walk you through the construction of an extremely simple dispatch engine programmed in GMPL. While my preferred language is Python, GLPK lets us focus exclusively on the equations behind the dispatch engine, enabling us to understand the basics of how dispatch works in the energy market.
Background
So what is a dispatch engine? It is a program that allows us to figure out the optimial volume of power that different power stations should dispatch into the grid. As power is instantly produced and consumed, the total volume of power generated must equal demand at any time. Given that power stations sell different volumes of power at varying prices, the purpose of a dispatch engine is to identify how much each power station should generate to meet demand, while minimizing the total cost to supply that power.
Another important detail is that all power stations are ultimately paid the same price for the power they generate within a given interval – the most expensive power station called on to dispatch in order to satisfy demand effectively becomes the ‘price setter’.
Our dispatch engine is based on the technique of linear optimization – you can find out more background on the technique here if you’re interested.
Installing
Install GLPK (GNU Linear Programming Kit) – you can do this with the following command once you have Anaconda installed:
conda install -c conda-forge glpkParameters
We’re going to start by setting out all the parameters for our optimization:
- We’re simulating a single region with a demand target of 150 MW
- We’ll have 3 power stations – 2 coal, and 1 gas
- Coal 1 sells power at $10/MWh and has a max output of 100 MW
- Coal 2 sells power at $20/MWh and has a max output of 100 MW
- Gas 1 sells power at $100/MWh and has a max output of 50 MW
Something interesting happened this week – on Tuesday at 1:20pm the dispatch price of electricity was $15,000/MWh in NSW and -$1,000 (yes, negative 1k) in Queensland at the exact same time. When you consider that these two states are next to each other and connected as part of a single grid known as the National Energy Market, this event may not appear to make much sense.
Understanding why these events occur is exactly why I got so interested in the energy market while I was working in consulting. A few of the things you have to take into account when analysing events like these include:
– How the interconnectors that connect the states work and how their capacity can be maxed out, leaving more expensive local generators to pick up the slack.
– The role that unexpected outages in the transmission lines can play
– The bidding behaviour of generators – solar farms will often bid in at negative prices to ensure that they are dispatched at any price they can get since their short run marginal cost (SRMC) is effectively 0
Purpose
I wrote this article for non-technical consultants at my firm to provide guidance on how to interact with a data scientist on a project and have confidence in their outputs.
The goal is to provide some advice for the kinds of checks you should be running to make sure no mistakes have crept into the analysis. I will be following up this post with an article I wrote which provides more of a general overview of the data science workflow for managers.
Checking data & outputs
While you should be doing spot and sense checks of the numbers you are given, the level of detail that you go into will come down to how much you trust the competency of your data scientist. Remember, ultimately you are the one who is going to have to stand in front of the client and stake your reputation on the numbers you present. A lot of tech/analytics people are often so focussed on finishing the analysis and getting you those numbers as quickly as possible that once they get a number they don’t always spend the extra time to check that that number makes sense. I have written a guide below for how to conduct the checks that could be done.
High Level Checks
Spot checks
- Pick a few cases (e.g. rows from your raw dataset) and make sure they end up matched with the correct data in the outputs
- As the consultant, you often have more domain expertise than your data scientist – try to look for edge cases in the data that can be used to test our assumptions about the analysis
Sense checks
- Do a ‘back of the envelope’ calculation to make sure that the numbers you’re given are within reason. If they are not then either:
- Something is wrong with the analysis
- Something is wrong with the raw data
- You’ve found something novel that may be of value
- If it is difficult to come up with a meaningful calculation, e.g. if and the data is too big to sum yourself in Excel, then you should ask the data scientist to do some simple sums of columns or groups in the data.
- Errors need to be traced back to their root cause (e.g. sometimes a major problem in the outputs can be a simple fix like correcting the data type of a column right back at the beginning of the analysis pipeline)
Granular Checks
Checks for individual datasets once they are read into Python/R
- Are all the expected columns/rows there? Keep track of those numbers.
- What data types are the columns (e.g. datetime, integer, decimal, etc.)? This is more important than you would think, and an easy way for subtle errors to creep into the analysis.
- Are there missing values in the places we expected? Are there more missing values than we expected?
- Are some rows or values duplicated in a way we did not expect?
- Are there unexpected unique values that could have crept into some columns?
Checks when combining datasets
- Frequently, combining multiple datasets will be a multistage process
- You want to make sure that the rows from your different raw datasets matches up to the same row in the outputs correctly
- Sometimes you may have a single row from one dataset match with multiple from another – ensure this matching process is correct.
- Do the number of rows and columns in your newly combined outputs make sense given the number of rows and columns in the raw data?
- Check that some rows which are meant to be missing after you merge datasets are actually gone
- Check that all the rows that are meant to remain are still there too
- Is it possible that the ID columns you’re combining data on may have missing values? You need to come up with a way to resolve this.
On Friday afternoon I attended a seminar by Professor John Christodoulou titled “Genomic Medicine: What is its place in the Australian healthcare system?”.
One of the interesting points he brought up was how genomics had changed the ability to diagnose many genetic diseases. In the past, diagnoses would often require a three step process:
- Clinical evaluation
- Biochemical screening
- Biopsies, imaging, etc
This process had a number of problems:
- It was prolonged and expensive
- Many biomarkers (especially for mitochondrial diseases) could be inaccurate. e.g. blood lactate levels
- The more definitive tests required advanced procedures (biopsies) which required anaesthetic and its associated risks
- An enzymatic diagnosis could still leave you ignorant of the genetic aetiology