Home » Content » Featured » Page 2
Category:

Featured

ContentEnergyFeaturedVisualisations

Primary frequency response in action!

by Jack Simpson
written by Jack Simpson

On the 25th of May, an explosion and fire at Callide Power Station initiated a series of events that culminated in blackouts affecting hundreds of thousands of Queenslanders.

The figure below shows the coal power station units at Callide, Stanwell, and Gladstone tripping as this event unfolded between 1:45-2:10pm.

Power station units are in MW (left Y axis) while the system frequency units are in Hz (right Y axis).

However, when I examined the two Gladstone units that did not trip, I saw something rather interesting. As you can see in the figures below, in response to a sharp drop in system frequency, both units momentarily boosted their output.

Continue Reading
0 comment
1 FacebookTwitterPinterestEmail
PV
ContentEnergyFeaturedLinks

South Australian Gas Generator Interventions

by Jack Simpson
written by Jack Simpson

AEMO’s Quarterly Energy Dynamics (QED) report just came out today for Q1 2021. Of particular interest is the South Australian region which has been experiencing incredibly low prices thanks to the high levels of renewables in the market.

At the same time, these low prices have created system security challenges for the market operator. In fact over the past quarter, AEMO has had to intervene in the market a record 70% of the time to ensure that at least one gas powered generator remained running.

Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Startup Catalyst Collage
BloggingContentFeaturedTips & Tutorials

What I learned Visiting Silicon Valley

by Jack Simpson
written by Jack Simpson

Back in 2017, I had the amazing opportunity to be one of 20 young Australians selected by Startup Catalyst to receive a scholarship for their Future Founders Mission to Silicon Valley. On this trip I had the opportunity to visit dozens of tech companies including Google, Facebook, Twitter, etc and learn from senior people within these organisations. At the time I was constantly jotting down little notes, and I thought I’d take some of them and write them up as a post now.

Topics

Ideas

  • You’ll have more tries if you pick a market right. Really challenge yourself to imagine if there’s a lot of customers who will be willing to pay money. You can get things wrong but if you guess right about the market size then there will be acquisition options. Be realistic about growth and market size
  • Customer discovery is important: make sure you meet a need and the market is big enough to support your business.
  • Don’t get too emotionally attached to your idea/business and let it become your identity – being too attached can lead to you making bad decisions.
  • For every yes you get, you’ll hear no at least 100 times. When getting started, people will reject you many times. You need to learn not to take it personally, just keep trying.
  • Execution is important but so are exploration and discovery.
  • If you just have one machine learning algorithm you don’t have much, people with more money or data can kill you. It’s how you use machine learning to make your product better, stickier etc or if you can build up more data than anyone else and sell that data. It’s more how you can use machine learning to make your product better than machine learning in itself
Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
ContentEnergyFeaturedVisualisations

Visualising the impact of Melbourne’s COVID-19 lockdown on Solar Panel Installations

by Jack Simpson
written by Jack Simpson

In July 2020, Melbourne entered its second COVID-19 lockdown – a lockdown which would not be lifted until late October. I thought it would be rather interesting to visualise the impact of this event on the number of PV system installations in each postcode of the city. These figures are based on the data provided by the Clean Energy Regulator which I merged with the spatial postcode data provided by the Australian Bureau of Statistics.

Pre-lockdown

Here’s how things looked in the months leading up to the lockdown – you can see how a few postcodes (in particular 3064 which appears to encompass Roxburgh Park) were having so many PV systems installed, I actually had to cap the maximum value of my heatmap at 300 so you could compare it to other postcodes in the figure.

Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Pandas Query
ContentData ScienceFeaturedProgrammingTips & Tutorials

How to improve the performance of Pandas with eval and query

by Jack Simpson
written by Jack Simpson

I should mention up-front that the techniques described in this post are really only worthwhile once you have a dataset in the millions of rows or above. Once your data hits this size, it is worth paying the initial optimisation overhead as it will save you memory and be faster overall.

Pandas’ eval and query is built on Python’s Numexpr library, and provides an optimised way to run a calculation or filter on a Pandas dataframe. For example, the code below shows you the traditional way of doing these things in Pandas:

start = '2020-02-10 08:20:00'
end = '2020-02-10 08:30:00'
duids = ['LYA4', 'BW02']

# traditional vectorized calculation
map_gen_df['DIST'] = np.sqrt(lya1_df['SEC_DIFF'].pow(2) + lya1_df['VALUE_DIFF'].pow(2))

# traditional filter
event_duid_df = map_gen_df[(map_gen_df['MMSNAME'].isin(duids))&(map_gen_df['TIMESTAMP_MIN']>=start)&(map_gen_df['TIMESTAMP_MIN']<=end)]
Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Conditional Code
ContentData ScienceFeaturedProgrammingTips & Tutorials

How to vectorize conditional calculations in Python

by Jack Simpson
written by Jack Simpson

Pandas and NumPy are fantastic libraries that enable you to take advantage of vectorization to write extremely efficient Python code. However, what happens when the calculation you wish to run changes based on the value in another column of your dataset?

For example, take a look at the dataset in the table below (along with the code to generate it):

GroupValue
A1
A1
B1
C1
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Group':['A','A','B','C'],
    'Value':[1,1,1,1]
})

Imagine I wish to create a third column (‘Result’) based on the following logic:

  • Multiply Value by 2 if Group == ‘A’
  • Multiply Value by 3 if Group == ‘B’
  • Multiply Value by 4 if Group == ‘C’
  • Fill with a missing value (nan) if none of the above is true
Continue Reading
0 comment
1 FacebookTwitterPinterestEmail
Power Lines
ContentEnergyExplanationsFeaturedTips & Tutorials

How to build a simple energy market dispatch engine

by Jack Simpson
written by Jack Simpson

This article will walk you through the construction of an extremely simple dispatch engine programmed in GMPL. While my preferred language is Python, GLPK lets us focus exclusively on the equations behind the dispatch engine, enabling us to understand the basics of how dispatch works in the energy market.

Background

So what is a dispatch engine? It is a program that allows us to figure out the optimial volume of power that different power stations should dispatch into the grid. As power is instantly produced and consumed, the total volume of power generated must equal demand at any time. Given that power stations sell different volumes of power at varying prices, the purpose of a dispatch engine is to identify how much each power station should generate to meet demand, while minimizing the total cost to supply that power.

Another important detail is that all power stations are ultimately paid the same price for the power they generate within a given interval – the most expensive power station called on to dispatch in order to satisfy demand effectively becomes the ‘price setter’.

Our dispatch engine is based on the technique of linear optimization – you can find out more background on the technique here if you’re interested.

Installing

Install GLPK (GNU Linear Programming Kit) – you can do this with the following command once you have Anaconda installed:

conda install -c conda-forge glpk

Parameters

We’re going to start by setting out all the parameters for our optimization:

  • We’re simulating a single region with a demand target of 150 MW
  • We’ll have 3 power stations – 2 coal, and 1 gas
    • Coal 1 sells power at $10/MWh and has a max output of 100 MW
    • Coal 2 sells power at $20/MWh and has a max output of 100 MW
    • Gas 1 sells power at $100/MWh and has a max output of 50 MW
Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Checking Analysis
ContentData ScienceFeaturedTips & Tutorials

How to verify the analysis of your data scientist

by Jack Simpson
written by Jack Simpson

Purpose

I wrote this article for non-technical consultants at my firm to provide guidance on how to interact with a data scientist on a project and have confidence in their outputs.

The goal is to provide some advice for the kinds of checks you should be running to make sure no mistakes have crept into the analysis. I will be following up this post with an article I wrote which provides more of a general overview of the data science workflow for managers.

Checking data & outputs

While you should be doing spot and sense checks of the numbers you are given, the level of detail that you go into will come down to how much you trust the competency of your data scientist. Remember, ultimately you are the one who is going to have to stand in front of the client and stake your reputation on the numbers you present. A lot of tech/analytics people are often so focussed on finishing the analysis and getting you those numbers as quickly as possible that once they get a number they don’t always spend the extra time to check that that number makes sense. I have written a guide below for how to conduct the checks that could be done.

High Level Checks

Spot checks

  1. Pick a few cases (e.g. rows from your raw dataset) and make sure they end up matched with the correct data in the outputs
  2. As the consultant, you often have more domain expertise than your data scientist – try to look for edge cases in the data that can be used to test our assumptions about the analysis

Sense checks

  1. Do a ‘back of the envelope’ calculation to make sure that the numbers you’re given are within reason. If they are not then either:
    • Something is wrong with the analysis
    • Something is wrong with the raw data
    • You’ve found something novel that may be of value
  2. If it is difficult to come up with a meaningful calculation, e.g. if and the data is too big to sum yourself in Excel, then you should ask the data scientist to do some simple sums of columns or groups in the data. 
  3. Errors need to be traced back to their root cause (e.g. sometimes a major problem in the outputs can be a simple fix like correcting the data type of a column right back at the beginning of the analysis pipeline)

Granular Checks

Checks for individual datasets once they are read into Python/R

  1. Are all the expected columns/rows there? Keep track of those numbers.
  2. What data types are the columns (e.g. datetime, integer, decimal, etc.)? This is more important than you would think, and an easy way for subtle errors to creep into the analysis.
  3. Are there missing values in the places we expected? Are there more missing values than we expected?
  4. Are some rows or values duplicated in a way we did not expect?
  5. Are there unexpected unique values that could have crept into some columns?

Checks when combining datasets

  1. Frequently, combining multiple datasets will be a multistage process
  2. You want to make sure that the rows from your different raw datasets matches up to the same row in the outputs correctly
  3. Sometimes you may have a single row from one dataset match with multiple from another – ensure this matching process is correct.
  4. Do the number of rows and columns in your newly combined outputs make sense given the number of rows and columns in the raw data?
  5. Check that some rows which are meant to be missing after you merge datasets are actually gone
  6. Check that all the rows that are meant to remain are still there too
  7. Is it possible that the ID columns you’re combining data on may have missing values? You need to come up with a way to resolve this.
0 comment
1 FacebookTwitterPinterestEmail
ContentData ScienceFeaturedMachine LearningResearch

Bees, lasers, and machine learning

by Jack Simpson
written by Jack Simpson

A couple of years ago I started my PhD at the Australian National University working to quantify honeybee behaviour. We wanted to build a system that could automatically track and compare different groups of bees within the hive.

I took the project as I had a background in biology, beekeeping and programming, and I wanted to work in a lab where I could learn from a supervisor who was incredibly knowledgeable about both biology and software development.

Continue Reading
0 comment
0 FacebookTwitterPinterestEmail
Newer Posts
Older Posts