Welcome New Members! We want to hear from you. Register, stop lurking and start posting!

Vic weather - How accurate are the models and Met agencies

All general weather related discussion & questions, including model discussions, longer-term outlooks & anything non-breaking weather.
Post Reply
J
JasmineStorm
Supercell
Reactions:
Posts: 1870
Joined: Thu Sep 22, 2016 9:40 pm
Location: Kyneton 527 ASL

Vic weather - How accurate are the models and Met agencies

Post by JasmineStorm »

Summary:

Commencing March 12th and finishing on April 1st, I collected 21 days of forecast and actual data associated with 3 weather agencies, their associated computer models for 4 Victorian locations. The purpose of this exercise was to find out how accurate their short term next day forecast was for maximum temperature and rainfall. It was to also find out which agency and model was the most accurate. The timing of the experiment was chosen based on data either side of the equinox, to ensure a variety of weather systems would be included into the data.

Agencies and of the Models:
BoM – MetEye 5pm daily
EC - European centre for medium range weather forecast model run 00z
GFS - Global Forecast System model 00z

Locations:
2 sea level locations either side of Port Philip bay, 1 elevated location south of divide, 1 elevated location north of the divide.
Melbourne, Geelong, Ferny creek and Kyneton were selected.

Sources:
Forecast - BoM, Windy TV, BSCH, YR, Pivotal weather, Ventusky
Actual - BoM and my PWS Kyneton

Criteria:
I used a Pass and Fail criteria for the outcome to be a percentage based accuracy, based on my own personal threshold for the forecast.

Personal Threshold:

Temp – Max forecast temperature is above or below by 1.6c degrees of actual recorded temperature would be regarded as a Fail.

Rain –
a/ If no rain was forecast and rainfall occurs, it is a fail.
b/ If rain is forecast but does not occur, it is a fail.
c/ Rain outside of 5mm of forecast is a fail.

Results:

Melbourne maximum temperature accuracy forecast
1st – BOM 67%
2nd – EC 52%
3rd – GFS 48%

Melbourne rainfall accuracy forecast
1st – EC 55%
2nd – GFS 45%
3rd – BoM 36%


General comments – BoM was a clear winner on temperature forecasting for Melbourne but was a clear loser on rainfall accuracy. EC was more consistent overall.

Geelong maximum temperature accuracy forecast
1st – BOM 52%
2nd – GFS 52%
3rd – EC 33%


Geelong rainfall accuracy forecast
1st – EC 58%
2nd – GFS 50%
3rd – BoM 50%

General comments – EC was poor on temperature with rainfall fairly close between them all.

Ferny creek maximum temperature accuracy forecast
1st – EC 62%
2nd – BoM 62%
3rd – GFS 38%


Ferny creek rainfall accuracy forecast
1st – GFS 58%
2nd – EC 33%
3rd – BoM 33%

General comments – EC and the BoM were clear winners in temperature but when it came to rainfall they were poor with GFS a clear winner.


Kyneton maximum temperature accuracy forecast
1st – BoM 48%
2nd – GFS 38%
3rd – EC 19%

Kyneton rainfall accuracy forecast
1st – BoM 90%
2nd – EC 90%
3rd – GFS 60%

General comments – BoM was a clear winner with temperature with EC extremely poor on temperature. BoM and EC were excellent on rainfall accuracy.
G
Gordon
Supercell
Reactions:
Posts: 2877
Joined: Thu Jun 17, 2010 10:01 am
Location: Near Gordon, Vic. 620 m asl

Re: Vic weather - How accurate are the models and Met agencies

Post by Gordon »

Nice work JS. The sample size isn't huge but very interesting reading nonetheless. I've meant to do the same for our place but haven't got around to it yet...

One thing your comparison raises is whether some Vic locations have more 'predictable' weather overall than others?
J
JasmineStorm
Supercell
Reactions:
Posts: 1870
Joined: Thu Sep 22, 2016 9:40 pm
Location: Kyneton 527 ASL

Re: Vic weather - How accurate are the models and Met agencies

Post by JasmineStorm »

Gordon wrote: Mon Apr 03, 2017 11:39 am Nice work JS. The sample size isn't huge but very interesting reading nonetheless. I've meant to do the same for our place but haven't got around to it yet...

One thing your comparison raises is whether some Vic locations have more 'predictable' weather overall than others?
Yes Gordon, temperature wise both Melbourne and Geelong seemed more predicable than the elevated spots but I was quite surprised how the BoM struggled with rainfall accuracy on 3 out of the 4 locations. Before the study, I assumed the BoM would of been a clear winner in both categories but that certainly wasn't the case, especially on rainfall. The one thing I was confident on was EC struggled on max temperatures for Kyneton, which proved to be correct.
User avatar
Australis(Shell3155)
Supercell
Reactions:
Posts: 3142
Joined: Mon Nov 30, 2009 8:05 pm
Location: FTG
Contact:

Re: Vic weather - How accurate are the models and Met agencies

Post by Australis(Shell3155) »

Not sure of which thread to pop this one into..

Post Reply