Vic weather - How accurate are the models and Met agencies

[JasmineStorm]

Summary:

Commencing March 12th and finishing on April 1st, I collected 21 days of forecast and actual data associated with 3 weather agencies, their associated computer models for 4 Victorian locations. The purpose of this exercise was to find out how accurate their short term next day forecast was for maximum temperature and rainfall. It was to also find out which agency and model was the most accurate. The timing of the experiment was chosen based on data either side of the equinox, to ensure a variety of weather systems would be included into the data.

Agencies and of the Models:
BoM – MetEye 5pm daily
EC - European centre for medium range weather forecast model run 00z
GFS - Global Forecast System model 00z

Locations:
2 sea level locations either side of Port Philip bay, 1 elevated location south of divide, 1 elevated location north of the divide.
Melbourne, Geelong, Ferny creek and Kyneton were selected.

Sources:
Forecast - BoM, Windy TV, BSCH, YR, Pivotal weather, Ventusky
Actual - BoM and my PWS Kyneton

Criteria:
I used a Pass and Fail criteria for the outcome to be a percentage based accuracy, based on my own personal threshold for the forecast.

Personal Threshold:

Temp – Max forecast temperature is above or below by 1.6c degrees of actual recorded temperature would be regarded as a Fail.

Rain –
a/ If no rain was forecast and rainfall occurs, it is a fail.
b/ If rain is forecast but does not occur, it is a fail.
c/ Rain outside of 5mm of forecast is a fail.

Results:

Melbourne maximum temperature accuracy forecast
1st – BOM 67%
2nd – EC 52%
3rd – GFS 48%

Melbourne rainfall accuracy forecast
1st – EC 55%
2nd – GFS 45%
3rd – BoM 36%

General comments – BoM was a clear winner on temperature forecasting for Melbourne but was a clear loser on rainfall accuracy. EC was more consistent overall.

Geelong maximum temperature accuracy forecast
1st – BOM 52%
2nd – GFS 52%
3rd – EC 33%

Geelong rainfall accuracy forecast
1st – EC 58%
2nd – GFS 50%
3rd – BoM 50%

General comments – EC was poor on temperature with rainfall fairly close between them all.

Ferny creek maximum temperature accuracy forecast
1st – EC 62%
2nd – BoM 62%
3rd – GFS 38%

Ferny creek rainfall accuracy forecast
1st – GFS 58%
2nd – EC 33%
3rd – BoM 33%

General comments – EC and the BoM were clear winners in temperature but when it came to rainfall they were poor with GFS a clear winner.


Kyneton maximum temperature accuracy forecast
1st – BoM 48%
2nd – GFS 38%
3rd – EC 19%

Kyneton rainfall accuracy forecast
1st – BoM 90%
2nd – EC 90%
3rd – GFS 60%

General comments – BoM was a clear winner with temperature with EC extremely poor on temperature. BoM and EC were excellent on rainfall accuracy.

[Gordon]

Nice work JS. The sample size isn't huge but very interesting reading nonetheless. I've meant to do the same for our place but haven't got around to it yet...

One thing your comparison raises is whether some Vic locations have more 'predictable' weather overall than others?

[JasmineStorm]

Nice work JS. The sample size isn't huge but very interesting reading nonetheless. I've meant to do the same for our place but haven't got around to it yet...

One thing your comparison raises is whether some Vic locations have more 'predictable' weather overall than others?

Yes Gordon, temperature wise both Melbourne and Geelong seemed more predicable than the elevated spots but I was quite surprised how the BoM struggled with rainfall accuracy on 3 out of the 4 locations. Before the study, I assumed the BoM would of been a clear winner in both categories but that certainly wasn't the case, especially on rainfall. The one thing I was confident on was EC struggled on max temperatures for Kyneton, which proved to be correct.

[Australis(Shell3155)]

Not sure of which thread to pop this one into..

[Blackee]

Weather forecasting technology and skills are more advanced than ever. That said, it’s unrealistic to expect pinpoint accuracy for your specific suburb. As I understand it, weather forecasting is less about being "all-knowing at all times" and more about giving the best possible guidance based on current data. If people take on board that as a guiding principle then there would likely be less frustration and criticism. Embracing uncertainty helps manage expectations and makes forecasts feel less misleading. No doubt, reliability of prediction and communication of weather will continue to improve.