Verification of Forecast Weather Surface Variables over Vietnam Using the National Numerical Weather Prediction System

Thenational numerical weather prediction systemofVietnam is presented and evaluated.The system is based on threemainmodels, namely, the Japanese Global Spectral Model, the US Global Forecast System, and the USWeather Research and Forecasting (WRF) model. The global forecast products have been received at 0.25and 0.5-degree horizontal resolution, respectively, and the WRF model has been run locally with 16 km horizontal resolution at the National Center for Hydro-Meteorological Forecasting using lateral conditions from GSM and GFS. The model performance is evaluated by comparing model output against observations of precipitation, wind speed, and temperature at 168 weather stations, with daily data from 2010 to 2014. In general, the global models providemore accurate forecasts than the regionalmodels, probably due to the low horizontal resolution in the regionalmodel. Also, the model performance is poorer for stations with altitudes greater than 500meters above sea level (masl). For tropical cyclone performance validations, the maximum wind surface forecast from global and regional models is also verified against the best track of Joint TyphoonWarningCenter. Finally, themodel forecast skill during a recent extreme rain event in northeastVietnam is evaluated.


Introduction
Vietnam stretches over 15-degree latitude and has a seasonal and relatively predictable weather.The northern regions have a humid subtropical climate, while the south is in the tropical zone.However, because of differences in latitude and the marked variety of topography, the climate tends to vary considerably from place to place [1].
The weather in Vietnam is dominated by the East Asia Monsoon.During the winter or dry season, extending roughly from November to April, the monsoon winds usually blow from the northeast along the China coast and across the Gulf of Tonkin, picking up considerable moisture; consequently the winter season in most parts of the country is dry only by comparison with the rainy or summer season [2].
The rainy season lasts from May or earlier to October.About 80-90 percent of the precipitation is related to the southwesterly summer monsoon, tropical cyclones from the East Sea of Vietnam, and other tropical disturbances.Annual rainfall ranges from 1200 to 3000 millimeters; the average of monthly precipitation amount in summer ranges from 150 to 300 millimeters [3].The average annual temperature is generally higher in the plains than in the mountains and plateaus.Temperatures range from a low of 5 ∘ C in December and January, the coolest months, to more than 37 ∘ C in April, the hottest month.Seasonal divisions are more clearly marked in the northern half than in the southern half of the country, where, except in some of the highlands, seasonal temperatures vary only by a few degrees, usually in the 21 ∘ C-28 ∘ C range.
Downscaling of global climate models indicates that even though the air temperature in Vietnam is expected to increase by up to 0.5 K per decade in summer, the precipitation signal depends more on season and region [4].Overall, an increase in rainfall can be expected, even though the influence of global warming on monsoon patterns is highly uncertain [5].It is generally expected that there will be a higher occurrence 2 Advances in Meteorology of extreme precipitation events with global warming, and improved accuracy of operational forecasts can be expected to be of increasing importance to society [6].
It is important to improve the short term forecasting in Vietnam in order to increase the preparedness and mitigate effects of weather induced events such as river flooding, landslides, and strong winds [7].In order to test and enhance the forecasting system at the National Hydro-Meteorological Service of Vietnam (NHMS), verification against routine observations is useful, preferably including several years of data.Other important phenomena affecting Vietnam are tropical cyclones.With over 3200 km coastal length, Vietnam is also affected by 8-12 tropical cyclones every year moving from Western North Pacific Ocean or generated in South China Sea, the East Sea of Vietnam [8].Tropical cyclones affect both land and sea by strong wind and heavy rainfall.
By comparing with satellite-derived precipitation, forecast validations from both global and regional models over Southeast Asia have previously been carried out.For example, Yuan et al. [9] validated precipitation forecast from GFS/NCEP system with PERSIAN data for 1-7-day forecast ranges.The results showed that, for the Southeast Asia region, the RMSE of GFS is 5-10 mm/day for 24-hour forecast range and large RMSE values were found over the coastal areas of the center and the south of Vietnam: Indonesia and the Philippines.At regional forecast scales, Hayashi et al. [10] verified the forecast from WRF-ARW model (running at 20 km horizontal resolution) with CMORPH observations over the northern parts of Southeast Asia (Indonesia) from 2007 to 2008.They showed that the forecasts of daily rainfall had a tendency of overestimation for threshold under 5 mm/3 h (bias > 2) and underestimation at higher threshold (bias below 0.3 for higher 10 mm/3 h).
The purpose of this paper is threefold.Firstly, we want to present a brief overview of the operational forecasting system and some characteristics of the weather in Vietnam.Secondly, we will discuss the performance of the forecasting models for different regions and seasons, with emphasis on wind and precipitation since extreme maxima of these two parameters are key threats to lives, livelihoods, and infrastructure.We will present verification statistics for the period 2010-2014.Finally, we will discuss the performance of the model system during an extreme rainfall event in the northern region of Vietnam in July/August 2015.

Materials and Methods
Operational forecasting at the National Center for Hydro-Meteorological Forecasting in Vietnam (NCHMF) is based on three main models, namely, the Japanese Global Spectral Model (GSM), the NOAA Global Forecast System (GFS), and the National Center for Atmospheric Research (NCAR, US) Advanced Research of Weather Research and Forecasting (WRF) model, which is run locally.
The WRF model version 3.1 [11] has been applied since 2010 as the main forecasting tool at NHMS Vietnam.Due to limited computing power available, it is run in a 1domain setup of 16 km resolution and other configurations including the Rapid Radiative Transfer Model (RRTMG) for long-and short-wave radiation schemes, the Monin-Obukhov surface layer scheme, the Noah land surface model land surface parameterization, the Yonsei University planetary boundary layer, and the Kain-Fritsch cumulus parameterization scheme.Boundary conditions are taken from GFS or GSM model with 0.5 deg resolution.Model output is given every 3 hours with 60-hour lead time.At the surface, the following parameters are analyzed: mean sea level pressure (MSLP), cloud cover, specific humidity, precipitation, 2 m temperature, and wind at 10 m.At standard pressure levels, air temperature, wind, geopotential height, and specific humidity are analyzed.
GFS is the global spectral model of NCEP (National Center for Environmental Predictions) (USA) and was put into operation in NCEP since 1988 [12].The GFS model is a model with an approximate horizontal resolution of 13 km for the first 10 days and 27 km from 240 to 384 hours (16 days).In the vertical, the model is divided into 64 layers and it produces forecast output every hour for the first 12 hours, every three hours through days 10, and every twelve hours through day 16.Since 2005, NCHMF began to receive GFS model output with resolution of 0.5 degrees and 27-level outputs.
GSM is the global spectral model of Japan Meteorological Agency (JMA), and it was put into operation at JMA in 1988 [13].The first model had horizontal resolution of 55 km and 16 verticals levels, the top isobaric level was 10 hPa, and primitive equations were written in sigma coordinates.In 2007, horizontal resolution was improved to 20 km and the number of vertical layers was increased from 40 to 60.In 2013, the range of the forecast run at 12 UTC was extended from 216 to 264 hours for both the GSM and the One-Week Ensemble Prediction System.NHMS Vietnam has applied GSM since 1997.Since 2008, NCHMF began to receive GSM model with resolution of 0.5 degrees and 21-level output.NCHMF also received GSM surface fields with resolution of 0.25 degrees since 2010.
The initial and boundary conditions for the WRF model are taken from above GSM and GFS forecasts at surface and pressure levels.Depending on driving boundaries, the WRF forecasts are named as WRF-GFS if using GFS and WRF-GSM if using GSM.An example of different rainfall forecast from GSM and GFS and downscaling forecast from WRF-GSM and WRF-GFS is illustrated in Figure 8.
During tropical cyclones, the track and intensity forecast are calculated from these models by finding the minimum sea surface pressure (as position of tropical cyclones) and maximum surface wind (as intensity of tropical cyclone).
The number of observation stations in Vietnam increased from 89 in 1988 to 178 in 2015, with 4 or 8 observations per day.In this paper, in order to verify model data, we use observation data from 168 stations.The spatial distribution of all local observations is plotted in Figure 1(a).Of these 168 stations, only about 18-24 stations are reported to WMO every 6 hours.The highest station density is in the Red River Delta area with approximately 1 station per 750 km 2 .The coarsest station density is in the central highland area with approximately 1 station per 3200 km 2 .On average, the current surface observation network density of Vietnam is about 1 station per 1500 km 2 for flat regions and 1 station per 2500 km 2 for complex regions.
Here we have used simple statistics in order to compare forecasts with observations.The 24 h forecast for the period 2010-2014 has been stored for each station and compared with observations.Three main verification scores will be used in this study including mean absolute error (MAE), root mean square error (RMSE), and equitable threat score (ETS) [14].The definition of ETS is as follows: where Hits random = (H + F)(H + M)/T, H is the hit rate of occurring rainfall (at a given threshold) for both forecast and observation, M is missed rate of occurring rainfall forecasts, F is the false alarm rate of the forecast, and T is a sum of H, M, and F and number of nonoccurring rainfalls for both forecast and observation.
For tropical cyclone performance validations, the maximum wind surface forecast from global and regional models will also be verified against the best track of Joint Typhoon Warning Center (JTWC) [15] for this period.

Results and Discussion
The observation quality control processes are based on the comparisons of observation values and other references including climatology and mean values of surrounding stations and from model forecast.If deviations of observations and references are high, these observations will be classified as high probability of wrong observation.The criteria for classification are also dependent on variables and climate regions.In our study, the precipitation and wind observation are simply controlled if they are within reasonable ranges.The temperature observations are accepted only when passing through two steps: (i) they are numerically with a reasonable range compared to climatology data and (ii) they are checked to be not too much off from their surrounding observations.For example of quality control results, on March 16, 2011, for WMO station Da Nang with ID 48885, the temperature of 00Z (7 am local time), 06Z (13 pm local time), 12Z (19 pm local time), and 18Z (01 am local time) was 20.3 ∘ C, 18.8 ∘ C, 17.9 ∘ C, and 18.4 ∘ C. The minimum of temperature of climate data for 06Z (13 pm local time) is 20.1 ∘ C (the mean value minus five times the standard deviation of temperature for Da Nang station from 1998 to 2010); therefore the value 18.8 ∘ C for 18Z had been qualified as wrong value even though actually this is caused by cold surge from the north.Another example also for Da Nang station on April 18, 2011, the temperature of 00Z, 06Z, 12Z, and 18Z was 25.7 ∘ C, 23.2 ∘ C, 23.9 ∘ C, and 22.7 ∘ C. The value 23.2 for 06Z was qualified also for the climatology check.In summary, Tables 1 and 2 show the percentage of data perfection in every month of 2010-2014 for temperature and wind speed, respectively.About 85% of the temperature observations and 88% of the wind speed observations are accepted for this period.
The maps for average wind (Figure 2) reveal temporal and regional patterns.Not surprisingly wind speeds (U10) are highest in the late afternoon (12Z observations).Wind speeds are highest along the coast (particularly the Red  Looking at forecasts of U10 (Figure 4), the two global models (GSF and GSM) have a RMSE compared to observations of around 1 m/s in most of the country, except in the central highland south and central coastal areas, where the RMSE is around 3 m/s.For the two regional realizations with global models on the boundaries (WRF-GFS and WRF-GSM), the RMSE is around 3-6 m/s for the entire country.Patterns are similar in both summer and winter, but with slightly higher errors in winter.There were no systematic biases for U10 from both global and regional forecasts.
About 42 tropical cyclones affecting Vietnam, with about 600 forecast cycles (see Table 3 and Figure 1(b)) from 2010 to 2014, have been verified.The global forecast mean absolute errors range from 9 to 10 m/s for 24 h forecast range and 10 to 11 m/s for 48 h forecast range.The regional forecasts perform 10-15% better compared to global errors, from 8 to 9 m/s for 24 h forecast and 8.5 to 9.5 m/s for 48 h forecast (see Table 4).Some track forecasts validations (not shown here) also for these tropical cycles showed that there were small differences between regional and global forecasts.
For 24-hour accumulated precipitation, RR24 (Figure 5), in winter, the GFS forecasts compare slightly better to observations compared to GSM, with MAE < 10 mm for most regions, while the regional models reveal errors of 10-20 mm in most regions.In summer, the GSM provides the most accurate forecast.Again, errors are highest in the central highland.In general, moderate rain is overestimated, while severe rains are underestimated (Tables 5 and 6), which corresponds to the results by Hayashi et al. [10].
The ETS was computed at three thresholds for this period and shown in Table 8.The GSM showed highest values for both 24 h forecast range (0.2) and 48 h forecast range (0.16).The regional models can compare with global forecast only at the threshold below 16 mm/24 h.At the higher threshold (over 50 mm/24 h), the ETS is reduced from 30 to 40% compared to the lower threshold.
For 2 m air temperature, T2M (Figure 6), errors tend to be smaller in the south, with GSM having the most accurate forecasts (MAE < 1 K in the south), otherwise mostly MAE < 2 K. Again, the regional models seem to have poorer skill than their global counterparts.
Finally, we have studied the extreme precipitation episode in the northeast region during the period from 25 July to 3 August 2015 (Figure 7).This region is characterized by a relatively dense observation network (49 stations, Figure 1).A tropical depression was stationary off the coast in this period, providing a northeast air flow with unprecedented 216 hr precipitation amounts at several stations.The highest precipitation was recorded at Cua Ong (StationID: 48836, 107.35E; 21.02N, 57 masl) with 1172 mm.This station is not in the GTS WMO list for Vietnam and consequently not assimilated in the global models.During this episode, GFS seems to be more accurate, with MAE for RR24 of around 70 mm in the affected region (Figure 8).The regional model forecasts reveal MAE of around 100 mm, showing that the extreme precipitation episode was grossly underpredicted by all models (Table 7).

Conclusions
The forecasting system at National Center for Hydro-Meteorological Forecasting, National Hydro-Meteorological Service of Vietnam, has been described and evaluated.The system is based on available data from two global models (GFS and GSM) as well as a regional model (WRF) which is run locally with 16 km resolution.Here, we have presented verification results for 24-hour accumulated precipitation (RR24), 10 m wind speed (U10), and air temperature at 2 m (T2m).Model data has been stored for 168 stations for the period 2010-2014 and compared with observations at these stations.Related tropical cyclones, the maximum surface wind, or intensity forecasts were also verified against the best track of JTWC.
For precipitation forecasts, the results show low skill of all models for thresholds over 16 mm per day.All models can provide correct forecasts over 90% for only small amount of rainfall (0-16 mm/24 h) and below 30% for heavy rainfall.The models can capture precipitation over 16 mm/day better in winter compared to summer (30-50% compared to 10-30% in summer).In general, the GSM has better skills in precipitation forecast than other systems (highest ETS).The skills of regional forecast for surface variables are clearly lower than global forecasts.For tropical cyclone forecasts, there is no improvement of track forecasts in regional forecasts but the regional forecast can provide slightly more accurate intensity than global forecasts.
For RR24, the MAE for all stations with height below 500 m was 6.8 mm for GSM, 5.8 mm for GFS, and 5.7 mm for both WRF-GSM and WRF-GFS.For stations with height above 500 m (29 stations) the MAE was 6.1 mm for GSM, 6.7 mm for GFS, 8.9 mm for WRF-GSM, and 10.3 mm for WRF-GFS.The poorer performance for high altitude stations is in accordance with Shea et al. [16].For U10 and T2m, there is no clear difference in errors for station above or below 500 m.
In addition, an extreme rain event during the period from 25 July to 3 August 2015 has been studied in particular.In general, the regional model has lower skill than the       global products.All models have lower skills in mountainous regions compared to lower regions.The resolution of WRF at 16 km is not enough for simulating the mesoscale phenomena after downscaling from global forecasts.The large errors from regional models can be reduced by using assimilation schemes in upcoming studies with local observations of Vietnam that have not earlier been submitted to the global telecommunication system of WMO.Only about 20-30 stations of Vietnam are reported and can be used in assimilation of global models.In the Advances in Meteorology 11 near future, new computing facilities and faster networks will be available, and the regional forecasts can be improved with increasing horizontal and vertical resolution.Systematic testing of different boundary layer schemes and surface physics will also be carried out for heavy rainfall cases occurring in Vietnam.
Even though the present study shows that the regional models perform poorer than available global products, NHMS Vietnam has gained competence and experience running WRF and is now prepared to run the model with nesting and higher resolution.
Figure 1: (a) Weather stations in Vietnam where 21 stations over the northwest are marked by green circles, 49 stations over the northeast are marked by red triangles, 12 stations over Red River Delta are marked by black stars, 20 stations over the north center are marked by inverted brown triangles, 15 stations over the middle center are marked by blue squares, 11 stations over the south center are marked by blue rhombuses, 17 stations over the central highland are marked by dark green pentagons, and 23 stations over the south are marked by pink circles.(b) Tropical cyclone activities over East Sea of Vietnam from 2010 to 2014.
Average winter 10 m wind speed (h)

Table 3 :
Tropical cyclones affecting Vietnam sea from 2010 to 2014.

Table 4 :
MAE of maximum surface wind forecast for tropical cyclone affecting East Sea of Vietnam from 2008 to 2014 (unit m/s) for 24 h and 48 h forecast ranges.Notice that the GSM in this table is in 0.5-degree resolution.

Table 5 :
Contingency table for RR24, winter (October-March), 2010-2014.Bold font indicates correct forecasts.The columns are the four observation categories.

Table 7 :
Same as Table3, but for the extreme event in the northeast region (49 stations), 27 July to 3 August 2015.

Table 8 :
ETS at three thresholds for 24 h and 48 h forecast range of GFS, GSM, WRF-GFS, and WRF-GSM in Vietnam from 2010 to 2014.