Appendix A: Objective Analysis (OBSGRID)

 

Table of Contents

 

Introduction

The goal of objective analysis in meteorological modeling is to improve meteorological analyses (the first guess) on the mesoscale grid by incorporating information from observations. Traditionally, these observations have been "direct" observations of temperature, humidity, and wind from surface and radiosonde reports. As remote sensing techniques come of age, more and more "indirect" observations are available for researchers and operational modelers. Effective use of these indirect observations for objective analysis is not a trivial task. Methods commonly employed for indirect observations include three-dimensional or four-dimensional variational techniques ("3DVAR" and "4DVAR", respectively), which can be used for direct observations as well.

This chapter discusses the objective analysis program, OBSGRID. Discussion of variational techniques (WRF-Var) can be found in Chapter 6 of this User’s Guide.

The analyses input to OBSGRID as the first guess are analyses output from the METGRID part of the WPS package (see Chapter 3 of this User’s Guide for details regarding the WPS package).

OBSGRID capabilities include:

Output from the objective analysis programs is used to:

Source of Observations

OBSGRID reads observations provided by the user in formatted ASCII text files. This allows users to adapt their own data to use as input to the OBSGRID program. This format (wrf_obs / little_r format) is the same format used in the MM5 objective analysis program LITTLE_R (hence the name). This format is also used by the WRF-Var program and when doing observational nudging in WRF (For more details on observational nudging see Chapter 5 of this User’s Guide).

Programs are available to convert NMC ON29 formatted files (see below) into the wrf_obs / little_r format. Users are responsible for converting other observations they may want to provide OBSGRID into this format. A user-contributed (i.e., unsupported) program is available in the utils/ directory for converting observations files from the GTS to wrf_obs/little_r format.

NCEP operational global surface and upper-air observations subsets as archived by the Data Support Section (DSS) at NCAR .

NMC Office Note 29 can be found in many places on the World Wide Web, including:
     http://www.emc.ncep.noaa.gov/mmb/data_processing/on29.htm

 

Objective Analysis techniques in OBSGRID

Cressman Scheme

Three of the four objective analysis techniques used in OBSGRID are based on the Cressman scheme; in which several successive scans nudge a first-guess field toward the neighboring observed values.

The standard Cressman scheme assigns to each observation a circular radius of influence R. The first-guess field at each grid point P is adjusted by taking into account all the observations that influence P.

The differences between the first-guess field and the observations are calculated, and a distance-weighted average of these difference values is added to the value of the first-guess at P. Once all grid points have been adjusted, the adjusted field is used as the first guess for another adjustment cycle. Subsequent passes each use a smaller radius of influence.

 

 

Ellipse Scheme

In analyses of wind and relative humidity (fields strongly deformed by the wind) at pressure levels, the circles from the standard Cressman scheme are elongated into ellipses oriented along the flow. The stronger the wind, the greater the eccentricity of the ellipses. This scheme reduces to the circular Cressman scheme under low-wind conditions.

 

 

 

Banana Scheme

In analyses of wind and relative humidity at pressure levels, the circles from the standard Cressman scheme are elongated in the direction of the flow and curved along the streamlines. The result is a banana shape. This scheme reduces to the Ellipse scheme under straight-flow conditions, and the standard Cressman scheme under low-wind conditions.

 

 

Multiquadric scheme

The Multiquadric scheme uses hyperboloid radial basis functions to perform the objective analysis. Details of the multiquadric technique may be found in Nuss and Titley, 1994: "Use of multiquadric interpolation for meteorological objective analysis." Mon . Wea . Rev ., 122, 1611-1631. Use this scheme with caution, as it can produce some odd results in areas where only a few observations are available.

 

Quality Control for Observations

A critical component of OBSGRID is the screening for bad observations. Many of these QC checks are optional in OBSGRID.

Quality Control on Individual Reports

 

The ERRMAX test

The ERRMAX quality-control check is optional, but highly recommended.

The Buddy test

The Buddy check is optional, but highly recommended.

Additional Observations

Input of additional observations, or modification of existing (and erroneous) observations, can be a useful tool at the objective analysis stage.

In OBSGRID, additional observations are provided to the program the same way (in the same wrf_obs / little_r format) as standard observations. Additional observations must be in the same file as the rest of the observations. Existing (erroneous) observations can be modified easily, as the observations input format is ASCII text. Identifying an observation report as "bogus" simply means that it is assumed to be good data -- no quality control is performed for that report.

Surface FDDA option

The surface FDDA option creates additional analysis files for the surface only, usually with a smaller time interval between analyses (i.e., more frequently) than the full upper-air analyses. The purpose of these surface analysis files is for later use in WRF with the surface analysis nudging option. This capability is currently under development in the WRF model.

A separate set of observational files is needed for the surface FDDA option in OBSGRID. These files must be listed by the namelist record2 option sfc_obs_filename. A separate observational file must be supplied for each analysis time from the start date to the end date at time interval INTF4D.

The LAGTEM option controls how the first-guess field is created for surface analysis files. Typically, the surface and upper-air first-guess (analysis times) is available at twelve-hour or six-hour intervals, while the surface analysis interval may be 3 hours (10800 seconds). So at analysis times, the available surface first-guess is used. If LAGTEM is set to .FALSE., the surface first-guess at other times will be temporally interpolated from the first-guess at the analysis times. If LAGTEM is set to .TRUE., the surface first guess at other times is the objective analysis from the previous time.

 

Objective Analysis on Model Nests

OBSGRID have the capability to perform the objective analysis on a nest. This is done manually with a separate OBSGRID process, performed on met_em_d0x files for the particular nest. Often, however, such a step is unnecessary; it complicates matters for the user and may introduce errors into the forecast. At other times, extra information available to the user, or extra detail that objective analysis may provide on a nest, makes objective analysis on a nest a good option.

The main reason to do objective analysis on a nest is if you have observations available with horizontal resolution somewhat greater than the resolution of your coarse domain. There may also be circumstances in which the representation of terrain on a nest allows for better use of surface observations (i.e., the model terrain better matches the real terrain elevation of the observation).

The main problem introduced by doing objective analysis on a nest is inconsistency in initial conditions between the coarse domain and the nest. Observations that fall just outside a nest will be used in the analysis of the coarse domain, but discarded in the analysis of the nest. With different observations used right at a nest boundary, one can get very different analyses.

 

 

How to run OBSGRID

Get the source code

The source code can be downloaded from: http://www.mmm.ucar.edu/wrf/download/get_source.html. Once the tar file is gunzipped (gunzip OBSGRID.TAR.gz), and untared (untar OBSGRID.TAR), and it will create a OBSGRID/ directory.

cd OBSGRID

 

Generate the executable

The only library that is required to build the WRF model is NetCDF.  The user can find the source code, precompiled binaries, and documentation at the UNIDATA home page (http://www.unidata.ucar.edu/software/netcdf/ ).

To successfully compile the utilities plot_level.exe and plot_sounding.exe, NCAR Graphics needs to be installed on your system. These routines are not necessary to run OBSGRID, but are useful for displaying observations.

./configure

Choose one of the configure options.

./compile

If successful, this will create the executable obsgrid.exe. Executables plot_level.exe and plot_sounding.exe, will be created if NCAR Graphics is installed.

Prepare the observations files

Preparing observational files is a user responsibility.

A program is available for users with access to NCAR's computers to download archived observations and reformat them into the wrf_obs/little_r format.

A program is also available for reformatting observations from the GTS stream (unsupported).

In general, there are two overall strategies for organizing observations into observations files. The easiest strategy is to simply put all observations into a single file. The second strategy, which saves some processing time by OBSGRID, is to sort observations into separate files by time (recommended).

Edit the namelist for your specific case

The most critical information you'll be changing most often are the start date, end date, and file names.

Pay particularly careful attention to the file name settings. Mistakes in observations file names can go unnoticed because OBSGRID will happily process the wrong files, and if there are no data in the (wrongly-specified) file for a particular time, OBSGRID will happily provide you with an analysis of no observations.

Run the program

Run the program by invoking the command:

./obsgrid.exe >& obsgrid.out

Check the obsgrid.out file for information and runtime errors.

Check your output

Examine the obsgrid.out file for error messages or warning messages. The program should have created the files called metoa_em*. Additional output files containing information about observations found and used and discarded will probably be created, as well.

Important things to check include the number of observations found for your objective analysis, and the number of observations used at various levels. This can alert you to possible problems in specifying observations files or time intervals. This information is included in the printout file.

You may also want to experiment with a couple of simple plot utility programs, discussed below.

There are a number of additional output files, which you might find useful. These are discussed below.

 

Output Files

The OBSGRID program generates several ASCII text files to detail the actions taken on observations through a time cycle of the program (sorting, error checking, quality control flags, vertical interpolation). In support of users wishing to plot the observations used for each variable (at each level, at each time), a file is created with this information. Primarily, the ASCII text files are for consumption by the developers for diagnostic purposes. The main output of the OBSGRID program is the gridded, pressure-level data set to be passed to the real.exe program (files metoa_em*).

In each of the files listed below, the text "_YYYY-MM-DD_HH:mm:ss.tttt" allows each time period that is processed by OBSGRID to output a separate file. The only unusual information in the date string is the final four letters "tttt" which is the decimal time to ten thousandths of a second. The bracketed "[sfc_fdda_]" indicates that the surface FDDA option of OBSGRID creates the same set of files with the string "sfc_fdda_" inserted.

metoa_em*

The final analysis files at surface and pressure levels. Generating this file is the primary goal of running OBSGRID.

 

 

wrfsfdda_dn

Use of the surface FDDA option in OBSGRID creates a file called wrfsfdda_dn This file contains the surface analyses at INTF4D intervals, analyses of T, TH, U, V, RH, QV, PSFC, PMSL, and a count of observations within 250 km of each grid point.

Due to the input requirements of the WRF model, data at the current time (_OLD) and data for the next time (_NEW) are supplied at each time interval.

result_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

This file contains a listing of all of the observations available for use by the OBSGRID program. The observations have been sorted and the duplicates have been removed. Observations outside of the analysis region have been removed. Observations with no information have been removed. All reports for each separate location (different levels but at the same time) have been combined to form a single report. Interspersed with the output data are lines to separate each report. This file contains reports discarded for QC reasons.

useful_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

This file contains a listing of all of the observations available for use by the OBSGRID program. The observations have been sorted and the duplicates have been removed. Observations outside of the analysis region have been removed. Observations with no information have been removed. All reports for each separate location (different levels but at the same time) have been combined to form a single report. Data which has had the "discard" flag internally set (data which will not be sent to the quality control or objective analysis portions of the code) are not listed in this output. No additional lines are introduced to the output, allowing this file to be reused as an observation input file.

These files can be used in WRF for observational nudging. See Chapter 5 of this User’s Guide for details on observational nudging.

discard_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

This file only contains a listing of the discarded reports. This is a good place to begin to search to determine why an observation didn't make it into the analysis. This file has additional lines interspersed within the output to separate each report.

qc_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

The information contained in the qc_out file is similar to the useful_out. The data has gone through a more expensive test to determine if the report is within the analysis region, and the data have been given various quality control flags. Unless a blatant error in the data is detected (such as a negative sea-level pressure), the observation data are not typically modified, but only assigned quality control flags. Any data failing the error maximum or buddy check tests are not used in the objective analysis.

obs_used_for_oa_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

This file lists data by variable and by level, where each observation that has gone into the objective analysis is grouped with all of the associated observations for plotting or some other diagnostic purpose. The first line of this file is the necessary FORTRAN format required to input the data. There are titles over the data columns to aid in the information identification. Below are a few lines from a typical file.

( 3x,a8,3x,i6,3x,i5,3x,a8,3x,2(g13.6,3x),2(f7.2,3x),i7 )
Number of Observations 00001214
Variable Press  Obs    Station Obs        Obs-1st   X         Y         QC
Name     Level  Number ID      Value      Guess     Location  Location  Value
U        1001   1      CYYT    6.39806    4.67690   161.51    122.96    0
U        1001   2      CWRA    2.04794    0.891641  162.04    120.03    0
U        1001   3      CWVA    1.30433   -1.80660   159.54    125.52    0
U        1001   4      CWAR    1.20569    1.07567   159.53    121.07    0
U        1001   5      CYQX    0.470500  -2.10306   156.58    125.17    0
U        1001   6      CWDO    0.789376  -3.03728   155.34    127.02    0
U        1001   7      CWDS    0.846182   2.14755   157.37    118.95    0

plotobs_out_[sfc_fdda_]YYYY-MM-DD_HH:mm:ss.tttt

Observations files used by plotting program plot_level.

 

Plot Utilities

The OBSGRID package provides two utility programs for plotting observations. These programs are called plot_soundings.exe and plot_levels.exe. These optional programs use NCAR Graphics, and are built. Both programs get additional input options from the namelist.input file.

plot_soundings.exe

Program plot_soundings.exe plots soundings. This program generates soundings from either the quality-controlled ("qc_out_yyyy-mm-dd_hh:mm:ss:ffff") or the non-quality-controlled ("useful_out_yyyy-mm-dd_hh:mm:ss:ffff") upper air data. Only data that are on the requested analysis levels are processed.

The program uses information from &record1 and &plot_souding in the namelist.input file to generate the required output.

The program create output file(s): sounding_<file_type>_<date>.cgm

 

 

plot_level.exe

Program plot_level.exe creates station plots for each analysis level. These plots contain both observations that have passed all QC tests and observations that have failed the QC tests. Observations that have failed the QC tests are plotted in various colors according to which test was failed.

The program uses information from &record1 and &plot_level in the namelist.input file to generate plots from the observations in the  file "plotobs_out_yyyy-mm-dd_hh:00:00:00.0000".

The program creates the file(s): levels_<date>.cgm or levels_sfc_fdda_<date>.cgm, depending on which file type is plotted.

 

Observations Format

To make the best use of the OBSGRID program, it is important for users to understand the wrf_obs/little_r Observations Format.

Observations are conceptually organized in terms of reports. A report consists of a single observation or set of observations associated with a single latitude/longitude coordinate.

Examples

Each report in the wrf_obs/little_r Observations Format consists of at least four records:

The report header record is a 600-character long record (much of which is unused and needs only dummy values) that contains certain information about the station and the report as a whole: location, station id, station type, station elevation, etc. The report header record is described fully in the following table. Shaded items in the table are unused:

Report header format

Variable

Fortran I/O Format

Description

latitude

F20.5

station latitude (north positive)

longitude

F20.5

station longitude (east positive)

id

A40

ID of station

name

A40

Name of station

platform

A40

Description of the measurement device

source

A40

GTS, NCAR/ADP, BOGUS, etc.

elevation

F20.5

station elevation (m)

num_vld_fld

I10

Number of valid fields in the report

num_error

I10

Number of errors encountered during the decoding of this observation

num_warning

I10

Number of warnings encountered during decoding of this observation.

seq_num

I10

Sequence number of this observation

num_dups

I10

Number of duplicates found for this observation

is_sound

L10

T/F Multiple levels or a single level

bogus

L10

T/F bogus report or normal one

discard

L10

T/F Duplicate and discarded (or merged) report.

sut

I10

Seconds since 0000 UTC 1 January 1970

julian

I10

Day of the year

date_char

A20

YYYYMMDDHHmmss

slp, qc

F13.5, I7

Sea-level pressure (Pa) and a QC flag

ref_pres, qc

F13.5, I7

Reference pressure level (for thickness) (Pa) and a QC flag

ground_t, qc

F13.5, I7

Ground Temperature (T) and QC flag

sst, qc

F13.5, I7

Sea-Surface Temperature (K) and QC

psfc, qc

F13.5, I7

Surface pressure (Pa) and QC

precip, qc

F13.5, I7

Precipitation Accumulation and QC

t_max, qc

F13.5, I7

Daily maximum T (K) and QC

t_min, qc

F13.5, I7

Daily minimum T (K) and QC

t_min_night, qc

F13.5, I7

Overnight minimum T (K) and QC

p_tend03, qc

F13.5, I7

3-hour pressure change (Pa) and QC

p_tend24, qc

F13.5, I7

24-hour pressure change (Pa) and QC

cloud_cvr, qc

F13.5, I7

Total cloud cover (oktas) and QC

ceiling, qc

F13.5, I7

Height (m) of cloud base and Q

Following the report header record are the data records. These data records contain the observations of pressure, height, temperature, dewpoint, wind speed, and wind direction. There are a number of other fields in the data record that are not used on input. Each data record contains data for a single level of the report. For report types that have multiple levels (e.g., upper-air station sounding reports), each pressure or height level has its own data record. For report types with a single level (such as surface station reports or a satellite wind observation), the report will have a single data record. The data record contents and format are summarized in the following table

Format of data records

Variable

Fortran I/O Format

Description

pressure, qc

F13.5, I7

Pressure (Pa) of observation, and QC

height, qc

F13.5, I7

Height (m MSL) of observation, and QC

temperature, qc

F13.5, I7

Temperature (K) and QC

dew_point, qc

F13.5, I7

Dewpoint (K) and QC

speed, qc

F13.5, I7

Wind speed (m s -1 ) and QC

direction, qc

F13.5, I7

Wind direction (degrees) and QC

u, qc

F13.5, I7

u component of wind (m s -1 ), and QC

v, qc

F13.5, I7

v component of wind (m s -1 ), and QC

rh, qc

F13.5, I7

Relative Humidity (%) and QC

thickness, qc

F13.5, I7

Thickness (m), and Q

The end data record is simply a data record with pressure and height fields both set to -777777.

After all the data records and the end data record, an end report record must appear. The end report record is simply three integers, which really aren't all that important.

Format of end_report records

Variable

Fortran I/O Format

Description

num_vld_fld

I7

Number of valid fields in the report

num_error

I7

Number of errors encountered during the decoding of the report

num_warning

I7

Number of warnings encountered during the decoding the report

QCFlags

In the observations files, most of the meteorological data fields also have space for an additional integer quality-control flag. The quality control values are of the form 2n, where n takes on positive integer values. This allows the various quality control flags to be additive yet permits the decomposition of the total sum into constituent components. Following are the current quality control flags that are applied to observations.

pressure interpolated from first-guess height      = 2 **  1 =      2
temperature and dew point both = 0                 = 2 **  4 =     16
wind speed and direction both = 0                  = 2 **  5 =     32
wind speed negative                                = 2 **  6 =     64
wind direction < 0 or > 360                        = 2 **  7 =    128
level vertically interpolated                      = 2 **  8 =    256
value vertically extrapolated from single level    = 2 **  9 =    512
sign of temperature reversed                       = 2 ** 10 =   1024
superadiabatic level detected                      = 2 ** 11 =   2048
vertical spike in wind speed or direction          = 2 ** 12 =   4096
convective adjustment applied to temperature field = 2 ** 13 =   8192
no neighboring observations for buddy check        = 2 ** 14 =  16384
----------------------------------------------------------------------
fails error maximum test                           = 2 ** 15 =  32768
fails buddy test                                   = 2 ** 16 =  65536
observation outside of domain detected by QC       = 2 ** 17 = 131072 

OBSGRID Namelist

The OBSGRID namelist file is called "namelist.input", and must be in the directory from which OBSGRID is run. The namelist consists of nine namelist records, named "record1" through "record9", each having a loosely related area of content. Each namelist record, which extends over several lines in the namelist.input file, begins with "&record<#>" (where <#> is the namelist record number) and ends with a slash "/".

The namelist records &plot_sounding and &plot_level are only used by the corresponding utilities.

Namelist record1

The data in namelist record1 define the analysis times to process: 

Namelist Variable

Value

Description

start_year

2000

4-digit year of the starting time to process

start_month

01

2-digit month of the starting time to process

start_day

24

2-digit day of the starting time to process

start_hour

12

2-digit hour of the starting time to process

end_year

2000

4-digit year of the ending time to process

end_month

01

2-digit month of the ending time to process

end_day