WRF Coding Conventions - Draft
	[The primary target audience for this release of the document is for those scientists and researchers who would like to specifically include a physics package into the WRF model.  Therefore, the majority of effort will be spent describing the intricacies of the model-layer in WRF, with just a tacit discussion of the other two layers. Later efforts of this document will remedy this deficiency.]

1. Introduction

1.1 Purpose

1.2 Conventions and Standards Scope

1.3 Definition of Terms

1.4 Document Outline

2. Generic WRF Conventions

2.1 Language Selection

2.2 Acceptable cpp Uses

2.3 Free Form Source

2.4 Explicit Variable Declarations

2.5 Global Variable Declarations

2.6 Variable Intent

2.7 Parallel and Threaded Choices

3. Driver-Layer Subroutine Conventions

4. Mediation-Layer Subroutine Conventions

5. Model-Layer Subroutine Conventions

5.1 Overview

5.2 Variable and Subroutine Conventions

5.3 File, Module and Subroutine Naming Conventions

5.4 Miscellaneous Variable Assumptions

Appendix A. Draft Model-Layer Module Template

1. Introduction

The overall goal of the WRF Model project is to develop a next-generation mesoscale forecast model and assimilation system that will advance both the understanding and prediction of important mesoscale precipitation systems, and promote closer ties between the research and operational forecasting communities.  The model is being developed as a collaborative effort among NCAR, NCEP, FSL and CAPS, together with the participation of a number of university scientists.

                                                                   NCAR USWRP Proposal
                                                                   31 December, 1998

1.1. Purpose

• to provide guidance to those developers who primarily wish to streamline the process for importing physics packages into the existing WRF framework, and

• to require a standard to which the WRF model is written to facilitate package inclusion.

There are two levels of compliance: "desired vs. required."  A minimum set of conventions, necessary to ensure that the resulting code lives up to the requirements for portability and efficiency, are required.  Developers who write code to the minimum required standard will be able to move the code into the WRF model without too much difficulty and with the assurance that the model will run as designed.  Code written to the more expansive desirable framework will be able to have the code included as part of the official release.

1.2 Conventions and Standards Scope

This document is to provide guidance for developers as to what constitutes either a minimally acceptable form of source code in its developmental stages or the desirable form of a more mature package which is ready for inclusion in the WRF model.  If an initial development effort relies on a legacy Fortran 77 set of subroutines (such as is the case for the plethora of physics packages that are expected to be imported into WRF), then a certain latitude is provided for these codes.  However, only fully compliant codes will be adopted and imported into official WRF releases.

1.3. Definition of Terms

1.4.  Document Outline

2. Generic WRF Conventions

2.1. Language Selection

Exceptions to this requirement are the use of cpp (the C language pre-processor) and the underlying language chosen for the MPI interface.  The use of cpp is required for conditional compilation and for permitting machine dependencies in the code.  The cpp utility also permits a more powerful and consistent method to include files from a specific directory.  This document is not applicable to the packages which are written as interfaces between WRF and MPI.  The need for cpp nearly implies an assumption for the requirement for some flavor of the Unix operating system.

All efforts should be made by developers to be writing only in strict compliance to the latest Fortran 90 standard.  To fix a few loose ends in the original Fortran 90, the Fortran 95 standard was adopted by ISO.  Both Fortran 90 and Fortran 95 selected features that were to be possibly discontinued in later releases (obsolescent features), and Fortran 95 actually removed some from the language standard.  In anticipation of the gradual shift to the ever-new definition of (yet another) Fortran, use of the following obsolescent and deleted features is discouraged (the D signifies that this is a desired convention, but not required to actually work):

• Fortran 90 obsolescent features
• arithmetic IF statement - D
• some forms of DO termination (DO loops that have neither an END DO nor a CONTINUE statement, but have a line number for the DO loop terminator on an executable statement) - D
• alternate RETURN labels - D
• FORTAN 95 obsolescent features
• statement function - D
• DATA statement among executable code - D
• assumed-length character functions - D
• fixed form source - D
• CHARACTER* form for character declarations - D
• Fortran 95 deleted features
• real and double precision DO variables - D
• branching to an END IF from outside the IF block - D
• PAUSE statement - D
• ASSIGN statement, assigned GO TO statement - D
• Hollerith editing - D

• operator overloading - If a user sees a plus sign, the user should be able to assume that this is the arithmetic additive operator. -D
• FORALL statement - While the attempt was to provide a standard way to handle threaded code, the effect is to sequentially perform DO-loops on single lines of code.  This is a performance problem.  The OpenMP standard has been chosen for all threaded sections of code. -R
• array syntax - The argument for array syntax is that the user specifies only which operations are to take place and the compiler figures out the most efficient way to handle the order of the operations.  Additionally, this is quite a convenience for developers since array syntax removes the need for the DO statements.  In practice, array syntax impacts performance and threaded regions. -R

• no COMMON BLOCKS - Depending on the specifics of the use, common blocks can minimally pose stylistic concerns or cause the parallel and threading design to completely fail.  -D
• no EQUIVALENCE statements - Automatic arrays may be generated on the stack, there is no need for an additional layer of obfuscation. -D
• no SAVE statements - There are implicit assumptions about the order in which a tile is called and the value of the variables in the SAVE statement.  This inhibits correct functioning of the threaded code. -R

2.2. Acceptable cpp Uses

2.2.1 File Inclusion

SUBROUTINE foo

   IMPLICIT NONE

   INTERFACE

#include <type_package1.int>

#include <type_package2.int>

#include <type_package3.int>

   END INTERFACE

when processed with cpp becomes:

SUBROUTINE foo

   IMPLICIT NONE

   INTERFACE

      SUBROUTINE type_package1 ( arg1 , arg2 )
         IMPLICIT NONE
         INTEGER :: arg1
         REAL :: arg2
      END SUBROUTINE type_package1

      SUBROUTINE type_package2 ( arg1 , arg2 )
         IMPLICIT NONE
         INTEGER :: arg1
         REAL :: arg2
      END SUBROUTINE type_package2

      SUBROUTINE type_package3 ( arg1 , arg2 )
         IMPLICIT NONE
         INTEGER :: arg1
         REAL :: arg2
      END SUBROUTINE type_package3

   END INTERFACE

2.2.2 Conditional Compilation

For example, let the following line be defined in the define.incl file:

#define type_package_init 1

In a subroutine which includes that file (define.incl), the cpp will remove the enclosing text if the macro is not defined.

SUBROUTINE foo

#include <define.incl>

#ifdef type_package_init
      CALL type_package_init
#endif

The ability to remove the subroutine calls is necessitated when the target routines will not be compiled and loaded into the executable.  Without the use of the #include, the user would see messages regarding unsatisfied externals.

2.3. Free Form Source

• white space - In free form source, Fortran 90 does not recognize the following as a valid call to subroutine foo:

• lack of line numbering - It was typical to put line numbers after column 72 in Fortran 77 programs since this was never accessed by the compiler and it served as a convenient editing solution.

• continuation lines - All lines which are to continue onto the next line end with an &, instead of using the leading character in column 6.

      x = x +
     *    (y+1)

• comments - Comments are signified by a ! and may be on the same line as compilable source lines.  All characters after the ! are taken as comments, until the end of the line.

2.4. Explicit Variable Declarations

program foo

   x=7

   call sub1

   print *,'x=',x

contains

   subroutine sub1

   character (len=10) :: x='abcdefghij'

   print *,'x inside sub1=',x

   end subroutine

end program foo

The output from the above program is given below.

 x inside sub1=abcdefghij
 x=   7.000000

By commenting out the character declaration for x in subroutine sub1, the output is as follows:

 x inside sub1=   7.000000
 x=   7.000000

While there are valid reasons to have variables provided by a scoping unit beyond the local routine boundaries, such as with routines inside a module sharing PRIVATE constants or arrays, this does not diminish the need to encourage users into a healthy appreciation of the time savings that is available through the use of explicit variable declarations and the use of the IMPLICIT NONE.

• Explicit varaiable declarations -D
• Use of IMPLICIT NONE -D

2.5. Global Variable Declarations

No common blocks, use modules.  Because of the architecture of the WRF data storage, common blocks that are not thread-safe will cause the code to fail when running with OpenMP. -D

Variables in the declaration area of a module that are not needed by outside routines (those subroutines not contained within the module) should be declared as PRIVATE.  Failure to do so will cause a possibly unreported conflict with variable values. -D

If a module defines a derived type variable, this module should be used for all subsequent variables of that type (do not make this a SEQUENCE type). -D

2.6. Variable INTENT

Use of INTENT attributes on all variables passed through the argument lists of subroutines is desirable. -D

2.7. Parallel and Threaded Choices

3. Driver-Layer Subroutine Conventions

The WRF driver-layer is to be generic enough to be replaced by another model's driver-layer.  This could imply a completely different data storage organization and parallel decomposition.  At this time, no requirements are placed on the model's driver-layer, as even the choice of language is not a requirement.

4. Mediation-Layer Subroutine Conventions

This section of the document is knowingly left empty to provide initial efforts for the model-layer specifically.

5. Model-Layer Subroutine Definition Conventions

5.1. Overview

The WRF model supports a two-level parallel decomposition.  The model is able to run on distributed memory, shared memory, clusters of shared memory machines, as well as on single processor architectures.  If users adhere to the rules for passing variables between the solver routine and the physics package, then the parallel aspects of the WRF code will be maintained in the new parameterization.

5.2. Variable and Subroutine Conventions

1. Logical, memory, run, and patch dimensions are provided unambiguously. At a minimum,
the run dimensions must be provided through argument lists to allow multiple
instances of the routine to be called concurrently within a patch. -R

2. All distributed State and I1 data is passed through argument lists. -R

3. All distributed local data is be defined to be tile-sized. -R

4. Data may be only passed into and out of a subroutine as part of the data
portion of a module only if all of the following are true: -D

a) The subroutine is CONTAINed within the module
b) The shared data is PRIVATE to the module
c) The data is strictly local to a package; that it is not accessible from any subroutine outside the package
d) The data is not distributed over either horizontal dimension
e) The data is defined (written) at package initialization and then ingested (read) for the remainder of the model run

5. A single model-layer wide module containing constants (such as gravity and circumference of the earth) is provided. It's name is module_constants and every routine that wants access to this should USE it. If a new package needs a constant not found in this file, the integrator will add the constant to module_constants and check for consistency/collisions with the rest of the model at that time. Otherwise we run the risk of having multiple and inconsistent constant definitions.  A rule of thumb is that users who have PARAMETER statements in their code probably need to add information to the module_constants file. -D

6. Packages will provide separate initialization routines. If such initialization is unnecessary, provide a stub. That allows a generic init routine to be generated automatically at build time simply by including a call to package_init in every package that is linked in. Packages will not use first-call "latch" mechanisms for initialization. Communication between the initialization routine and the rest of the package will respect the rules for data interchange specified above.  The package_init routine is not threaded but it is able to be called in parallel.  This routine may provide diagnostic print information. -R [This *print* thing violates some rules.  Is that OK?]

7. A storage and loop order is prescribed: (i,j,k). The interface with the WRF solver routine and the package initialization route must both use this ordering. -R

8. Internal to the physics package, ordering may be with the same as prescribed to avoid the penalty of the additional cost of copying or data transposition.  If there is no significant performance issue with the internal choice of the loop ordering, the code may be implemented into the standard WRF release.

9. Inline documentation for each physics package is minimally required to have all of the following items at the top of each routine: -D

• author, email address
• subroutine purpose
• supporting reference papers
• revision history
• who, when, what, why
• description of all subroutine input and output variables
• description of all local variables
• description of all variables used or modified by USE association

10. Each physics package is a separate module.  Each package has its own initialization routine.  Other than the initialization routine, each package has a single entry point which is to be used from outside the module. -R

11. Subroutines in physics modules do not call subroutines in other physics modules.  All physics packages are called from a package driver routine.  For example, all radiation modules are called from the rad_driver routine (which is called by the solver subroutine). -R

12. Physics packages are tile-callable from the package driver.  If the package has loops over decomposed dimensions (i and j), then the internal subroutines must also be tile-callable. -R

13. Other than diagnostic print statements preceeding a call to the provided error handler    routine and print information provided in the package_initialization routine, no I/O is permitted in any of the model layer routines. -D

14. STOP statements are not permitted in the model-layer routines.  Use the provided error handler routine instead. -D

15. Derived types are not passed through the argument lists from the solver to the physics package driver routine or the physics package initialization routine. -R

16. Each physics module should include the define.inc file as a #include option.  This will allow the code to be modified with conditional compilation.  The package initialization and main entry routines should have the executable portion of the code surrounded by appropriate #ifdef statements. -D

17. No heap memory allocation is permitted in the physics routines (such as with the use of an ALLOCATE statement).  The user may allocate as many I2 work arrays as necessary (tile-sized and automatic), which are stack based and evaporate on exiting the routine. -R

5.3. File, Module and Subroutine Naming Conventions

The file which contains the subroutines and variables for the radiation package from Smith, would be called module_rad_smith.F (note that this is a module!).  If there is an include file (such as for local DATA statements), this file would be called rad_smith.inc.  When subroutines wish to access this module through USE association, the module name is module_rad_smith.  The initialization routine in the module is named rad_smith_init.  The entry routine into the module (the routine which the solver routine calls) is named rad_smith_driver.  The user needs to modify the define.inc file to include the line for the particular physics package (i.e. #define rad_smith). -R

5.4. Miscellaneous Variable Assumptions

• Available variables from the WRF solver routine are given below.  Packages requiring other variables must be diagnosed inside the package interface routine. -R
• u, v, T, p, [What are these for real?]
• Variables output from the
• All input and output variables are in SI units. -R
• All data provided by the solver routine is on the Arakawa-C grid, and there are fields that are vertically staggered.  Column based packages which assume co-located mass and momentum fields will simply incur averaging overhead as part of the computation. -R [Do we really provide Arakawa A or C grid?]
• The first dimension of arrays with decomposed dimensions is i.  This dimension increases from left to right, with the starting index of the domain defined as 1.  The second dimension for arrays with decomposed dimensions is j.  This index increases from the lower edge of the horizontal domain to the upper edge of the horizontal domain, with the starting index of the domain defined as 1.  The first grid point of an array with decomposed dimensions is in the lower-left hand corner, and it increases to the right.  The last point in any horizontal slab is in the upper right corner.
• For arrays with a vertical dimension, the index is k.  The k=1 index is located near the surface of the earth, with the index increasing upwards towards the top of the model.

1. How to deal with lateral boundaries cleanly and removing such considerations
from routines that shouldn't be considering them. For example, some of the
physics modules in the current prototype are checking for lateral boundaries to
avoid computing on the last row or column. This shouldn't be in the routine
itself since physics is fundamentally column-oriented and doesn't care where it
is in a domain. Rather, this should be addressed by the caller in determining
the run-dimensions for the routine.

2. Likewise, grid staggering issues crop up at various points in the low level
routines when they don't need to be or shouldn't be addressed at that level.  What is necesary to permit users to know about vertical staggering?  [Could this be handled with naming conventions?]

3. Over-dimensioning of local arrays to allow computation out onto temporary
"halo" regions. Some of this is happening at points within the code now such as
the dpzxy arrays in advance_uv. Not a problem with doing this but it needs to
be conventionalized.

4. If we chose an I-innermost storage order, we need to prescribe a recommended
method for handling minor loops over vertical in physics packages where the
number of iterations is a function of the state of the column -- for example in
PBL or radiation code. This may require some kind of conditional mask over the
inner-most I-loops within those loops. We do not want to allow the practice of
over-iterating some columns (as is done in MM5's implementation of the
Blackadar scheme, for example) because this introduces a non-local dependency
and may produce different answers than if the number of iterations is computed
locally for each column.

• gather-scatter OK
• 2d array into 1d array OK for longer vectors when internal to physics package

5. Adjoint required for each physics package: who does this?

6. External documentation for each physics package: who does this, what format?

Appendix A: Draft Model-Layer Module Template