CHAPTER 11: SCD FACILITIES
back to the Computing Guide Contents

NCAR's Scientific Computing Division (SCD) handles large-scale computing tasks. This division maintains facilities for large-scale parallel computing, data storage, and output that we do not have in MMM. This chapter provides limited information on accessing SCD facilities. Additional information can be found in SCD documents. The following subjects will be covered:

11.1 SCD - Getting Started

11.1.1 Getting an SCD Account

To use SCD's facilities, you need an account on their system. To do this, you must be authorized on a project number. Sudie Kelly (ext. 8951, e-mail: kellys@ucar.edu, the visitor coordinator, can assist you with project number authorization. Either Sudie or a member of the systems staff will submit an account activation request to SCD. You will then be given a user name and associated user (or scientist) number. A good resource for new users of the SCD facilities is the Using SCD Computers.

11.1.2 The SCD Consulting Office

You can get assistance on the use of SCD facilities from the SCD Consulting Office. The phone number is ext.1278, and the e-mail address is consult1@ucar.edu. The SCD User Documentation Catalog is also available on line.

11.1.3 The Daily Bulletin

SCD publishes a bulletin each working day to keep users informed of upcoming changes in the system and other items of interest. A copy of the SCD Daily Bulletin is available on-line, or you can retrieve the current daily bulletin through MIGS using the following command

        nrnet dailyb /mmmtmp/username/migs/dailyb.out


Return to top of page

11.2 Accessing the Mass Storage System (MIGS)

11.2.1 MIGS Basics

The primary system we use for moving files between MMM and SCD machines (including submitting batch jobs to the IBM systems) is the Masnet/Internet Gateway Server (MIGS). MIGS is accessible from all of our workstations. To use this server, you need a MIGS account. To obtain an account, submit an assist request. After your account has been set up, you will receive an e-mail message telling you to log on and change your password, but DO NOT actually change the password on MIGS.

A multitude of MIGS commands exist to move files of various types between various machines. For instance, you can send files to the Mass Storage System and retrieve files from it, and direct tapes to be imported or exported. All commands are detailed in the MIGS User Guide, and briefly in the MIGS Quick Reference card. Some of the more common commands will be mentioned in later sections. All begin with the nrnet command, followed by a verb describing the funtion you want to perform, and other parameters. 

        Syntax: nrnet verb filename [parameters]
where filename is a local filename, either an input or output filename, depending upon the verb selected, and parameters are any required or optional remote or local parameters, also depending upon the verb. Remote parameters are preceded by an r, and local parameters are preceded by an l. Remote applies to the destination machine; local applies to the originating machine.

All MIGS output is returned to a specific directory, /mmmtmp/your_username/migs. Note that this directory is subject to being scrubbed. However, files in the MIGS directory will always be retained for at least three days.

To use MIGS, the following entries must be in your .rhosts file on the MMM systems and in your .rhosts file on the SCD computers. 

        migs masgw
        migs.ucar.edu masgw

11.2.2 Accessing the Mass Storage System via MIGS

The SCD Mass Storage System (MSS) stores all types of files. Its primary use is for data input and output files used by programs on the IBM systems, but it can be used to store any files. It has a file structure similar to UNIX systems, with subdirectories separated by the character, /. To send files to the MSS directly from MMM computers, and retrieve files from it, use the MIGS nrnet verbs msput and msget. The format for sending a file to the MSS is 
        nrnet msput filename l [local parameters] 
        r flnm =/ SCDUSER/ msfile [remote parameters]
Optional remote (r) parameters of particular interest are 
rtpd =DDD       Retention period, 1-367 days (default is 30).

r =rpass        Read password.

w =wpass        Write password.

cmnt            Comment (enclose in single quotes if 
                it contains blanks).
These must be placed to the right of r since they are remote (MSS) parameters. For example, rtpd=367 specifies a retention period of 367 days from the last access date rather than the creation date. The default format is transparent (l df=tr, move the file byte-by-byte), and is specified right after filename. Ordinarily, you should use transparent format with both msput and msget, even though the file itself may be a binary or character file.

To retrieve a file from the MSS, the syntax is similar. 

        nrnet msget filename  r flnm=/SCDUSER/msfile
Since MIGS has write access only to the migs subdirectory on /mmmtmp, msget must be initialized in that directory. MIGS can read files from all directories. You must use the read password parameter r=rpass if the file was written with a read password; otherwise, exclude this remote parameter. The other remote parameters, rtpd, w, and cmnt are not applicable when getting a file from the MSS.

For a list of mass-store files, enter 

        nrnet msallinfo mss.inp
where mss.inp is a file that characterizes how you want the list to look. An example of this input file is

        NAME=/USERNAME/! (all uppercase)
        FORMAT=DATE
        OUTPUT=NAME, CRTIM, EXPDATE, RETPD, SIZE, DF
        END
        NAME=/uSERNAME/! (first letter lowercase, rest 
                          of word uppercase)
        FORMAT=DATE
        OUTPUT=NAME, CRTIM, EXPDATE, RETPD, SIZE, DF
        END

This produces an output from msinfo with the MSS filenames, creation dates, last access dates, expiration dates, retention periods (in days), sizes (in bytes), and formats BI (binary), TR (transparent), or CH (cos-blocked) character of all files (! is a wild card character) that belong to the SCD logon, USERNAME.

The second entry shows any files that will be deleted soon (within 30 days). You can rename files in this latter category to /USERNAME/... with the msmv command to prevent their deletion.

11.2.3 Distributed Computing Environment access to the MSS

An alternative method for accessing the Mass Storage System from a subset of the divisional computers is the Distributed Computing Environment set of commands developed by SCD. These commands are comparable to the commands that are directly available on the SCD systems. The DCE command set is available on the Alpha and Linux platforms, and also the dataserver bay. The list of available commands follows. There are on-line man pages for the use of all of these commands. 
        msallinfo   - List all info about files
        mscd        - Change working directory
        mscdsetup   - Set up to use mscd and mspwd
        mschproj    - Change project number of files
        mscomment   - Change comments associated with files
        msdu        - Display directory usage statistics
        msexp       - Export files into the MSS
        msfind      - Find files and run commands on them
        msimp       - Import files into the MSS
        msintro     - Man page - introduction to the MSS DCS commands
        msls        - List files on MSS
        msmetadata  - Submit an MSS metadata command
        msmv        - Renames MSS files
        mspasswd    - Change password of files
        mspwd       - Print working directory
        msrawinfo   - List MSS-specific information about MSS Master File       
                      Directory
        msrcp       - Copy files to/from the MSS
        msrecover   - Recover recently deleted/purged files
        msretention - Change retention period of files
        msrm        - Remove files
        msstage     - Place files online or offline
        mstouch     - Update the last-referenced date
It should be noted that for the DCE commands to work from the data server, bay, a special environment variable needs to be set. The following lines should be added to your .login file if they are not already there. 
set HOST = `uname -n`
if($HOST == bay) then
   setenv RPC_SUPPORTED_NETIFS ge0
endif

Return to top of page 

11.3 The IBM Systems

11.3.1 Submitting Jobs to the IBM Systems

The name of the IBM systems available for processing are babyblue, blackforest, and bluesky. There are nrnet verbs for sending jobs to these systems, but there are also MMM scripts, sendbb, sendbl, and sendbs, which have additional funtionality. 
        Syntax: sendbl filename
OR  
        Syntax: nrnet bljob filename
where filename is the name of the file with the LoadLeveler script on /mmmtmp/username/migs to be run on blackforest. Output will be returned to /mmmtmp/username/migs. You can give the output file any name you choose with the batchname command in your script. By default, the file will be named PA.XXnnnn.aaaaa, where XX is the two-character machine code from which the job was submitted (HM for hop), nnnn is the four-digit sequence number assigned when the job is submitted, and aaaaa is a randomly-generated number to uniquely identify your job.

To run a number of jobs that use the same FORTRAN source code, the sendbl command allows you to insert just a reference to the file you want to include. It will expand each file that is referenced in this way before sending the final job to blackforest. Within the file, the following line will expand the contents of filename at that location into the job that is submitted to blackforest. 

        include(filename)
Note that the parentheses are needed, and that the full pathname of the file is required. Multiple include commands can exist in each file, and the inclusions occur exactly where the commands appear in the file. If you use nrnet bljob filename instead of sendbl, the file will be sent, but blackforest will probably not understand how to proceed when it encounters the include command. Therefore, use this method only when the script contains no include commands.

11.3.2 Accessing the Mass Storage System from the IBM Systems

From blackforest, the commands for moving files back and forth from the MSS are msget and msput. The default file format for these commands is "transparent." The files are read in the same format in which they were written. 
        Syntax: msget [options] crayfile /SCDUSER/msfile
To read the file as data to a FORTRAN program, you can read the file from MSS and assign the read unit number in a single command. 
        msget fort.nn /SCDUSER/msfile
where nn is the FORTRAN read unit number. If the MSS file was originally written by specifying the format (e.g., -f = BI), it must be read with that format. If you do not know the format, you can find this information with the msinfo command. (See Section 11.2.2.)

To write a file to the MSS from blackforest, use the msput command. 

        Syntax: msput [options] crayfile /SCDUSER/msfile
Some optional parameters are 
        -t n       Retention period in days 
        -w rpass   Write password
        -r wpass   Read password
        -ccomment  Comment (enclose in double quotes 
                   if containing blanks)
        -nowarn
Of these, only -r is applicable to msget. For example, to write the file gmeta to MSS with a retention period of 60 days and a write password of mine to SCDUSER's MSS file, program/gmeta, enter 
        msput -t 60 -w mine gmeta /SCDUSER/program/gmeta

11.3.3 Sending Output Back to MMM

Use the rcp command to send output to MMM workstations. To do this, the proper .rhosts files must be set up. 
Syntax:rcp filename user@host.mmm.ucar.edu:/mmmtmp/user/mmmfile
For example, user jones could copy the metacode file produced in the program to his or her area on /mmmtmp on laurel by entering 
rcp gmeta jones@laurel.mmm.ucar.edu:/mmmtmp/jones/prog1.gmeta
It is most efficient to rcp directly to the disk where you want to look at the output. It is not necessary to always rcp to /mmmtmp. Be sure enough space is available on the receiving disk before blackforest executes rcp. (To do this, use the command df on the workstations. ) Otherwise, some or all of the output will be lost.

11.3.4 Checking Job Status

The easiest way to obtain the status of your batch jobs is to log directly onto blackforest and look at the queues. The command to obtain the status of currently running jobs is batchview.

Hardcopy References