Using UpsAnd

Table of Contents

• Introduction
• Node Allocations
• Home Directory
• Compiling
• Running Your Code
• Monitoring Jobs
• Visualization
• FAQ's

Introduction

This page contains information regarding compiling and running C and FORTRAN code on the UCSC beowulf, UpsAnd. UpsAnd is a 120 node beowulf cluster, with each node consisting of two 1.4 GHz Athlon processors and 1 GB of RAM.

Logging onto UpsAnd is done via ssh:

1. first log onto Andromeda: ssh andromeda.pmc.ucsc.edu -l [username]
2. then log onto UpsAnd: ssh upsand

Copying files from/to UpsAnd is done via scp: while on UpsAnd,

1. scp [username]@your.host.computer:[files] . (to copy files from your host computer to UpsAnd)
2. scp [files] [username]@your.host.computer:. (to copy files from UpsAnd to your host computer)

Back to Table of Contents

Node Allocations

The following is a list of PI's, their email, and the current node division.

PI	email	Nodes
Asphaug	asphaug@emerald.ucsc.edu	None
Glatzmaier	glatz@emerald.ucsc.edu	n111 - n116, n121 - n126, n131 - n136, n141 - n146
Lin	lin@ucolick.org	n211 - n216, n221 - n226, n231 - n236, n241 - n246
Primack	joel@scipp.ucsc.edu	n311 - n316, n321 - n326, n331 - n336, n341 - n346
Woosley	woosley@ucolick.org	n411 - n416, n421 - n426, n431 - n436, n441 - n446

Another good email to have is Shawfeng Dong, dong@ucolick.org. Shaw administrates the beowulf and is responsible for keeping it up and running, updating and installing software and pretty much all the details.

Back to Table of Contents

Home Directory

On the frontend UpsAnd each user has a home directory (/home1/userid or /home2/userid), which is also NFS mounted by all the nodes. The home directories locate on two RAID hard disk arrays, with a capacity of 700 GB each.

Warning !: You are advised to back up your data frequently. The RAID arrays only provide minimum protection against one hard disk failure.

Warning !: The home directories sometimes get inconveniently full, so much so that simulations have crashed. Please keep your directory under 50 GB. Check your disk use periodically by doing a "du -k" from your home directory.

Although not recommended, if you need to write files local to the compute nodes write them to /scratch, which is a bit under 32 GB on each compute node. To get files off a compute node that have been written into /scratch, do the following on the frontend UpsAnd:

rcp n112:/scratch/yourfile.dat .

This will copy the file called yourfile.dat from /scratch on n112 to the directory you are in on the frontend.

Back to Table of Contents

Compiling

We recommend you to use PGI compilers to compile your code. PGI High-Performance Compilers and Tools (version 5.1) are installed on UpsAnd. Please refer to the following links for online documentation:

1. PGI User's Guide (PDF)
2. PGI Tools Guide (PDF)
3. PGF77 Reference Manual
4. PGHPF User's Guide
5. PGHPF Reference Manual
6. PGDBG Debugger (PDF)

Using MPI (Message Passing Interface) is the only supported method to write and run parallel programs on UpsAnd. ScaMPI, Scali's optimized implementation of MPI, is installed on UpsAnd. For technical details, please refer to the following online documentation:

1. ScaMPI User's Guide (PDF)
2. The Message Passing Interface (MPI) standard

You may also want to check the directory /opt/scali/examples/src or /opt/scali/doc/ on UpsAnd for various examples and information.

Below I include a sample Makefile used to compile the C program GADGET on UpsAnd (courtesy Thomas Cox), another sample Makefile for the Fortran program NBODY5, and a command line for the compilation of Gary's FORTRAN code. These tools will be valuable references for editing your makefiles and/or creating scripts to compile your C or FORTRAN code.

• TJ's Makefile for C program GADGET: the makefile TJ uses to build GADGET. Salient features are:
  1. CC = pgcc: the C compiler is pgcc (the Portland group C compiler);
  2. CFLAGS = $(OPTIONS) -I/opt/scali/include: the path /opt/scali/include needed to be specified;
  3. LIBS = -lm -L/opt/scali/lib -lmpi: the path /opt/scali/lib needed to be specified and the mpi library needed to be explicitly added here.
• Shaw's Makefile for Fortran program NBODY5. Salient features are:
  1. FORTRAN = pgf77: the Fortran compiler is pgf77 (one can also use pgf90);
  2. FFLAGS = -O2 -tp=athlonxp -pc=64 -fastsse -D_REENTRANT -D PUREMPI -D PARALLEL -I/opt/scali/include: the path /opt/scali/include needed to be specified as in C;
  3. LIBS = -lm -L/opt/scali/lib -lmpi -lfmpi: as in C, the path for MPI library needed to be specified.
• If you prefer command line, you can also compile your code similar to the following (kindly supplied by Gary):
  

  pgf77 -D_REENTRANT -Ktrap=fp -byteswapio -fast -tp=athlonxp -Mvect=prefetch,sse -r8 -o ax a.f -I/opt/scali/include -L/usr/pgi/linux86/5.1/lib -lstd -L/opt/scali/lib -lmpi -lfmpi (all in one line)

  Note that it also demonstrates a few optimization options that you can use to produce efficient codes.

Back to Table of Contents

Running Your Code

There are two methods of submitting and running programs on the cluster: mpimon and OpenPBS.

• Using mpimon (or mpirun) to submit jobs:
  

  For technical details, please refer to ScaMPI User's Guide (PDF). In general the command to submit your MPI job with mpimon is:

  mpimon -init_comm_world ./exec -- nxxx 2 nxxx 2 ...

  In the above command, the 2 after nxxx tells mpi to run 2 mpi tasks on one node. Each node has 2 processors. If you want to run only one process per node, leave off the 2's after each nxxx.
  

  Here we include two examples for running programs on UpsAnd:

  • A sample command provided by Gary, which runs the program amhd18sx on 9 nodes using 2 processors per node:

    nohup time mpimon -init_comm_world amhd18sx -- n111 2 n112 2 n113 2 n114 2 n115 2 n116 2 n121 2 n 122 2 n123 2 & (all in one line)

    Note that nohup and time are standard UNIX commands; they are not required to submit the job.
  • a script provided by TJ, which attempts to run the program P-Gadget on nodes n311-n314 using both processors on each node. This script also attempts to send TJ an email after the run is finished.

• Using OpenPBS (ScaOPBS) to submit jobs: (PLEASE DO NOT USE THIS METHOD!)
  

  We have also installed ScaOPBS on UpsAnd. ScaOPBS is the Scali binary packaging of OpenPBS, which provides an easy way for submitting, monitoring, modifying and deleting MPI jobs. For technical details, please refer to the following links for online documentation:

  1. Chapter 8 of Scali System Guide (PDF)
  2. OpenPBS System Administrator's Guide (PDF)

  In general the command to submit your MPI job using ScaOPBS is:

  scasub -mpimon -np 4 -npn 2 ./exec

  This example will start a total of 4 processes, two processes per node.

  A brief summary of ScaOPBS/OpenPBS commands:

  • scasub - a wrapper of qsub for easy submission of ScaMPI and MPICH parallel jobs.
  • qsub - submit pbs job
  • qstat - show status of pbs batch jobs
  • pbsnodes - pbs node manipulation
  • qdel - delete pbs batch job
  • qmgr - pbs batch system manager

  Please see man pages for each command for detailed description and options.

Back to Table of Contents

Monitoring Jobs

Since users play an active role in selecting the nodes for their jobs and monitoring their jobs while they are running there exists several tools to view the current status of the beowulf.

• scadesktop : A GUI tool provided by Scali that can be used to submit jobs, monitor nodes status, as well as manage the beowulf system. Please see Chapter 5 and 6 of Scali System Guide (PDF) for more details.
• scaconftool : The command line equivalent of scadesktop. Please see Chapter 7 of Scali System Guide (PDF) for more details.
• our_node_status: Simple script to rsh to all Primack nodes and view what processes are running.
• all_node_status: Same as our_node_status, only queries the entire beowulf.

Back to Table of Contents

Visualization

There is a visualization machine called twopiter. Please contact Shawfeng Dong (dong@ucolick.org) for information regarding twopiter.

Back to Table of Contents

FAQ's

Q: How do I change my login shell?
A: To change shell type, "chsh", at the current shell prompt and type the appropriate responses.

Q: What about graphics and visualization?
A: No graphics are allowed on the compute nodes due to throughput and security considerations. Thus there are no graphics applications or X11 libraries on the compute nodes. Forwarding of packets from the compute nodes on the internal network is not allowed beyond the frontend. X11 forwarding is not allowed. Visualization is to be done on the Visualization workstation so kindly provided by Joel Primack. Please contact Shawfeng Dong for information regarding this facility (dong@ucolick.org).

Q: Can I put some 3rd party software on UpsAnd?
A: Due to manpower considerations installation and maintenance of 3rd party software are the responsibility of the user. The software and supporting libraries must be installable in the user's own directory. Installation of plotting, graphics viewers, and other visualization packages are not encouraged, even on the frontend, since this leads to performance issues that impact the activity of other users.

Back to Table of Contents