Saguaro HPC Computing

Supercomputing at ASU:

Using Saguaro

You will need to email the staff at AC2C Computing center to have a Saguro account added. Once you have your account you can log on via SSH and a Terminal. Mac and Linux-based systems (like Ubuntu) have a Terminal built in. Windows is more complicated, but you can download some programs for ssh. Also, you can download VMWare and run a virtual drive with Ubuntu from a Windows operating system. This takes a bit of time, but I recommend it, because its nice to use. 

To log onto Saguaro, open up a Terminal and type:

ssh username@saguaro.fulton.asu.edu

password ************

To find your way around you can type cd and ls and cd .. to navigate and peek at the files in the folder directory. You will probably want to set up new folders to begin with. To make a new folder type: mkdir new_folder.

I recommend using lower case letters and avoid using spaces, instead_using_underscores. To find hidden files, like your .bash_profile file on your home Saguro folder, type:  ls -a.

To set permissions on folders or files:     

ls -l      chmod 700 file    

You will see something like:

rwxr-xr-x username user bio.pl

An example of the permissions encoded might be rwxr-xr-x. What this means is that the owner can do anything with the file, but group owners and the rest of the world can only read or execute it. (w-write,xr-execute and read,x-execute).

There is an easy way to do this and a harder way. Just memorize the permissions codes. Read access is the number 4, write is 2, and execute is 1 (4+2+1=7). So for you and you only, 7 is the permission you want for yourself.

chmod 700   bio.txt      Only you can read, write to, or execute bio.txt

chmod 777   bio.txt      Everybody can read, write to, or execute bio.txt

chmod 744   bio.txt      Only you can read, write to, or execute bio.txt. Everybody can read bio.txt

If you want to set permissions for an entire folder type:

chmod  -R   700   /home/username/folder_name OR if you are already in the directory where the folder is

chmod  -R   700   folder_name

The hard way is to learn the permissions coding system, which I will go ahead and tell you: Every permissions mode has a corresponding code number, and one number that corresponds to any mode. Every one of the three digits on the mode number corresponds to one of the three permission triplets. (u, g and o) Every permission bit in a triplet corresponds to a value: 4 for r, 2 for w, 1 for x. If the permission bit you add this value to the number of the permission triplet. So if a file has rwxr-xr-x permissions we do the following calculation:

Triplet for u: rwx => 4 + 2 + 1 = 7  (user)

Triplet for g: r-x => 4 + 0 + 1 = 5  (group owners)

Triplet for o: r-x => 4 + 0 + 1 = 5  (world)

That gives: 755. In other words, 755 is a code way to say 'Other people can read or run this file, but only I should be able to modify it' and 777 means 'everyone has full access to this file.'

#	Permission	rwx
7	full	111
6	read and write	110
5	read and execute	101
4	read only	100
3	write and execute	011
2	write only	010
1	execute only	001
0	none	000

It's a good idea to get used setting permissions correctly, so I recommend making a new directory, and a few new files to get used to it.

mkdir test

cd test

emacs test1.txt

ctrl+x and ctrl+c

cp test1.txt test2.txt

cp test2.txt text3.txt

ls -l

chmod 700 test1.txt

chmod 755 test2.txt

ls -l

cd ..

chmod  -R   700   test

cd test

ls -l

On Saguro, you want to exclude other users from accessing your files openly. Become familiar with the read-write-execute permissions. But BE CAREFUL!!!!! because if you set the permission wrong, you may lock yourself out of your own files!!!

Moving Large Files:

When you are transferring large files, you will need to use Datamover. If you do not, you will be warned by the computing staff about locking up the logon nodes. You will need to go to system preferences and find out your IP and your logon name,

To download stuff from Saguaro back to your local computer:

First, ssh into Saguaro. Then type:

scp  name@datamover1.fulton.asu.edu:/home/username/file.txt  username@111.111.11:/Users/username/Desktop/

name@datamover1.fulton.asu.edu's password:

Password: **********

file.txt                                                               100%  385KB 385.0KB/s   00:00

Use -r if you want to move a whole folder. Alternatively, you can compress or tar the file before moving it.

scp -r   name@datamover1.fulton.asu.edu:/scratch/username   /Volumes/scratch/

To download stuff from the local computer up to Saguaro:

From your local Terminal

scp /Users/username/Desktop/file.txt   username@datamover1.fulton.asu.edu:/home/username

Password: ******

file.txt                                                               100%  385KB 385.0KB/s   00:00

Don't forget that instead of typing out a long file path name, you can drag the file over to Terminal and it will record the path, after your cursor location. This only works locally, of course.

Okay, there are two spaces where you will store your stuff on Saguaro. Use a : after datamover to specify where you want the files to go.

1) At your home folder

    ex. /home/username/progs

2) On your scratch space

    ex. /scratch/username/25Jun

You don't want to datamove huge files to your home drive, since space is limited. You can only store large drives on your scratch space for a few months at a time. You will usually receive a notice every 30 days that the scratch space will be purged. 

To submit jobs to saguaro, you will need a script. This is just a text file that will tell the job how to run. You can literally copy this into a text file and save and use on Saguaro.   

script.txt:

--------------------------------------------------------------

#!/bin/bash

#PBS -l nodes=12

#PBS -l walltime=96:00:00

#PBS -l nodeset=ANYOF:FEATURE:westmere

#PBS -j oe

#PBS -o /scratch/username/aapl49

module load python

cd /scratch/username2/alignments/Acar_genome

bowtie2-build Anolis_genome.fa Anolis_genome

exit

----------------------------------------------------------------

The -o is the output directory. The nodeset can be set to other machines, or left to random. Check the Saguro .pdf for more info. Sometimes you will need modules (preinstalled software on Saguaro). The max time for a job is 96 hours. Make sure that you cd into the correct folder, or if your files aren't in the same folder that you specify the full path name to each file.

Often you will need to edit these scripts right on Saguro. Do this with the emacs command. Enter the directory of the file you want to edit. Then type:

emacs script.txt

Emacs is important to use, so its best to Google Emacs commands and get used to it! To save a quit type control-x and control-c at the same time. To suspend the shell, you can type control+z. To save as you can control+w, then typing the name of the file after the file path before hitting enter. Also, the terminal will only refresh if all windows are closed, then you open it again. If you want to refresh it with your window still open, type: source ~/.bash_profile.

When you are ready to submit a job, type:

qsub script.txt

To check on your job you can type either:

qstat -a

watch qstat -a

You can note the job ID number from qstat -a.

To watch an activity monitor ranked by CPU used:

top -o cpu

Often, your job will be in the queue and may take a while. To see what jobs are running before you, you can type:

showq 

pbsnodes  (also gives you information about what nodes are being used)

Often you will need to check if a job is running correctly. You can collect information from:

qpeek jobnumber

qmem job number

For example:

qmem 5102197

+---------+---------------------+---------------------+---------------------+

| Host    |   RAM (Used/Total)           Swap (Used/Total)   | Load Average |

+---------+---------------------+---------------------+---------------------+

|   s6-6  |   15.57GB / 15.67GB |       7.23GB / 7.81GB |    2.56, 1.76, 1.56 |

|  s21-10 |  751.73MB / 15.67GB |    30.93MB / 7.81GB |  0.00, 0.00, 0.00 |

+---------+---------------------+---------------------+---------------------+

If you see gigabytes of Swap being used, this is not a good sign and you should cancel the job and try to run it with more processors, nodes or on the very large SMP machine.

To run a job on the SMP (32-core) machine, in the script just specify:

#PBS   -q smp

To delete a job:

qdel jobnumber

ex. qdel 477551

To hold a job:

qhold jobnumber

To release a hold:

qrls jobnumber