Advanced & Special Topics

Jupyter

For installation, see the JupyterLab section on our Getting Started page.

Kernels

"Kernels are programming language specific processes that run independently and interact with the Jupyter Applications and their user interfaces. ipykernel is the reference Jupyter kernel built on top of IPython, providing a powerful environment for interactive computing in Python" (source).

See links below to install additional versions of Python or other languages like Julia or R:

• Installing the IPython kernel
• Jupyter kernels (e.g., Julia, R)

Miniconda

For installation, see the Miniconda section on our Getting Started page.

Conda environments

By keeping them separate and individually activatable (terminal) or accessible (via kernels in Jupyter), you can tailor conda environments to contain only necessary packages for a particular project and upgrade Python or a package without breaking any working environments for another project.

The NAS Supercomputer Cluster

NASA's High End Computing Capability (HECC) Computing Environment encompasses several different supercomputing clusters, including the Pleiades, Aitken, Endeavour, and Electra supercomputers. Our group has access to the Pleiades supercomputer, so it is helpful to know some of the basic commands used around the system.

Detailed information about the HECC system can be found at the HECC Website or at the HECC Knowledge Base. Important information about the basic structure of HECC, the workflow of job submissions, and troubleshooting account details is outlined below.

System Overview

Job Submission and Tracking

HECC Uses the Portable Batch System (PBS) for job submissions. There are several different job queues that are used for job submissions.

The major types of queues used on HECC are:

• devel - 1 job, 2 hrs max
• debug - 2 jobs, 2 hrs max
• normal - unlimited jobs, 8 hrs max
• long - unlimited jobs, 5 days max
• low - unlimited jobs, 4 hrs max

To specify which job queue to use, as well as the number of cores/processors to be used for the job, you can add a header in the PBS input file:

# PBS -q normal

Jobs are submitted using the qsub command and the .pbs file job_file:

qsub <job_file>

The status of current jobs can be checked using qstat, for example:

qstat -u <username>

shows all jobs currently running under your username.

Account Management

The HECC system has some Job Accounting Utilities that can be used to check how much of your computing resources have been used up. While connected to the system, use the acct_ytd or acct_query commands:

acct_ytd

acct_query -b dd/mm/yyyy -e dd/mm/yyyy

acct_ytd lists the year-to-date usage for any GID resources available to you, while acct_query lists detailed information about resource usage (how many resources went towareds each queue, for example). The beginning and end dates to be searched can be specified.

Each user has a limit to the amount of data they are able to store on the system, known as a quota. To check your quota usage on the Pleiades filesystems, use

quota -v

The /nobackup filesystems have a different amount of storage available. To see how much data you are currently using here, use the command

lfs quota -hu <username> /nobackup/<username>

replacing <username> with your username.

For long-term storage of files, NAS recommends moving your files to the Lou system instead.