Skip to content

Commit

Permalink
Adding documentation for cache subcommand
Browse files Browse the repository at this point in the history
  • Loading branch information
skchronicles committed Jul 7, 2023
1 parent 80ac6ac commit 9f1d1ed
Showing 1 changed file with 26 additions and 22 deletions.
48 changes: 26 additions & 22 deletions docs/usage/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,34 +3,34 @@
## 1. About
The `mir-seek` executable is composed of several inter-related sub commands. Please see `mir-seek -h` for all available options.

This part of the documentation describes options and concepts for <code>mir-seek <b>cache</b></code> sub command in more detail. With minimal configuration, the **`cache`** sub command enables you to cache remote resources for the mir-seek pipeline. Caching remote resources allows the pipeline to run in an offline mode. The cache sub command can also be used to pull our pre-built reference bundles onto a new cluster or target system.
This part of the documentation describes options and concepts for <code>mir-seek <b>cache</b></code> sub command in more detail.

The cache sub command creates local cache on the filesysytem for resources hosted on DockerHub or AWS S3. These resources are normally pulled onto the filesystem when the pipeline runs; however, due to network issues or DockerHub pull rate limits, it may make sense to pull the resources once so a shared cache can be created and re-used. It is worth noting that a singularity cache cannot normally be shared across users. Singularity strictly enforces that its cache is owned by the user. To get around this issue, the cache subcommand can be used to create local SIFs on the filesystem from images on DockerHub.
With minimal configuration, the **`cache`** sub command enables you to cache remote software containers from [Dockerhub](https://hub.docker.com/u/skchronicles). Caching remote software containers allows the pipeline to run in an offline mode where no requests are made. The cache sub command can also be used to pull our pre-built software container onto a new cluster or target system.

## 2. Synopsis
These containers are normally pulled onto the filesystem when the pipeline runs; however, due to network issues or DockerHub pull rate limits, it may make sense to pull the resources once so a shared cache can be created. It is worth noting that a singularity cache cannot normally be shared across users. Singularity strictly enforces that a cache is owned by the user. To get around this issue, the cache subcommand can be used to create local SIFs on the filesystem from images on DockerHub. The path of these locally cached SIFs can be passed to the run sub commands --sif-cache option.

Coming Soon!
Caching software containers is fast and easy! In its most basic form, <code>mir-seek <b>cache</b></code> only has *one required input*.

<!-- ```text
$ ./mir-seek cache [-h] --sif-cache SIF_CACHE \
[--resource-bundle RESOURCE_BUNDLE] \
[--dry-run]
## 2. Synopsis
```text
$ ./mir-seek cache [--help] [--dry-run] \
--sif-cache SIF_CACHE
```

The synopsis for each command shows its parameters and their usage. Optional parameters are shown in square brackets.

A user **must** provide a directory to cache remote Docker images via the `--sif-cache` argument. Once the cache has pipeline completed, the local sif cache can be passed to the `--sif-cache` option of the <code>mir-seek <b>build</b></code> and <code>mir-seek <b>run</b></code> subcomand. This enables the build and run pipeline to run in an offline mode.
A user **must** provide a directory to cache remote Docker images via the `--sif-cache` argument. Once the cache has pipeline completed, the local sif cache can be passed to the `--sif-cache` option of the <code>mir-seek <b>run</b></code> subcomand. This enables the pipeline to run in an offline mode.

Use you can always use the `-h` option for information on a specific command.

### 2.1 Required Arguments

`--sif-cache SIF_CACHE`

> **Path where a local cache of SIFs will be stored..**
> *type: string*
> **Path where a local cache of SIFs will be stored.**
> *type: path*
>
> Any images defined in *config/containers/images.json* will be pulled into the local filesystem. The path provided to this option can be passed to the `--sif-cache` option of the <code>mir-seek <b>build</b></code> and <code>mir-seek <b>run</b></code> subcomand. This allows for running the build and run pipelines in an offline mode where no requests are made to external sources. This is useful for avoiding network issues or DockerHub pull rate limits. Please see mir-seek build and run for more information.
> Any images defined in *config/containers.json* will be pulled into the local filesystem. The path provided to this option can be passed to the `--sif-cache` option of the <code>mir-seek <b>run</b></code> subcomand. This allows for running the build and run pipelines in an offline mode where no requests are made to external sources. This is useful for avoiding network issues or DockerHub pull rate limits. Please see mir-seek run for more information.
>
> ***Example:*** `--sif-cache /data/$USER/cache`
Expand All @@ -40,7 +40,7 @@ Each of the following arguments are optional and do not need to be provided.

`-h, --help`
> **Display Help.**
> *type: boolean*
> *type: boolean flag*
>
> Shows command's synopsis, help message, and an example command
>
Expand All @@ -49,9 +49,9 @@ Each of the following arguments are optional and do not need to be provided.
---
`--dry-run`
> **Dry run the pipeline.**
> *type: boolean*
> *type: boolean flag*
>
> Displays what steps in the pipeline remain or will be run. Does not execute anything!
> Only displays what software container will be cached locally. Does not execute anything!
>
> ***Example:*** `--dry-run`
Expand All @@ -62,11 +62,15 @@ srun -N 1 -n 1 --time=12:00:00 -p interactive --mem=8gb --cpus-per-task=4 --pty
module purge
module load singularity snakemake

# Step 1.) Dry run cache to see what will be pulled
./mir-seek cache --sif-cache /scratch/$USER/cache \
--dry-run
# Step 2.) Cache remote resources locally
./mir-seek cache --sif-cache /scratch/$USER/cache
# Step 1.) Dry run to see what will be pulled
./mir-seek cache --sif-cache /data/$USER/cache \
--dry-run

# Step 2.) Cache remote resources locally.
# This command will NOT automatically submit
# a job to the cluster. As so, we recommend
# submitting this next command to the cluster
# as a job. Download speeds will vary so it
# is best to set the wall time a few hours.
./mir-seek cache --sif-cache /data/$USER/cache
```
-->

0 comments on commit 9f1d1ed

Please sign in to comment.