Split disp-s1 out from dolphin (#122)

* start outlining change of the split

* remove modules and tests for PGE/netcdf product

* remove pge/product references from `s1_disp`

also move 2 functiosn for `disp-s1` into `_utils`

* remove isce3/pillow/h5netcdf

* add file for unwrapping optinal reqs

* update specfile and extra deps

* rename `scratch` to `work` diretory, remove `output/`

* remove benchmark/release scripts added to `disp-s1`

* add `py.typed` to package to indicate that package is typed

see isce-framework/tophu#23 and https://peps.python.org/pep-0561/#packaging-type-information

* add a check for unwrap optional deps in `stitch_and_unwrap`

* match readme to getting started

* update to setup-micromamba, skip macos/latest for now

* go back to no label on os

* add back in ubuntu

.github/workflows/test-build-push.yml

@@ -14,24 +14,19 @@ jobs:
   pytest:
     strategy:
       matrix:
-        os:
-          - label: Linux
-            runner: ubuntu-latest
-          - label: macOS
-            runner: macos-latest
-
+        os: [ubunutu-latest, macos-latest]
         deps:
           - label: Latest
-            spec: ""
+            spec: >-
+              isce3
           - label: Minimum
-            spec: |
+            spec: >-
               python=3.8
+              isce3
               gdal=3.5
               h5py=3.6
-              h5netcdf=1.0
               numpy=1.20
               numba=0.54
-              pillow==7.0
               pydantic=2.1
               pymp-pypi=0.4.5
               pyproj=3.3
@@ -40,23 +35,29 @@ jobs:
               scipy=1.5
               shapely=1.8
               threadpoolctl>=3.0
+        exclude:  # TODO: Remove this once pymp is gone
+          - os: macos-latest
+            deps:
+              label: Latest
 
       fail-fast: false
-    name: ${{ matrix.os.label }} • ${{ matrix.deps.label }}
-    runs-on: ${{ matrix.os.runner }}
+    name: ${{ matrix.os }} • ${{ matrix.deps.label }}
+    runs-on: ${{ matrix.os }}
     defaults:
       run:
         shell: bash -l {0}
     steps:
       - name: Checkout
         uses: actions/checkout@v3
       - name: Setup environment
-        uses: mamba-org/provision-with-micromamba@main
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: conda-env.yml
           environment-name: dolphin-env
-          extra-specs: ${{ matrix.deps.spec }}
-          channels: conda-forge
+          create-args: ${{ matrix.deps.spec }}
+          condarc: |
+            channels:
+              - conda-forge
       - name: Install
         run: |
           pip install --no-deps .
@@ -81,10 +82,10 @@ jobs:
         uses: actions/checkout@v3
         with:
           fetch-depth: 0
-      - name: Set up Python 3.10
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: '3.10'
+          python-version: '3.11'
       - name: Set environment variables for docker build
         run: |
           pip install setuptools_scm # Install setuptools_scm to get version number

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Unreleased
 
+**Changed**
+
+- Split apart OPERA-specific needs from more general library/workflow functionality
+- Removed the final NetCDF product creation
+  - Many rasters in the `scratch/` folder are of general interest after running the workflow
+  - Changed folder structure so that there's not longer a top-level `scratch/` and `output/` by default
+- Changed the required dependencies so the `isce3` unwrapper is optional, as people may wish to implement their own custom parallel unwrapping
+
+**Dependencies**
+
+Dropped:
+- h5netcdf
+- pillow
+
+Now optional:
+- isce3 (for unwrapping)
+
 # [0.3.0](https://github.com/opera-adt/dolphin/compare/v0.2.0...v0.3.0) - 2023-08-23
 
 **Added**

MANIFEST.in

@@ -1,2 +1,3 @@
 include src/dolphin/shp/glrt_cutoffs.csv
 include src/dolphin/shp/kld_cutoffs.csv
+include src/dolphin/py.typed

README.md

@@ -12,11 +12,19 @@ High resolution wrapped phase estimation for InSAR using combined PS/DS processi
 `dolphin` is available on conda:
 
 ```bash
+# if mamba is not already installed: conda install -n base mamba
 mamba install -c conda-forge dolphin
 ```
-
 (Note: [using `mamba`](https://mamba.readthedocs.io/en/latest/mamba-installation.html#mamba-install) is recommended for conda-forge packages, but miniconda can also be used.)
 
+
+`dolphin` has the ability to unwrap interferograms, but requires the optional dependency of `isce3` to use the python bindings to [SNAPHU](https://web.stanford.edu/group/radar/softwareandlinks/sw/snaphu/).
+To install both dolphin and isce3 through conda-forge, run
+```bash
+mamba install -c conda-forge isce3 dolphin
+```
+
+
 To install locally:
 
 1. Download source code:

conda-env-unwrapping.yml

@@ -0,0 +1,9 @@
+# Can be used to update the environment with the following command:
+#
+#     conda env update --name dolphin-env --file conda-env-unwrapping.yml
+# https://docs.conda.io/projects/conda/en/latest/commands/update.html
+channels:
+  - conda-forge
+dependencies:
+  - isce3>=0.14
+  # - tophu  # Once tophu added to conda forge, this will be installed

conda-env.yml

@@ -5,14 +5,12 @@ dependencies:
   - python>=3.8
   - pip>=21.3  # https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#editable-installation
   - git  # for pip install, due to setuptools_scm
-  - isce3        # >=0.14.0 # Right now, isce3 is messes up conda's solvers. Should move to optional.
+  # - isce3        # isce3 for unwrapping has been moved to optional
   - gdal>=3.3
   - h5py>=3.6
   - hdf5!=1.12.2 # https://github.com/SciTools/iris/issues/5187 and https://github.com/pydata/xarray/issues/7549
-  - h5netcdf>=1.0
   - numba>=0.54
   - numpy>=1.20
-  - pillow>=7.0
   - pydantic>=2.1
   - pymp-pypi>=0.4.5
   - pyproj>=3.3

docker/create-lockfile.sh

@@ -5,42 +5,82 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-readonly HELP='usage: ./create-lockfile.sh ENVFILE > specfile.txt
+readonly HELP='usage: ./create-lockfile.sh --file ENVFILE [--pkgs PACKAGE ...] > specfile.txt
 
-Create a conda lockfile from an environment YAML file for reproducible
-environments.
-
-positional arguments:
-ENVFILE     a YAML file containing package specifications
+Create a conda lockfile from an environment YAML file and additional packages for reproducible environments.
 
 options:
--h, --help  show this help message and exit
+--file ENVFILE    Specify a YAML file containing package specifications.
+--pkgs PACKAGE    Specify additional packages separated by spaces. Example: --pkgs numpy scipy
+-h, --help        Show this help message and exit
 '
 
-main() {
-    # Get absolute path of input YAML file.
-    local ENVFILE
-    ENVFILE=$(realpath "$1")
+install_packages() {
+    local ENVFILE=$(realpath "$1")
+    shift
+    local PACKAGES="$@"
+
+    # Prepare arguments for the command
+    local FILE_ARG="--file /tmp/$(basename "$ENVFILE")"
+    local PKGS_ARGS=(${PACKAGES[@]})
 
     # Get concretized package list.
     local PKGLIST
-    PKGLIST=$(docker run --network=host \
-        -v "$ENVFILE:/tmp/environment.yml:ro" --rm \
-        mambaorg/micromamba:1.1.0 bash -c '\
-            micromamba install -y -n base -f /tmp/environment.yml > /dev/null && \
-            micromamba env export --explicit')
+    PKGLIST=$(docker run --rm --network=host \
+        -v "$ENVFILE:/tmp/$(basename "$ENVFILE"):ro" \
+        mambaorg/micromamba:1.1.0 bash -c "\
+            micromamba install -y -n base $FILE_ARG ${PKGS_ARGS[*]} > /dev/null && \
+            micromamba env export --explicit")
 
     # Sort packages alphabetically.
     # (The first 4 lines are assumed to be header lines and ignored.)
-    echo "$PKGLIST" | (sed -u 4q; sort)
+    echo "$PKGLIST" | (
+        sed -u 4q
+        sort
+    )
+}
+
+main() {
+    local ENVFILE=""
+    local PACKAGES=()
+
+    while [[ "$#" -gt 0 ]]; do
+        case $1 in
+        --file)
+            shift
+            if [[ -z "${1-}" ]]; then
+                echo "No file provided after --file" >&2
+                exit 1
+            fi
+            ENVFILE="$1"
+            shift
+            ;;
+        --pkgs)
+            shift
+            while [[ "$#" -gt 0 && ! "$1" =~ ^-- ]]; do
+                PACKAGES+=("$1")
+                shift
+            done
+            ;;
+        -h | --help)
+            echo "$HELP"
+            exit 0
+            ;;
+        *)
+            echo "Unknown option: $1" >&2
+            echo "$HELP"
+            exit 1
+            ;;
+        esac
+    done
+
+    if [[ -z "$ENVFILE" ]]; then
+        echo 'No environment file provided' >&2
+        echo "$HELP"
+        exit 1
+    fi
+
+    install_packages "$ENVFILE" "${PACKAGES[@]}"
 }
 
-if [[ "${1-}" =~ ^-*h(elp)?$ ]]; then
-    echo "$HELP"
-elif [[ "$#" -ne 1 ]]; then
-    echo 'Illegal number of parameters' >&2
-    echo "$HELP"
-    exit 1
-else
-    main "$@"
-fi
+main "$@"