Skip to content

Commit

Permalink
Add yaml as a Webdataset test dependency, adjust to new WDS API
Browse files Browse the repository at this point in the history
- the latest webdataset python package caries silent yaml
  dependency what break DALI test. This PR adds a missing package
  to the installation process.
- removes a usage of the `shardshuffle` parameters in webdataset
  example as it is no longer available in webdataset API

Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
  • Loading branch information
JanuszL committed Aug 30, 2021
1 parent 3fb2a83 commit 63caea8
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion docs/examples/use_cases/webdataset-externalsource.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@
" \n",
" def webdataset_generator():\n",
" bytes_np_mapper = (lambda data: np.frombuffer(data, dtype=np.uint8),)*len(extensions)\n",
" dataset_instance = (wds.WebDataset(paths, shardshuffle=False)\n",
" dataset_instance = (wds.WebDataset(paths)\n",
" .to_tuple(*extensions)\n",
" .map_tuple(*bytes_np_mapper))\n",
" \n",
Expand Down
2 changes: 1 addition & 1 deletion qa/TL0_jupyter/test_pytorch.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash -e
# used pip packages
pip_packages="pillow jupyter numpy matplotlib torch torchvision webdataset"
pip_packages="pillow jupyter numpy matplotlib torch torchvision webdataset pyyaml"
target_dir=./docs/examples

test_body() {
Expand Down
2 changes: 1 addition & 1 deletion qa/TL1_jupyter_conda/test_pytorch.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash -e
# used pip packages
pip_packages="pillow jupyter numpy matplotlib torch torchvision webdataset"
pip_packages="pillow jupyter numpy matplotlib torch torchvision webdataset pyyaml"
target_dir=./docs/examples

# populate epilog and prolog with variants to enable/disable conda
Expand Down

0 comments on commit 63caea8

Please sign in to comment.