Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Saving Parcels output directly in zarr format #1199

Merged
merged 89 commits into from
Oct 5, 2022
Merged

Saving Parcels output directly in zarr format #1199

merged 89 commits into from
Oct 5, 2022

Commits on Jul 12, 2022

  1. SIgnificant simplification of baseparticlefile by using ds.to_zarr()

    @erikvansebille
    erikvansebille committed Jul 12, 2022
    Configuration menu
    Copy the full SHA
    f232cf1 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2022

  1. Updating particlefile.write() to support deleting particles

    @erikvansebille
    erikvansebille committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    f08fea6 View commit details
    Browse the repository at this point in the history

  2. Updating particlefilesoa and -aos to new simpler writing

    @erikvansebille
    erikvansebille committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    f354d1a View commit details
    Browse the repository at this point in the history

  3. Updating output-logger message in pset.execute()

    @erikvansebille
    erikvansebille committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    831ca7c View commit details
    Browse the repository at this point in the history

  4. Updating test_particle_file to zarr dumping (first part)

    @erikvansebille
    erikvansebille committed Jul 13, 2022
    Configuration menu
    Copy the full SHA
    684d9cc View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2022

  1. Update baseparticlefile.py

    @erikvansebille
    erikvansebille committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    084a68d View commit details
    Browse the repository at this point in the history

  2. Further additions to test_particle_file

    @erikvansebille
    erikvansebille committed Jul 15, 2022
    Configuration menu
    Copy the full SHA
    8101eb8 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2022

  1. Updating dump_to_zarr to support adding and removing particles

    @erikvansebille
    erikvansebille committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    762a4ea View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2022

  1. dump_to_zarr passing all tests in test_particle_file

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    d5554a9 View commit details
    Browse the repository at this point in the history

  2. Vecorising the dump_to_zarr

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    97a2950 View commit details
    Browse the repository at this point in the history

  3. Removing convert_npydir_to_netcdf.py script as not needed with dump_t… 

    …o_zarr
    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    a40c110 View commit details
    Browse the repository at this point in the history

  4. Cleanign up particlefilesoa.py and particlefileaos.py

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    6a4f2c4 View commit details
    Browse the repository at this point in the history

  5. Removing old dump_to_zarr attempt via xarray DataSets appending 

    Did not work because can't append in two dimensions at the same time
    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    4ff05ed View commit details
    Browse the repository at this point in the history

  6. Removing conert_at_end keyword from baseparticlefile (as not needed w… 

    …ith zarr output)
    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    8cb7a2d View commit details
    Browse the repository at this point in the history

  7. Cleaning up test_particle_file

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    0e6f5d7 View commit details
    Browse the repository at this point in the history

  8. Fixing dump_to_zarr in unit tests in other function than test_particl… 

    …e_file
    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    1909047 View commit details
    Browse the repository at this point in the history

  9. Update example_mitgcm.py

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    ad70153 View commit details
    Browse the repository at this point in the history

  10. Update parcels_tutorial to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    fe3ae3d View commit details
    Browse the repository at this point in the history

  11. Update NedetedFields tutorial to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    e527c00 View commit details
    Browse the repository at this point in the history

  12. Update tutorial_Argofloats to use zarr

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    8bed36e View commit details
    Browse the repository at this point in the history

  13. Update tutorial_SummedFields.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    01cfbc4 View commit details
    Browse the repository at this point in the history

  14. Update tutorial_delaystart.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    d98efa2 View commit details
    Browse the repository at this point in the history

  15. Update tutorial_Argofloats.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    93e4340 View commit details
    Browse the repository at this point in the history

  16. Update tutorial_diffusion.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    b97c73e View commit details
    Browse the repository at this point in the history

  17. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    65cc361 View commit details
    Browse the repository at this point in the history

  18. Update tutorial_interaction to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    494145e View commit details
    Browse the repository at this point in the history

  19. Update tutorial_interpolation to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    c19e112 View commit details
    Browse the repository at this point in the history

  20. Update tutorial_nemo_curvilinear to use zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    b172e26 View commit details
    Browse the repository at this point in the history

  21. Updating tutorial_plotting to default zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    3837830 View commit details
    Browse the repository at this point in the history

  22. Update tutorial_parcels_structure to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    6859a42 View commit details
    Browse the repository at this point in the history

  23. Update tutorial_periodic_boundaries.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    1177146 View commit details
    Browse the repository at this point in the history

  24. Update tutorial_sampling.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    7cb9a32 View commit details
    Browse the repository at this point in the history

  25. Update tutorial_timevaryingdepthdimensions.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    b91728e View commit details
    Browse the repository at this point in the history

  26. Update tutorial_particle_field_interaction.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    e489036 View commit details
    Browse the repository at this point in the history

  27. Update tutorial_particle_field_interaction.ipynb

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    8c533e1 View commit details
    Browse the repository at this point in the history

  28. Updating Field.to_write for new zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    7678189 View commit details
    Browse the repository at this point in the history

  29. Updating unit test for field.to_write to zarr output

    @erikvansebille
    erikvansebille committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    2a95e29 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2022

  1. Adding support for MPI in dump_to_zarr 

    Note that for simplicity, all writing is done on the rank==0 processor. This could potentially be improved
    @erikvansebille
    erikvansebille committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    04cb9ba View commit details
    Browse the repository at this point in the history

  2. Removing .nc extension in output files from tests, tutorials and exam… 

    …ples

Also generating an error when a filename in ParticleFile has extension '.nc'
    @erikvansebille
    erikvansebille committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    f7b0e6a View commit details
    Browse the repository at this point in the history

  3. Further changes from .nc to .zarr in unitest particlefiles

    @erikvansebille
    erikvansebille committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    ef946ee View commit details
    Browse the repository at this point in the history

  4. Adding chunks argument to ParticleFile

    @erikvansebille
    erikvansebille committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    14b3d03 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2022

  1. Update to chunk treatment in dump_to_zarr 

    Including an error if chunk trajs is too small
    @erikvansebille
    erikvansebille committed Jul 22, 2022
    Configuration menu
    Copy the full SHA
    d31c5f3 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2022

  1. Changing default zarr chunk to 10 timesteps 

    Also adding unit test for chunks in zarr
    @erikvansebille
    erikvansebille committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    d82a67c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    e4ea433 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2022

  1. Cleanup of particlefile.to_write()

    @erikvansebille
    erikvansebille committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    0c63c2c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    fa79b21 View commit details
    Browse the repository at this point in the history

  3. Updating more examples to zarr output

    @erikvansebille
    erikvansebille committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    f2e4da9 View commit details
    Browse the repository at this point in the history

  4. Moving extending zarr dimensions to separate method

    @erikvansebille
    erikvansebille committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    da449d8 View commit details
    Browse the repository at this point in the history

  5. Simplifying data_dicts from collections_toDict

    @erikvansebille
    erikvansebille committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    31b3a6b View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2022

  1. Removing use of toDictionary 

    And also speeding up after dramatic slowdown in previous refactor
    @erikvansebille
    erikvansebille committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    0ef5d96 View commit details
    Browse the repository at this point in the history

  2. Making trajectory a to_write='once' variable by default 

    Also fixing some breaking integration tests and further cleaning up the code
    @erikvansebille
    erikvansebille committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    b997bc3 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2022

  1. Further cleanup of baseparticlefile 

    Also adding argument `create_new_zarrfile` to ParticleFile, to either create (default) or append to file
    @erikvansebille
    erikvansebille committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    cd11094 View commit details
    Browse the repository at this point in the history

  2. Simplifying computation of to_write='once' variable indices

    @erikvansebille
    erikvansebille committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    b2caf85 View commit details
    Browse the repository at this point in the history

  3. Updating baseparticlefile to support mpi

    @erikvansebille
    erikvansebille committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    9e5381b View commit details
    Browse the repository at this point in the history

  4. Further fix to mpi and zarr

    @erikvansebille
    erikvansebille committed Jul 28, 2022
    Configuration menu
    Copy the full SHA
    5bf4955 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2022

  1. Fixing bug in setting fileoffsets when particleset is empty

    @erikvansebille
    erikvansebille committed Jul 29, 2022
    Configuration menu
    Copy the full SHA
    cca1a32 View commit details
    Browse the repository at this point in the history

  2. Fixing bug in MPI-zarr writing where extending ons-dimension could be… 

    … handled by multiple threads
    @erikvansebille
    erikvansebille committed Jul 29, 2022
    Configuration menu
    Copy the full SHA
    381c375 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2022

  1. Cleaning up baseparticlefile

    @erikvansebille
    erikvansebille committed Aug 1, 2022
    Configuration menu
    Copy the full SHA
    ffd3b42 View commit details
    Browse the repository at this point in the history

  2. Renaming particle.fileid to particle.once_written

    @erikvansebille
    erikvansebille committed Aug 1, 2022
    Configuration menu
    Copy the full SHA
    adfd755 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 1, 2022
    Configuration menu
    Copy the full SHA
    0d07501 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 3, 2022
    Configuration menu
    Copy the full SHA
    641ad20 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 4, 2022
    Configuration menu
    Copy the full SHA
    4bac223 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2022

  1. Removing for-loop from baseparticlefile 

    Still needed in aos-mode; but not anymore in soa-mode. So moving for-loop to collectionaos
Also renaming some variables to make their use clearer
    @erikvansebille
    erikvansebille committed Aug 5, 2022
    Configuration menu
    Copy the full SHA
    0667291 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    c2e0965 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2022

  1. Updating test_mpi to also run with 8 particles 

    Currently fails
    @erikvansebille
    erikvansebille committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    229488d View commit details
    Browse the repository at this point in the history

  2. Fixing small bug when chunks < maxids in baseparticlefile

    @erikvansebille
    erikvansebille committed Aug 24, 2022
    Configuration menu
    Copy the full SHA
    b1c5d1e View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    8a18cfb View commit details
    Browse the repository at this point in the history

  2. Updating delaystart tutorial to use zarr output

    @erikvansebille
    erikvansebille committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    18c75c2 View commit details
    Browse the repository at this point in the history

  3. Updating MPI version of zarr writing 

    Simplifying (and speeidng up) the writing to zarr in MPI mode, by letting each processor write to its own file and then combine with `xr.merge()` at the end
    @erikvansebille
    erikvansebille committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    f789075 View commit details
    Browse the repository at this point in the history

  4. Fixing bug when MPI not installed

    @erikvansebille
    erikvansebille committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    6e2c05e View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2022

  1. Making trajectory a coordinate variable in zarr 

    For easier merging of multiple zarr outputs in MPI mode
    @erikvansebille
    erikvansebille committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    dd61b21 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    6a9a32b View commit details
    Browse the repository at this point in the history

  2. Simplifying code in baseparticlefile for findings ids to write

    @erikvansebille
    erikvansebille committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    31ea667 View commit details
    Browse the repository at this point in the history

  3. Adding an obs coordinate to the zarr file

    @erikvansebille
    erikvansebille committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    1a9731d View commit details
    Browse the repository at this point in the history

  4. Add unit test to check if zarr trajectory.dtype is int64

    @erikvansebille
    erikvansebille committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    44b8593 View commit details
    Browse the repository at this point in the history

  5. Fixing bug where trajectory written as float 

    In xarray version 2022.6.0. Thanks for noticing, @JamiePringle!
    @erikvansebille
    erikvansebille committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    4279a20 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2022

  1. Fixing naming of zarr files for MPI

    @erikvansebille
    erikvansebille committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    8a02ccd View commit details
    Browse the repository at this point in the history

  2. Adding info on zarr output in MPI to documentation

    @erikvansebille
    erikvansebille committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    3365460 View commit details
    Browse the repository at this point in the history

  3. Adding info on MPI concatenation to output tutorial

    @erikvansebille
    erikvansebille committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    1f2b2a0 View commit details
    Browse the repository at this point in the history

  4. Updating text of zarr explanation in MPI and output tutorials

    @erikvansebille
    erikvansebille committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    051c8bf View commit details
    Browse the repository at this point in the history

  5. Updating default zarr chunks to 1 obs

    @erikvansebille
    erikvansebille committed Aug 30, 2022
    Configuration menu
    Copy the full SHA
    66807da View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2022

  1. Updating MPI documentation to also import os.path

    @erikvansebille
    erikvansebille committed Aug 31, 2022
    Configuration menu
    Copy the full SHA
    8846861 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2022

  1. Add documentation to combine lMPI run output into single Zarr store 

    Code to combine output from MPI run into a single Zarr file, re-chunk the data, and change variable types, and add variables to the zarr file even for very large output sizes.
    @JamiePringle
    JamiePringle committed Sep 6, 2022
    Configuration menu
    Copy the full SHA
    01bf12f View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2022

  1. Moving documentation_largeoutputfile to parcels/examples folder

    @erikvansebille
    erikvansebille committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    125dc61 View commit details
    Browse the repository at this point in the history

  2. Adding cell on large output runs to mpi documentation 

    Note link will not work yet; only when merged into master
    @erikvansebille
    erikvansebille committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    0b1f4a2 View commit details
    Browse the repository at this point in the history

  3. update documentation to fix xaray datetime64[ns] bug 

    change documentation_MPI.ipynb and documentation_LargeRunsOutput.ipynb to work around slow .to_zarr() of datasets that contain datetime variables.
    @JamiePringle
    JamiePringle committed Sep 13, 2022
    Configuration menu
    Copy the full SHA
    cfd4821 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2022

  1. Merge branch 'master' into dump_to_zarr

    @erikvansebille
    erikvansebille committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    a2dacfd View commit details
    Browse the repository at this point in the history

Commits on Oct 5, 2022

  1. Pinning dask version to 2022.9.0 

    Since this version still worked, and 2022.9.2 gives an error with https://github.com/OceanParcels/parcels/actions/runs/3180136958/jobs/5199374392
    @erikvansebille
    erikvansebille committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    c1f4f36 View commit details
    Browse the repository at this point in the history

  2. Fixing 3D plotting bug in Windows

    @erikvansebille
    erikvansebille committed Oct 5, 2022
    Configuration menu
    Copy the full SHA
    0326f77 View commit details
    Browse the repository at this point in the history