-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Saving Parcels output directly in zarr format #1199
Commits on Jul 12, 2022
-
Configuration menu - View commit details
-
Copy full SHA for f232cf1 - Browse repository at this point
Copy the full SHA f232cf1View commit details
Commits on Jul 13, 2022
-
Configuration menu - View commit details
-
Copy full SHA for f08fea6 - Browse repository at this point
Copy the full SHA f08fea6View commit details -
Configuration menu - View commit details
-
Copy full SHA for f354d1a - Browse repository at this point
Copy the full SHA f354d1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 831ca7c - Browse repository at this point
Copy the full SHA 831ca7cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 684d9cc - Browse repository at this point
Copy the full SHA 684d9ccView commit details
Commits on Jul 15, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 084a68d - Browse repository at this point
Copy the full SHA 084a68dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8101eb8 - Browse repository at this point
Copy the full SHA 8101eb8View commit details
Commits on Jul 19, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 762a4ea - Browse repository at this point
Copy the full SHA 762a4eaView commit details
Commits on Jul 20, 2022
-
Configuration menu - View commit details
-
Copy full SHA for d5554a9 - Browse repository at this point
Copy the full SHA d5554a9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 97a2950 - Browse repository at this point
Copy the full SHA 97a2950View commit details -
Configuration menu - View commit details
-
Copy full SHA for a40c110 - Browse repository at this point
Copy the full SHA a40c110View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a4f2c4 - Browse repository at this point
Copy the full SHA 6a4f2c4View commit details -
Removing old dump_to_zarr attempt via xarray DataSets appending
Did not work because can't append in two dimensions at the same time
Configuration menu - View commit details
-
Copy full SHA for 4ff05ed - Browse repository at this point
Copy the full SHA 4ff05edView commit details -
Removing conert_at_end keyword from baseparticlefile (as not needed w…
…ith zarr output)
Configuration menu - View commit details
-
Copy full SHA for 8cb7a2d - Browse repository at this point
Copy the full SHA 8cb7a2dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0e6f5d7 - Browse repository at this point
Copy the full SHA 0e6f5d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1909047 - Browse repository at this point
Copy the full SHA 1909047View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad70153 - Browse repository at this point
Copy the full SHA ad70153View commit details -
Configuration menu - View commit details
-
Copy full SHA for fe3ae3d - Browse repository at this point
Copy the full SHA fe3ae3dView commit details -
Configuration menu - View commit details
-
Copy full SHA for e527c00 - Browse repository at this point
Copy the full SHA e527c00View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8bed36e - Browse repository at this point
Copy the full SHA 8bed36eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 01cfbc4 - Browse repository at this point
Copy the full SHA 01cfbc4View commit details -
Configuration menu - View commit details
-
Copy full SHA for d98efa2 - Browse repository at this point
Copy the full SHA d98efa2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 93e4340 - Browse repository at this point
Copy the full SHA 93e4340View commit details -
Configuration menu - View commit details
-
Copy full SHA for b97c73e - Browse repository at this point
Copy the full SHA b97c73eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 65cc361 - Browse repository at this point
Copy the full SHA 65cc361View commit details -
Configuration menu - View commit details
-
Copy full SHA for 494145e - Browse repository at this point
Copy the full SHA 494145eView commit details -
Configuration menu - View commit details
-
Copy full SHA for c19e112 - Browse repository at this point
Copy the full SHA c19e112View commit details -
Configuration menu - View commit details
-
Copy full SHA for b172e26 - Browse repository at this point
Copy the full SHA b172e26View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3837830 - Browse repository at this point
Copy the full SHA 3837830View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6859a42 - Browse repository at this point
Copy the full SHA 6859a42View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1177146 - Browse repository at this point
Copy the full SHA 1177146View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7cb9a32 - Browse repository at this point
Copy the full SHA 7cb9a32View commit details -
Configuration menu - View commit details
-
Copy full SHA for b91728e - Browse repository at this point
Copy the full SHA b91728eView commit details -
Configuration menu - View commit details
-
Copy full SHA for e489036 - Browse repository at this point
Copy the full SHA e489036View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8c533e1 - Browse repository at this point
Copy the full SHA 8c533e1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7678189 - Browse repository at this point
Copy the full SHA 7678189View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a95e29 - Browse repository at this point
Copy the full SHA 2a95e29View commit details
Commits on Jul 21, 2022
-
Adding support for MPI in dump_to_zarr
Note that for simplicity, all writing is done on the rank==0 processor. This could potentially be improved
Configuration menu - View commit details
-
Copy full SHA for 04cb9ba - Browse repository at this point
Copy the full SHA 04cb9baView commit details -
Removing .nc extension in output files from tests, tutorials and exam…
…ples Also generating an error when a filename in ParticleFile has extension '.nc'
Configuration menu - View commit details
-
Copy full SHA for f7b0e6a - Browse repository at this point
Copy the full SHA f7b0e6aView commit details -
Configuration menu - View commit details
-
Copy full SHA for ef946ee - Browse repository at this point
Copy the full SHA ef946eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 14b3d03 - Browse repository at this point
Copy the full SHA 14b3d03View commit details
Commits on Jul 22, 2022
-
Update to chunk treatment in dump_to_zarr
Including an error if chunk trajs is too small
Configuration menu - View commit details
-
Copy full SHA for d31c5f3 - Browse repository at this point
Copy the full SHA d31c5f3View commit details
Commits on Jul 25, 2022
-
Changing default zarr chunk to 10 timesteps
Also adding unit test for chunks in zarr
Configuration menu - View commit details
-
Copy full SHA for d82a67c - Browse repository at this point
Copy the full SHA d82a67cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e4ea433 - Browse repository at this point
Copy the full SHA e4ea433View commit details
Commits on Jul 26, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 0c63c2c - Browse repository at this point
Copy the full SHA 0c63c2cView commit details -
Configuration menu - View commit details
-
Copy full SHA for fa79b21 - Browse repository at this point
Copy the full SHA fa79b21View commit details -
Configuration menu - View commit details
-
Copy full SHA for f2e4da9 - Browse repository at this point
Copy the full SHA f2e4da9View commit details -
Configuration menu - View commit details
-
Copy full SHA for da449d8 - Browse repository at this point
Copy the full SHA da449d8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 31b3a6b - Browse repository at this point
Copy the full SHA 31b3a6bView commit details
Commits on Jul 27, 2022
-
And also speeding up after dramatic slowdown in previous refactor
Configuration menu - View commit details
-
Copy full SHA for 0ef5d96 - Browse repository at this point
Copy the full SHA 0ef5d96View commit details -
Making trajectory a to_write='once' variable by default
Also fixing some breaking integration tests and further cleaning up the code
Configuration menu - View commit details
-
Copy full SHA for b997bc3 - Browse repository at this point
Copy the full SHA b997bc3View commit details
Commits on Jul 28, 2022
-
Further cleanup of baseparticlefile
Also adding argument `create_new_zarrfile` to ParticleFile, to either create (default) or append to file
Configuration menu - View commit details
-
Copy full SHA for cd11094 - Browse repository at this point
Copy the full SHA cd11094View commit details -
Configuration menu - View commit details
-
Copy full SHA for b2caf85 - Browse repository at this point
Copy the full SHA b2caf85View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e5381b - Browse repository at this point
Copy the full SHA 9e5381bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5bf4955 - Browse repository at this point
Copy the full SHA 5bf4955View commit details
Commits on Jul 29, 2022
-
Configuration menu - View commit details
-
Copy full SHA for cca1a32 - Browse repository at this point
Copy the full SHA cca1a32View commit details -
Fixing bug in MPI-zarr writing where extending ons-dimension could be…
… handled by multiple threads
Configuration menu - View commit details
-
Copy full SHA for 381c375 - Browse repository at this point
Copy the full SHA 381c375View commit details
Commits on Aug 1, 2022
-
Configuration menu - View commit details
-
Copy full SHA for ffd3b42 - Browse repository at this point
Copy the full SHA ffd3b42View commit details -
Configuration menu - View commit details
-
Copy full SHA for adfd755 - Browse repository at this point
Copy the full SHA adfd755View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0d07501 - Browse repository at this point
Copy the full SHA 0d07501View commit details
Commits on Aug 3, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 641ad20 - Browse repository at this point
Copy the full SHA 641ad20View commit details
Commits on Aug 4, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 4bac223 - Browse repository at this point
Copy the full SHA 4bac223View commit details
Commits on Aug 5, 2022
-
Removing for-loop from baseparticlefile
Still needed in aos-mode; but not anymore in soa-mode. So moving for-loop to collectionaos Also renaming some variables to make their use clearer
Configuration menu - View commit details
-
Copy full SHA for 0667291 - Browse repository at this point
Copy the full SHA 0667291View commit details
Commits on Aug 22, 2022
-
Configuration menu - View commit details
-
Copy full SHA for c2e0965 - Browse repository at this point
Copy the full SHA c2e0965View commit details
Commits on Aug 24, 2022
-
Updating test_mpi to also run with 8 particles
Currently fails
Configuration menu - View commit details
-
Copy full SHA for 229488d - Browse repository at this point
Copy the full SHA 229488dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b1c5d1e - Browse repository at this point
Copy the full SHA b1c5d1eView commit details
Commits on Aug 25, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 8a18cfb - Browse repository at this point
Copy the full SHA 8a18cfbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 18c75c2 - Browse repository at this point
Copy the full SHA 18c75c2View commit details -
Updating MPI version of zarr writing
Simplifying (and speeidng up) the writing to zarr in MPI mode, by letting each processor write to its own file and then combine with `xr.merge()` at the end
Configuration menu - View commit details
-
Copy full SHA for f789075 - Browse repository at this point
Copy the full SHA f789075View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e2c05e - Browse repository at this point
Copy the full SHA 6e2c05eView commit details
Commits on Aug 26, 2022
-
Making trajectory a coordinate variable in zarr
For easier merging of multiple zarr outputs in MPI mode
Configuration menu - View commit details
-
Copy full SHA for dd61b21 - Browse repository at this point
Copy the full SHA dd61b21View commit details
Commits on Aug 29, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 6a9a32b - Browse repository at this point
Copy the full SHA 6a9a32bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 31ea667 - Browse repository at this point
Copy the full SHA 31ea667View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1a9731d - Browse repository at this point
Copy the full SHA 1a9731dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 44b8593 - Browse repository at this point
Copy the full SHA 44b8593View commit details -
Fixing bug where trajectory written as float
In xarray version 2022.6.0. Thanks for noticing, @JamiePringle!
Configuration menu - View commit details
-
Copy full SHA for 4279a20 - Browse repository at this point
Copy the full SHA 4279a20View commit details
Commits on Aug 30, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 8a02ccd - Browse repository at this point
Copy the full SHA 8a02ccdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3365460 - Browse repository at this point
Copy the full SHA 3365460View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1f2b2a0 - Browse repository at this point
Copy the full SHA 1f2b2a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 051c8bf - Browse repository at this point
Copy the full SHA 051c8bfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 66807da - Browse repository at this point
Copy the full SHA 66807daView commit details
Commits on Aug 31, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 8846861 - Browse repository at this point
Copy the full SHA 8846861View commit details
Commits on Sep 6, 2022
-
Add documentation to combine lMPI run output into single Zarr store
Code to combine output from MPI run into a single Zarr file, re-chunk the data, and change variable types, and add variables to the zarr file even for very large output sizes.
Configuration menu - View commit details
-
Copy full SHA for 01bf12f - Browse repository at this point
Copy the full SHA 01bf12fView commit details
Commits on Sep 13, 2022
-
Configuration menu - View commit details
-
Copy full SHA for 125dc61 - Browse repository at this point
Copy the full SHA 125dc61View commit details -
Adding cell on large output runs to mpi documentation
Note link will not work yet; only when merged into master
Configuration menu - View commit details
-
Copy full SHA for 0b1f4a2 - Browse repository at this point
Copy the full SHA 0b1f4a2View commit details -
update documentation to fix xaray datetime64[ns] bug
change documentation_MPI.ipynb and documentation_LargeRunsOutput.ipynb to work around slow .to_zarr() of datasets that contain datetime variables.
Configuration menu - View commit details
-
Copy full SHA for cfd4821 - Browse repository at this point
Copy the full SHA cfd4821View commit details
Commits on Oct 4, 2022
-
Configuration menu - View commit details
-
Copy full SHA for a2dacfd - Browse repository at this point
Copy the full SHA a2dacfdView commit details
Commits on Oct 5, 2022
-
Pinning dask version to 2022.9.0
Since this version still worked, and 2022.9.2 gives an error with https://github.com/OceanParcels/parcels/actions/runs/3180136958/jobs/5199374392
Configuration menu - View commit details
-
Copy full SHA for c1f4f36 - Browse repository at this point
Copy the full SHA c1f4f36View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0326f77 - Browse repository at this point
Copy the full SHA 0326f77View commit details