-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[request] HPC filesystem to Cloud #924
Labels
enhancement
New feature or request
Comments
Hi @TomNicholas - thanks for the suggestion! I believe this should be possible if a @lynnliu030 could VM-to-VM transfers be extended to support on-prem? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Please describe problem to be solved
A lot of scientists (particularly Climate Scientists) are now sharing their data via public cloud buckets (e.g. ERA-5 on GCP, see this paper for rationale). However, transferring data from large numerical simulations produced on HPC systems is often really challenging. I want to use skyplane to transfer large amounts of data from a HPC local filesystem to the cloud.
(Optional): Suggest a solution
I want to be able to use Skyplane locally on a HPC filesystem (e.g. Lustre) to transfer TBs (or even PBs) of data to the cloud. Even better would be if I could call Skyplane from Python, to integrate with existing tooling for moving scientific datasets to the cloud (which currently can only pull data from other public data portals elsewhere on the internet.)
The text was updated successfully, but these errors were encountered: