Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(blob): Add multipart option #517

Merged
merged 33 commits into from
Jan 12, 2024
Merged

feat(blob): Add multipart option #517

merged 33 commits into from
Jan 12, 2024

Conversation

vvo
Copy link
Member

@vvo vvo commented Dec 7, 2023

This PR adds a new option to the put() method: multipart: boolean.

Multipart uploads are useful when uploading a big file to Vercel Blob.
When you use multipart: true in a put() call. Then:

  • We turn the body into a (web) stream.
  • We read the stream part by part, buffering into memory only what we need to maximize network utilization.

This current PR is an advanced proof of concept and needs some adjustments.

TODO:

  • Test in a browser (might need to adjust import from 'stream')
  • Test in an edge function (might need to adjust import form 'stream')
  • Add a retry mechanism in fetch when uploading parts
  • Feedback and validate API
  • Feedback and validate implementation
  • Unit tests + errors handling (stream errors) + better handling of cancelling multipart uploads
  • Integration tests

Not in scope:

  • configurable retries, there will be a default algorithm
  • configurable concurrent uploads (and memory), there is a default a algorithm

@vvo vvo self-assigned this Dec 7, 2023
Copy link

changeset-bot bot commented Dec 7, 2023

🦋 Changeset detected

Latest commit: 1b7a841

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@vercel/blob Minor
vercel-storage-integration-test-suite Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link

vercel bot commented Dec 7, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
vercel-storage-next-integration-test-suite ✅ Ready (Inspect) Visit Preview Jan 12, 2024 1:37pm

@@ -59,7 +59,6 @@
}
},
"dependencies": {
"jest-environment-jsdom": "29.7.0",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why it was here in the first place

@vvo vvo enabled auto-merge (squash) January 12, 2024 13:39
@vvo vvo merged commit 898c14a into main Jan 12, 2024
13 checks passed
@vvo vvo deleted the feat/blob/multipart branch January 12, 2024 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants