Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

samples: add batch request sample and test #656

Merged
merged 4 commits into from
Jan 5, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions samples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ for instructions on setting up credentials for applications.
<summary><b>List of Samples</b></summary>

* [Activate HMAC Key](#activate-hmac-key)
* [Batch Request](#batch-request)
* [Add Bucket Conditional IAM Binding](#add-bucket-conditional-iam-binding)
* [Add Bucket Default Owner](#add-bucket-default-owner)
* [Add Bucket IAM Member](#add-bucket-iam-member)
Expand Down Expand Up @@ -158,6 +159,15 @@ View the [source code](https://github.com/googleapis/python-storage/blob/main/sa

`python storage_activate_hmac_key.py <ACCESS_ID> <PROJECT_ID>`

-----
### Batch Request
[![Open in Cloud Shell][shell_img]](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/googleapis/python-storage&page=editor&open_in_editor=samples/snippets/storage_batch_request.py,samples/README.md)

View the [source code](https://github.com/googleapis/python-storage/blob/main/samples/snippets/storage_batch_request.py). To run this sample:


`python storage_batch_request.py <BUCKET_NAME> <PREFIX>`

-----

### Add Bucket Conditional IAM Binding
Expand Down
15 changes: 15 additions & 0 deletions samples/snippets/snippets_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import requests

import storage_add_bucket_label
import storage_batch_request
import storage_bucket_delete_default_kms_key
import storage_change_default_storage_class
import storage_change_file_storage_class
Expand Down Expand Up @@ -509,3 +510,17 @@ def test_storage_configure_retries(test_blob, capsys):
assert "The following library method is customized to be retried" in out
assert "_should_retry" in out
assert "initial=1.5, maximum=45.0, multiplier=1.2, deadline=500.0" in out


def test_batch_request(test_bucket):
blob1 = test_bucket.blob("b/1.txt")
blob2 = test_bucket.blob("b/2.txt")
blob1.upload_from_string("hello world")
blob2.upload_from_string("hello world")

storage_batch_request.batch_request(test_bucket.name, "b/")
blob1.reload()
blob2.reload()

assert blob1.metadata.get("your-metadata-key") == "your-metadata-value"
assert blob2.metadata.get("your-metadata-key") == "your-metadata-value"
59 changes: 59 additions & 0 deletions samples/snippets/storage_batch_request.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
#!/usr/bin/env python

# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import sys

"""Sample that uses a batch request.
This sample is used on this page:
https://cloud.google.com/storage/docs/batch
For more information, see README.md.
"""

# [START storage_batch_request]

from google.cloud import storage


def batch_request(bucket_name, prefix=None):
"""Use a batch request to patch a list of objects with the given prefix in a bucket."""
# The ID of your GCS bucket
# bucket_name = "my-bucket"
# The prefix of the object paths
# prefix = "directory-prefix/"

client = storage.Client()
bucket = client.bucket(bucket_name)

# Accumulate in a list the objects with a given prefix.
blobs_to_patch = [blob for blob in bucket.list_blobs(prefix=prefix)]

# Use a batch context manager to edit metadata in the list of blobs.
# The batch request is sent out when the context manager closes.
with client.batch():
for blob in blobs_to_patch:
metadata = {"your-metadata-key": "your-metadata-value"}
blob.metadata = metadata
blob.patch()

print(
f"Batch request edited metadata for all objects with the given prefix in {bucket.name}."
)


# [END storage_batch_request]

if __name__ == "__main__":
batch_request(bucket_name=sys.argv[1], prefix=sys.argv[2])