Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perfstress][Storage] Added Datalake perf tests #15861

Merged
merged 7 commits into from
Mar 3, 2021
Merged

[Perfstress][Storage] Added Datalake perf tests #15861

merged 7 commits into from
Mar 3, 2021

Conversation

annatisch
Copy link
Member

No description provided.

@ghost ghost added the Storage Storage Service (Queues, Blobs, Files) label Dec 18, 2020
@kasobol-msft
Copy link
Contributor

@jaschrep-msft

@jaschrep-msft
Copy link
Member

I see no concerns.

mikeharder
mikeharder requested changes Dec 21, 2020
View reviewed changes
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/append.py Show resolved Hide resolved
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/append.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/append.py

def run_sync(self):
data = RandomStream(self.args.size)
self.file_client.append_data(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can appending to the same file over and over cause issues with service perf or correctness (e.g. is there a max file size)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will check in with the Storage team to see if they can shed light on that

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be no problems. When you keep re-appending, you would be rewriting the appended data (i.e the last append wins - assuming same length and offset). As for the maximum size, its 4.75 TiB (100 MiB X 50,000 blocks) for version 2016-05-31 and later, and 195 GiB (4 MiB X 50,000 blocks) for all older versions (since a datalake file is a block blob).

sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/read.py Outdated Show resolved Hide resolved
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/read.py Show resolved Hide resolved
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/upload.py Show resolved Hide resolved
sdk/storage/azure-storage-file-datalake/tests/perfstress_tests/upload_from_file.py Show resolved Hide resolved
@rakshith91 rakshith91 removed their request for review December 30, 2020 19:43
@annatisch annatisch mentioned this pull request Jan 14, 2021
Closed
@annatisch annatisch merged commit 895ce54 into Azure:master Mar 3, 2021
iscai-msft added a commit that referenced this pull request Mar 3, 2021
@iscai-msft
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …
c31c0f9 
…into add_sample_check

* 'master' of https://github.com/Azure/azure-sdk-for-python: (388 commits)
  [text analytics] add normalized_text (#17074)
  Renaming with_token identity function (#17066)
  Adapt to azure core's cloud event (#17063)
  align perf tests with js (#17069)
  [Perfstress][Storage] Added FileShare perf tests (#15834)
  [formrecognizer] Adding custom forms perf test (#16969)
  Fix LanguageShort typo (#17068)
  sas creds updates (#17065)
  [eventgrid] Fix Sample eh (#17064)
  [Perfstress][Storage] Added Datalake perf tests (#15861)
  [text analytics] Healthcare n-ary relations (#16997)
  ServiceBus dict-representation acceptance and kwarg-update functionality  (#14807)
  [text analytics] add perf tests (#17060)
  Add cloud event to core (#16800)
  [Perf] Small fixes to storage-blob (#17055)
  [EG] Regenerate Code (#17053)
  Scrub batch shared keys (#17030)
  [Tables] Add SAS to tables (#16717)
  T2 containerservice 2021 03 03 (#17050)
  Addressing issues with CredScan (#16944)
  ...
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Mar 4, 2021
@iscai-msft
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …
2bb425f 
…into add_query_kwarg

* 'master' of https://github.com/Azure/azure-sdk-for-python: (69 commits)
  march release (Azure#16966)
  Release of Device Update for IoT Hub SDK for Python. (Azure#17005)
  Add Get-AllPackageInfoFromRepo (Azure#16947)
  Track1 package is incorrectly set as track2 (Azure#17075)
  [text analytics] add normalized_text (Azure#17074)
  Renaming with_token identity function (Azure#17066)
  Adapt to azure core's cloud event (Azure#17063)
  align perf tests with js (Azure#17069)
  [Perfstress][Storage] Added FileShare perf tests (Azure#15834)
  [formrecognizer] Adding custom forms perf test (Azure#16969)
  Fix LanguageShort typo (Azure#17068)
  sas creds updates (Azure#17065)
  [eventgrid] Fix Sample eh (Azure#17064)
  [Perfstress][Storage] Added Datalake perf tests (Azure#15861)
  [text analytics] Healthcare n-ary relations (Azure#16997)
  ServiceBus dict-representation acceptance and kwarg-update functionality  (Azure#14807)
  [text analytics] add perf tests (Azure#17060)
  Add cloud event to core (Azure#16800)
  [Perf] Small fixes to storage-blob (Azure#17055)
  [EG] Regenerate Code (Azure#17053)
  ...
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Mar 4, 2021
@iscai-msft
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …
f66bdb4 
…into add_reprs

* 'master' of https://github.com/Azure/azure-sdk-for-python: (71 commits)
  EG - more docs imrpovement (Azure#17079)
  [EventHubs] add logging.info to warn the usage of partition key of non-string type (Azure#17057)
  march release (Azure#16966)
  Release of Device Update for IoT Hub SDK for Python. (Azure#17005)
  Add Get-AllPackageInfoFromRepo (Azure#16947)
  Track1 package is incorrectly set as track2 (Azure#17075)
  [text analytics] add normalized_text (Azure#17074)
  Renaming with_token identity function (Azure#17066)
  Adapt to azure core's cloud event (Azure#17063)
  align perf tests with js (Azure#17069)
  [Perfstress][Storage] Added FileShare perf tests (Azure#15834)
  [formrecognizer] Adding custom forms perf test (Azure#16969)
  Fix LanguageShort typo (Azure#17068)
  sas creds updates (Azure#17065)
  [eventgrid] Fix Sample eh (Azure#17064)
  [Perfstress][Storage] Added Datalake perf tests (Azure#15861)
  [text analytics] Healthcare n-ary relations (Azure#16997)
  ServiceBus dict-representation acceptance and kwarg-update functionality  (Azure#14807)
  [text analytics] add perf tests (Azure#17060)
  Add cloud event to core (Azure#16800)
  ...
iscai-msft added a commit to iscai-msft/azure-sdk-for-python that referenced this pull request Mar 4, 2021
@iscai-msft
Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python …
bf5990f 
…into http_response_json

* 'master' of https://github.com/Azure/azure-sdk-for-python: (165 commits)
  EG - more docs imrpovement (Azure#17079)
  [EventHubs] add logging.info to warn the usage of partition key of non-string type (Azure#17057)
  march release (Azure#16966)
  Release of Device Update for IoT Hub SDK for Python. (Azure#17005)
  Add Get-AllPackageInfoFromRepo (Azure#16947)
  Track1 package is incorrectly set as track2 (Azure#17075)
  [text analytics] add normalized_text (Azure#17074)
  Renaming with_token identity function (Azure#17066)
  Adapt to azure core's cloud event (Azure#17063)
  align perf tests with js (Azure#17069)
  [Perfstress][Storage] Added FileShare perf tests (Azure#15834)
  [formrecognizer] Adding custom forms perf test (Azure#16969)
  Fix LanguageShort typo (Azure#17068)
  sas creds updates (Azure#17065)
  [eventgrid] Fix Sample eh (Azure#17064)
  [Perfstress][Storage] Added Datalake perf tests (Azure#15861)
  [text analytics] Healthcare n-ary relations (Azure#16997)
  ServiceBus dict-representation acceptance and kwarg-update functionality  (Azure#14807)
  [text analytics] add perf tests (Azure#17060)
  Add cloud event to core (Azure#16800)
  ...
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-python that referenced this pull request Sep 8, 2021
CodeGen from PR 15861 in Azure/azure-rest-api-specs
aa38e18 
[Hub Generated] Review request for Microsoft.Aadiam to add version preview/2020-03-01-preview and stable/2020-03-01 (Azure#15861)

* Adds base for updating Microsoft.Aadiam from version preview/2020-03-01-preview to version 2020-03-01

* Updates readme

* Updates API version in new specs and examples

* Adding changes from previous PR

* Ran prettier, fixed types.json reference

* Remove changes to preview version

* Fix python failure

* Try switching error locations

* Change error location back

* fix old examples

* Try to fix python check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikeharder mikeharder mikeharder approved these changes

@tasherif-msft tasherif-msft tasherif-msft approved these changes

@amishra-dev amishra-dev Awaiting requested review from amishra-dev

@kasobol-msft kasobol-msft Awaiting requested review from kasobol-msft

@xiafu-msft xiafu-msft Awaiting requested review from xiafu-msft

@zezha-msft zezha-msft Awaiting requested review from zezha-msft

@jaschrep-msft jaschrep-msft Awaiting requested review from jaschrep-msft

@danieljurek danieljurek Awaiting requested review from danieljurek

@mitchdenny mitchdenny Awaiting requested review from mitchdenny

@scbedd scbedd Awaiting requested review from scbedd

@weshaggard weshaggard Awaiting requested review from weshaggard

Assignees
No one assigned
Labels
Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@annatisch @kasobol-msft @jaschrep-msft @mikeharder @tasherif-msft