Skip to content

Commit

Permalink
[Internal] Msdata/Direct: Refactors msdata/direct branch with latest …
Browse files Browse the repository at this point in the history
…v3 master and Cosmos.Direct v3.30.8 (#3856)

* [Internal] Client Telemetry: Fixes tests leaking environment variables (#3517)

* Adding log lines

* More logs

* Debugging further

* Removing other builds

* And more debugging

* Wrong build parameters

* Wrong category

* Removing noise

* Fixing test

* Adding utils

* Adding test with client telemetry enabled

* Fixing leak in Client Telemetry Tests

* Reenabling test

* cpu monitor initialization

* Adding name details

* Undo another file

* Updated change log and bumped up the version. (#3526)

* Query: Fixes performance regression on target partition on some ORDER BY queries with continuation (#3525)

* Revert performance regression caused by #1289

* Remove irrelevant comment

* Add a test for validating formatted filters for the target partition

* [Internal] AI Integration: Adds SubStatusCode Information in attributes (#3533)

* wip

* regenerated baseline

* add test

* fix test

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Diagnostics: Removes unused properties and reduces size (#3519)

* Remove starttime and id from diagnostics string

* Remove starttime and id from diagnostics string

* Update xml files for trace baseline tests

* Update xml files for trace baseline tests

* Update xml files for trace baseline tests

* Whitespaces update

* Changed starttime position based on code review

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] docs: Add address caches conceptual flow (#3534)

* Rough draft

* Some more changes

* Adding a new section

* Some more refinement

* Some changes 

Lets get the feedback

* Some MISC changes

* Release: Adds SDK version and changelog for 3.31.2 (#3546)

* Bumped SDK version and changelog

* Added contracts file for 3.31.2

* [Internal] Documentation: Removes invalid comment from ReadThroughputAsync (#3516)

* [Internal] nugetconfig: Removes specific overrides (#3551)

* Query: Fixes partition range evaluation for spatial queries (#3495)

* Initial commit

* Update.

* Pull/Rebase

* Addressed comments.

* Build fix

* [Internal] Documentation: Refactors cache content on its own document (#3554)

* Add cache file

* Moving content away

* [Internal] Emulator unit tests: Adds IdEncoding unit tests for ComputeGateway (#3556)

* Adding IdEncoding unit tests for ComputeGateway

* Reacting to code review feedback

* Updating md files

* Change Feed Processor: Adds support for Resource Tokens (#3566)

* Adding rid parsing

* Test

* Removing dead code

* Contract update

* [Internal] AI integration: Refactor code how container and database name is flowing to opentelemetry module (#3532)

* wip

* WIP

* Revert "WIP"

This reverts commit 71275de54b9e67fa54a37e79d450b9597e173934.

* Revert "wip"

This reverts commit 586fa9865cc3f40dabd7ef90fb3e0cf499a045bc.

* wip add containe and database info

* redesign how container and database name information flows into opne telemetry data

* test fix

* fix test

* fix tests

* fix typos

* baseline test fix

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* [Internal] AI Integration: Adds a new flag in DistributedTracingOptions (#3562)

* add flag in diagnostic options

* test fixes

* rename variable

* test fix

* add validation

* fix baseline

* test fix

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Query: Fixes default to BadRequestException in case of internal errors in ServiceInterop (#3399)

* Don't default to BadRequestException in case of errors in ServiceInterop

* Incorporate code review feedback

* Fix build error

* fix up failing test

* [Internal] Query: Adds unit tests for Merge/Split implementation with OptimisticDirectExecution pipeline (#3510)

* Added tests to test different aspects of merge/split support with OptimisticDirectExecution pipeline. Tests check for gone exception handling, pipeline switching etc.

* Added gone exception simulation tests.

* Added new tests and improved test infra

* Removed ParalleContEvocation test. Fixed comments

* Removed CreateParallelCrossPartitionPipelineStateAsync() as it is not being used anymore

* Removed while loop in CreateDocumentContainerAsync()

* Fixed comments.

* Updated ExecuteGoneExceptionOnODEPipeline()

* Added type Assert for ExecuteGoneExceptionOnODEPipeline()

* Updated OptimisticDirectExecution pipeline abbreviation

* Updated TestBaseline folder with new xml

* Client Retry Policy: Adds HTTP timeouts with request-level cross-region retry (#3555)

* Fixes to ReadThroughputAsync for databases with no provisioned throughput and null as request options

* fixed failure to ReadReplaceThroughputResponseTests

* Added Stream Method

* Ran UpdateContract.ps1

* Encryption implemtation

* Fixed spelling error

* Update Microsoft.Azure.Cosmos.Encryption/src/EncryptionDatabase.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Variable name change

* Update Microsoft.Azure.Cosmos/src/Resource/Database/Database.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Suggested Changes and fixes

* Removed manufactured ResponseMessage + nits

* Simplified PR

* nits

* nits

* initial changes TODO: Update tests

* updated tests

* nits'

* Ran UpdateContracts.ps1

* nits + requested changes

* Delete NuGet.Config

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Update Microsoft.Azure.Cosmos/src/HttpClient/HttpTimeoutPolicy.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Use Cosmos Exception Factory, Simplified Tests

* removed unused code

* nits: removed unused code

* removed unused code

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>
Co-authored-by: Nalu Tripician <ntripician@microsoft.com>

* Documentation: Fixes EUAP in Comments (#3579)

* Query: Fixes incorrect FeedResponse.Count when result contains undefined elements (#3574)

* Do not maintain an independent count on QueryResponse that can go out of sync

* Add more test coverage for QueryResponse<T>.Count

* Output the correct count from CosmosElementSerializer when the input contains CosmosUndefined

* Add untyped tests for CosmosUndefined

* Remove commented code

* removed allr eference (#3581)

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Trace: Fixes Tracing/diagnostics hour-times to 24Hours (#3577)

* Trace: Fixes Tracing/diagnostics hour-times to 24Hours

* fixing baseline tests

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* AI Integration: Adds cosmetic fixes (#3576)

* wip

* status code int and internal and client kind activity

* remove unused imports

* update baselines

* fix test

* fixed baseline tests

* fix tests

* update base tetss

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Query: Adds ALL Scalar Expression (#3509)

* Add SqlAllScalarExpression to v3 DOM

* updated generated parser files

* Parsing for ALL

* Added tests for ALL and baselines

* Added more tests

* added new test, cleanup

* cleaning & fix typos

* fixed typo

* Added new baseline test file names to csproj file

* renamed AggregateAll to AggregateSubquery to accomodate FIRST and LAST later

* Added keywords for 'left' and 'right' and respective function calls

* fixed bug from last commit

* cleaning

* replace tabs with spaces

* cleaning

* Release: Adds API contracts for 3.31.2-preview (#3586)

* [Internal] sccignore: Adds a .sccignore file to apply an exception for artifacts configuration issues (#3589)

* [Internal] PermissionTests: Adds CosmosPermissionTests Coverage (#3593)

* Ensures that both Direct and Gateway connection modes are tested
* Validates that container read works with PermissionMode.Read (test was previously only validating that Delete was blocked - i.e. the negative case).

* [Internal] AI Integration: Refactors to Operation prefix and add tests (#3583)

* add tests for otel and custome listener

* clean up

* null pointer fix

* fix tets

* handle event generation also at operation level

* added documentation

* wip

* change event sourcename

* rename event Name

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* [Internal] HttpTimeoutPolicy: Removes Data Plane Writes from being able to be retried (#3607)

* data plane writes no longer failover on timeout

* removed duplication of test\

* [Internal] Performance Testing: Adds Distributed Tracing option in benchmarks (#3611)

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* [Internal] Benchmark: Refactors code to make Memory Stream capacity configurable (#3624)

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* add new LatestVersion changefeed mode that has same behavior as Incremental; renamed FullFidelity to AllVersionsAndDeletes (#3596)

* Query: Fixes handling of CosmosUndefined, CosmosGuid and CosmosBinary in unordered DISTINCT (#3632)

* Handle CosmosUndefined, CosmosGuid and CosmosBinary in DistinctMap.UnorderedDistinctMap

* Address code review feedback and remove unnecessary allocations from DistinctQueryPipelineStageTests

* [Internal] Subpartitioning: Adds updates to test coverage for subpartitioning (#3618)

* updates to test coverage for subpartitioning

* bug fixes

* now useses Assert.ThrowsException

* Seperated into multiple tests for clarity

* Put MultiHash test into seperate test file

* nit

* [Internal] ContainerProperties: Fixes version reset when setting PartitionKeyPath (#3637)

* Remember previous value

* test

* [Internal] AI Integration: Adds CorrelationId and Activity Id Attributes for query operation (#3630)

* add activityid in Otel attributes

* added correlation id

* operation type fix

* remove test changes

* test fix

* fix baseline test

* rename correlationId

* fix tests again

* include only not null attributes in test

* fixed tests

* changefeedxml

* test fix

* ordering activity in operationname oerder

* fix test

* review comments

* refator header getter setter

* clean up

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Documentation: Fixes CosmosClientBuilder.WithConnectionModeGateway parameter description (#3643)

* Fixed CosmosClientBuilder.WithConnectionModeGateway documentation

* Update Microsoft.Azure.Cosmos/src/Fluent/CosmosClientBuilder.cs

Co-authored-by: Ruben Bartelink <ruben@bartelink.com>

Co-authored-by: Augsten <aaugsten@harbourvest.com>
Co-authored-by: Ruben Bartelink <ruben@bartelink.com>

* Upgrade Resiliency: Refactors Implementation for Opening Rntbd Connections to Backend Replicas in Direct Mode. (#3640)

* Code changes to refactor implementations for opening connections to all replicas. Fixed test failures due to Direct package upgrade.

* Code changes to add poland central region as a part of Regions.cs

* Code changes to update contract to reflect new regions.

* Revert "Code changes to update contract to reflect new regions."

This reverts commit f171b3c1c9889043556ddf96bcd33ccd79565ad9.

* Revert "Code changes to add poland central region as a part of Regions.cs"

This reverts commit 1aafbf18f6d80e9a92baa301b6b23cf065e4b155.

* [Preview] AI integration: Adds IsDistributedTracingEnabled flag as public API to enable/disable this feature (#3598)

* make api public for preview

* add null check

* fix tests

* singleton listener initialization

* assign null to listeners

* fix test

* concurrent bag in listener

* renamed to LatencyThresholdForDiagnosticEvent

* renamed to IsDistributedTracingEnabled

* updated xml

* update contract

* made latency threshold flag internal

* fix test

* regeneratebaselines

* update documentation

* rename builder api

* add docs

* updated contracts and all

* doc update

* import cleanup

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* Change Feed Processor: Fixes behavior with StartTime on Local (#3645)

* To UTC

* Test

* [Internal] Client Telemetry: Refactors code to use base useragent string (#3653)

* [Internal] AI Integration: Refactors code to rename event name (#3648)

* first draft

* rename event name

* updated xmls

* update files

* Region Availability: Adds Poland Central Region For Public Usage (#3656)

* Client Encryption: Adds validation code to check if the Key Vault URI provided in wrap metadata is a valid key identifier. (#3642)

* Check if the key vault uri provided is a valid Kid

* test fix.

* update changelog and build props

* Update Directory.Build.props

* Update Microsoft.Azure.Cosmos.Encryption.csproj

* Fixed preview version

* Refactor

* Update EncryptionDatabaseExtensions.cs

* [Internal] Query: Adds Split Support for Ode (#3572)

* Added tests to test different aspects of merge/split support with OptimisticDirectExecution pipeline. Tests check for gone exception handling, pipeline switching etc.

* Added gone exception simulation tests.

* Added new tests and improved test infra

* Removed ParalleContEvocation test. Fixed comments

* Removed CreateParallelCrossPartitionPipelineStateAsync() as it is not being used anymore

* Removed while loop in CreateDocumentContainerAsync()

* Fixed comments.

* Updated ExecuteGoneExceptionOnODEPipeline()

* Added type Assert for ExecuteGoneExceptionOnODEPipeline()

* Replaced try-catch with if statement in MoveNextAsync()

* Added delegate to access TryCreateCoreContextAsync()

* Added check to confirm Ode pipeline is not called in fallback plan

* Updated method name from OptimisticDirectExecutionContext() to TryCreateOptimisticDirectExecutionContext()

* Using delegate instead of Func<>.

* Ode fallback plan always calls Specialized pipeline

* Using ServiceInterop/Gateway to get QueryPlan for Specialized Pipeline

* Added new test to check handling of failing fallback pipeline

* Code cleanup

* Added logic for handling non ODE continuation tokens

* Moved delegate away from member variables

* Added tests for Merge case

* Updated method names

* Added checks for tryCatch

* Updated SetCancellationToken() to use Try

* Updated TryUnwrapContinuationToken()

* Removed changes in FlakyDocumentContainer.cs

* Removed unused imports

* Updated comments

* Fixed comments and cleaned up test code

* Added CosmosElement null check in TryUnwrapContinuationToken()

* Removed FlakyDocumentContainer.cs from pull request

* Removed unused imports

* Updated TryUnwrapContinuationToken()

* Update MoveNextAsync() call in OptimisticDirectExecutionQueryBaselineTests.cs

* Made MergeTestUtil.IsFailedFallbackPipelineTest a readonly property

* Added IsPartitionSplitException() overload to take CosmosElement

* Fixed bug regarding syntax error queries

* [Internal] AI Integration : Fixes operation type  for batch (#3660)

* fix op type

* fix conflict

* CosmosClientOptions: Adds ServerCertificateCustomValidationCallback for Http and TCP (#3636)

* Adding ServerCertificateCustomValidationCallback in clientoptions

* Adding Server callback for Http and fixing tests

* Fixing failing E2Etests

* Resolving merge conflicts

* Running update contracts script

* Running Update contracts script

* Running Update contracts script

* Reverting the v3 version change

* Update based on review comments

* Added unit tests

* Added remarks for callback delegate

* Ran update contracts script

* Update based on review comments

* Ran update contracts script

* Updated unit tests

* Making ssl validation function private

* Updating test files

* Update remarks for sslvalidation public contract

* Added emulator tests for server validation

Co-authored-by: Debdatta Kunda <87335885+kundadebdatta@users.noreply.github.com>

* Query: Adds EnableOptimisticDirectExecution flag to QueryRequestOptions (#3664)

* Added new flag to QueryRequestOptions to allow customers to use Ode pipeline

* Updated comments in QueryRequestOptions.cs

* Renamed enabledOde to enableOde

* Removed default setting for EnableOptimisticDirectExecution

* [Internal] Tests: Removes Direct/HTTPS emulator tests (#3679)

* Removing direct/https tests

* mppreference

* [Internal] Benchmark : Fixes issue with dependency on Cosmos Project (#3673)

* users/sourabhjain/benchmarkfix

* update pipeline

* Revert "users/sourabhjain/benchmarkfix"

This reverts commit 81b48f0.

* fix compilation error

* add parama for preview pkg also

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* LocalQuorum: Adds Quorum reads on Consistent Prefix Accounts (#3680)

Co-authored-by: DESKTOP-ED57J7H\Prashanth Venkataram <vprashanth1987@hotmail.com>

* 3.32.0: Adds new SDK version and contract files (#3687)

* 3.32.0: Adds new SDK version and contract files

* 3.32.0: Adds new SDK version and contract files

* Updating changelog version

* Updating changelog version

* Added more commits to changelog and updated release contract

* Added documentation tags PR in changelog

* Updated changelog based on reviews

* Updated PR decsription in changelog

* Update changelog.md

Updated full fidelity change description

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Samples: Adds change feed pull model samples (#3646)

* add change feed pull samples

* refactor appsettings validation

* addressing pr comments

* move task delay

* update sample to use latest change feed mode names

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Tests: Refactors emulator CI (#3688)

* [Internal] GitHub Template: Adds needs-investigation label (#3708)

By default, all "Bug report" issues will have "needs-investigation"

* Adding fabric bot action (#3709)

* CosmosNullReferenceException: Refactors CosmosNullReferenceException to pass along InnerException property on parent NullReferenceException (#3713)

* Passed inner exception details to NullReferenceException ctor when instantiating CosmosNullReferenceException.

* Added unit tests.

* Addressed PR feedback.

* [Internal] PriorityRequests: Fixes header value (#3714)

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Query: Adds single physical partition check for OptimisticDirectExecution queries (#3699)

* Added single physical partition check for Ode queries. Updated test infrastructure for Ode emulator tests too.

* Refactored emulator tests to have all the test cases at the top of the file

* Updated TryGetTargetRangeOptimisticDirectExecutionAsync()

* Uodated logic on how many times CreateIngestQueryDeleteAsync() gets called

* Added debug asserts for partitionKeyDefinition

* Added pageSizeOptions parameter in CreateInput() in EmulatorTests

* Fixed comments

* [Internal] CTL: Fixes Reservoir Sampling Logic (#3712)

* Code changes to fix the reservoir sampling logic in CTL

* Code changes to modify help text on reservoir type.

* Code changes to address minor code refactor.

* Diagnostics: Adds startDate in Summary (#3707)

* Adding start date

* Updated trace files

* PR comment

* Committing missing tests

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Client Telemetry: Adds network information in the payload (#3691)

* first draft

* clean code

* add test

* fix test

* add replica info collection

* fix substatuscode and  operation type error

* fix code

* collect http infor in cache also

* message to stacktrace

* remove rntbd recording from cache

* print proper exception message

* fix test

* cleanup unused code and added few status codes in ignore list

* fixed all exception logging

* refactor conditions

* fix tests

* Documentation: Fixes Database.ReadAsync description (#3457)

* Documentation: Modify retry time to timespan 

Modify retry time in seconds to timespan of parameter maxRetryWaitTimeOnThrottledRequests (Method: [WithThrottlingRetryOptions](https://docs.microsoft.com/en-us/dotnet/api/microsoft.azure.cosmos.fluent.cosmosclientbuilder.withthrottlingretryoptions?view=azure-dotnet#feedback))
Resolves Azure/azure-sdk-for-net#29567

* Documentation: Fixes DeleteItemAsync Example

Documentation: Fixes DeleteItemAsync Example

* Documentation: Fixes ItemRequestOptions Example

Documentation: Fixes ItemRequestOptions Example

* Documentation:Update Database.ReadAsync description

Documentation: Update Database.ReadAsync description

* Documentation: Update Database.ReadAsync description

Documentation: Update Database.ReadAsync description

* Documentation: Updated ToStreamIterator example

Documentation: Updated ToStreamIterator example

* Modified StreamIterator section

* Update Microsoft.Azure.Cosmos/src/Resource/Database/Database.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Remarks correction

* Revert the StreamIterator changes

Revert the StreamIterator changes

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>
Co-authored-by: Kiran Kumar Kolli <kirankk@microsoft.com>

* Upgrade Resiliency: Adds Implementation for Validating the Unhealthy Backend Replicas in Direct mode (#3631)

* Code changes to implement replica validation in dotnet v3 sdk.

* Cosmetic changes to add inline code comments.

* Code chages to address review comments.

* Code changes to cover a scenario for async cache.

* Code changes to refactor async non-blocking cache code.

* Code changes to address minor review comments.

---------

Co-authored-by: Kiran Kumar Kolli <kirankk@microsoft.com>

* ReadMany: Fixes BadRequest when using Ids with single quotes (#3732)

* Use parameters

* Emulator tests

* Release: Adds SDK version and changelog for 3.32.1 (#3733)

* version bump

* Contracts

* [Internal] Build: Fixes static tool analysis versions (#3736)

* Update Binskim and follow warnings

* Fixing task

* More version bumps

* binskim args

* Padding

* policheck

* postanalysis parameters

* analysis settings

* Query: Fixes System.ArgumentException when using PartitionKey.None on x86, Linux or in Optimistic Direct Execution (#3730)

* Minor clean up of OptimisticDirectExecutionQueryTests

* More minor cleanup in OptimisticDirectExecutionQueryTests

* Add emulator tests for the bypass query parsing scenario

* Handle PartitionKey.None while creating QueryIterator. This is a workaround for the PartitionKeyInternal.None not following its own contract

* Fix up to correctly handle PartitionKey.None using the CachedContainerQueryProperties

* Add more tests where PartitionKey.None maps to PartitionKey.Undefined

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Query: Adds FIRST and LAST Scalar Expressions (#3629)

* Add FIRST and LAST objects and update visitors

* add FIRST LAST evaluation and update offline engine visitors

* Add FIRST and LAST to parser

* update another visitor

* Fix typo

* fix typo

* added new tests and baselines

* cleaning

* cleaning

---------

Co-authored-by: neildsh <35383880+neildsh@users.noreply.github.com>
Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Subpartitioning: Adds support for Prefix Partition Key searches for sub partitioned containers (#3109)

* 1st round of changes to support subpartitioning for ChangeFeed

* name change and moved a method to FeedRangePartitionKey

* support for prefix partition change feed and query including unit and emulator tests. need to verify using FeedRange with QueryIterator

* splitting out unit tests from emulator tests

* 1st round of changes to support subpartitioning for ChangeFeed

* name change and moved a method to FeedRangePartitionKey

* support for prefix partition change feed and query including unit and emulator tests. need to verify using FeedRange with QueryIterator

* splitting out unit tests from emulator tests

* moved logic to choose for prefix partition query and change feed

* additional conditional to check for MultiHash partitionkeydefinition kind

* removed unnecessary using

* cleanup

* unnecessary using

* removed PREVIEW from proj

* unit tests for ResolveFeedRangeBasedOnPrefixContainerAsync

* change access modifier

* since I changed to static, needed to fix tests

* dealing with some testing nits

* removed Console.Writeline

* slight change in test use containerProperties and partitionKeyDefinition

* big fix for resolve feedrange, changes to error behavior, added additional test coverage

* test changes

* ran updateContracts.ps1

* reverting to 3ad5309

* Ran Update Contracts

* ran updateContracts.ps1

* UpdateContracts.ps1

* updatecontracts

* removed bad check from test, removed changes from updatecontracts.ps1

* revert UpdateContracts changes

* Reverting to 756a123

* removed accidental change from changelog

* removed unwanted change from Directory.build.props

* added test clean up

* removed console.writelines

* updates to query test

* removed preview flag

* ran UpdateContrats.ps1

* no longer recread feedRange for queries unless using prefix pk

* simplified if statement

* fixed bug

---------

Co-authored-by: Nalu Tripician <nalutrip@icloud.com>

* [Internal] ClientTelemetry: Adds logic to limit payload size to 2 MB (#3717)

* first draft

wip

fix test and logic

* resolve conflicts

* limit 2 mb

* ad callback

* fix tests

* code refactor

* cosmos json to newtosoft json

* clean up files

* fix logging to argumrnt based

* code refactor

* add null check

* Query: Fixes regression from LINQ custom serializer fix (#3749)

Co-authored-by: Minh Le <leminh@microsoft.com>
Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] LocalQuorum: Refactors override (i.e. strong) to allow from any account consistency (#3753)

- Localquorum override (i.e. strong) to allow from any account consistency
- Facilitates no-downtime downgrade of existing accounts (i.e. existing Strong/bounded accounts migration to Eventual)

* Release: Adds SDK version and change log for 3.32.2 (#3752)

* version bump

* contract

* changelog

* version bump

* Fixing changelog text

* Add another PR

* Subpartitioning: Adds APIs for public release and increase REST API version (#3763)

* initial commit, Http version issues still needs to be resolved

* updateContracts + update Http Version

* updateded version in test

* update contracts

* requested changed

* changed name in comments to subpartitioning from multihash

* undid changes to TestLiteralSerialization.xml

* removed changes to non API.json files

* removed non XXXAPI.json file changes

* changed verbage on public comments

* changed error message to reflect verbage change

* Change Feed Processor: Fixes LeaseLostException leaks on notification APIs for Renew scenarios (#3775)

* Adding cases

* Tests

* Upgrade Resiliency: Refactors GatewayAddressCache to Mark TransportAddresses to Unhealthy when Connection Reset Event Occurs (#3768)

* Code changes to mark the transport uri to unhealthy for which a connection reset event occures through the connection state listener.

* Code changes to bump up the direct version.

* Code changes to clean up Gateway Address Cache.

* Code changes to fix pipeline build.

* Code changes to fix serilization test failures.

* Code changes to add force refresh to the gateway function callback delegate.

* [Preview] PriorityBasedExecution: Adds PriorityLevel as a RequestOption (#3672)

* Added Priority Level as a Request Option

* Changed Priority Level Low and High to 1 and 2 respectively

* Bumped DirectVersion

* Added made PriorityLevel internal for non preview packages

* Deleted PriorityLevelTests.cs

* Added description of Priority Level in RequestOptions.cs

* Modified comments for PriorityLevel in RequestOptions.cs

* Updated Contracts

* Modified Remarks for PriorityLevel and added see also link.

* Updated contracts

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Client Telemetry: Adds sampling logic for network level telemetry (#3750)

* sampling logic add

* throttlinfix

* fix tests

* fix tests

* remove dispose from sampler

* add 412

* draft push

* refactor code for sampling

* clean up

* start testing

* adding tests

* wip

* test fix

* fix test

* fix tests

* deleted extra file

* remove one toList

* add custome logic for sampling

* code refactor

* replace key

* remove commented code from test

* added more comments

* simpler sampler logic

* refactor code

* [Internal] Emulator Test : Fixes test when running with distributed tracing enabled. (#3751)

* enable DT for everything

* try1

* code refactoring

* fix pipeline

* try 2

* Revert "try 2"

This reverts commit 61643f9b4dcd2ab1898fd68b111897bed0bc6bcf.

* Revert "fix pipeline"

This reverts commit 4b217f5f6a087b1dd889b804fd8b477accd46419.

* Revert "code refactoring"

This reverts commit d1ff0ddc17ad655360988d171bd033617752f91f.

* Revert "try1"

This reverts commit bf9c41b43684f59719c3673d6f9c7450bbdc5142.

* lazy factory and threadsafe

* scope factory in function

* try non static

* add flag

* fix test

* add consoles

* more console with fix

* fix tests

* temporarily enable dt

* fuix tests

* dispose listener

* fix tests

* remove delay

* revert flag change

* dynamically get number of test in trace class

* Release: Adds SDK version and changelog for 3.32.3 (#3788)

* Adding contract and version bump

* Adding changelog

* SummaryDiagnostics: Refactors Code to Remove Dependency of HttpResponseHeadersWrapper to fetch Sub Status Codes (#3792)

* Code changes to fetch sub status code from http response/ content headers.

* Code changes to address review comments.

* Code changes to return the first element from the sub status list.

* Code changes to update first or default.

* [Internal] Documentation: Fixes API name to NoSQL (#3795)

* [Internal] CTL: Fixes docker image pipeline (#3794)

* [Internal] AI Integration : Refactors useragent attribute name as per Otel conventions (#3784)

Co-authored-by: Fabian Meiswinkel <fabian@meiswinkel.com>

* HttpClient: Adds Properties to the Http messages if available (#3803)

* Passing properties

* test

* Documentation: Refactors SQL API reference to NoSQL API (#3793)

Co-authored-by: Kiran Kumar Kolli <kirankk@microsoft.com>

* [Internal] Perf test: Refactors code to re-enable performance tests (#3785)

* enable perf tests

* updated json

* upgrade benchmark

* update results

* updated benchmark results

* HttpClient: Adds detection of DNS changes through use of SocketsHttpHandler for .NET 6 and above (#3762)

* initial commit

* removed unneeded usings

* added validation callback, still needs tests

* nits + fixes

* added additional test

* test change

* removed unneeded Dispose calls

* removed unnneed dispose calls

* requested changes

* added pooledConnectionLifetime as client option

* nit

Co-authored-by: Kevin Pilch <me@pilchie.com>

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Update Microsoft.Azure.Cosmos/src/CosmosClientOptions.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* suggested changes

* remove test, reorder usings

* updated contracts

* removed all non XXXAPI.json changes from UpdateContracts run

* removed public surface, added random timespan

* removed change from unrelated file

* Update Microsoft.Azure.Cosmos/src/HttpClient/CosmosHttpClientCore.cs

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* added thread safe random method

* nit

* fixed merge mistake

* added reflection failsafe/error tracing

* nits

* added back removed if

* fixed formatting

* changed random method, fixed serverCertificateCustomValidation

---------

Co-authored-by: Kevin Pilch <me@pilchie.com>
Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Tests: Fixes Open Telemetry attributes for ReadMany test (#3805)

* Fixing test

* New baseline

* Undo some changes

* [Internal] Client Telemetry: Refactors code to run client telemetry data processing task in background. (#3783)

* first draft

* remove failure count test

* refactporing

* code refactor

* create task with timeout

* fix test

* code refactoring

* fix timeout code

* space fix

* not failing if processor is taking time

* fix procrsser test

* code refactor

* refactor and test fix

* Patch: Adds Move Operation (#3389)

* Basic changes to introduce move operator

* Added "from" object in patch spec operation.
Added testcase block.

* Fixed testcase.

* Changes made to address comments'

* Added comment regarding enum mutations

* Formatted comment

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* Moved summary location.

* Ran UpdateContracts.ps1

---------

Co-authored-by: Amaan Haque <t-amaanhaque@microsoft.com>
Co-authored-by: Amaan Haque <amaanhaque@microsoft.com>
Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Pipelines: Adds nightly build to produce packages (#3802)

* Support cleaning

* wire previous content delete

* as text

* with variable

* another test

* param with types

* as string

* no delete

* no quotes

* undoing

* re-adding quotes

* testing empty

* trying another test

* readding version

* fixing publishing artifacts

* fixing parameter

* Fixing official pipeline

* version 5

* fixing main pipeline

* test with true

* using start time

* nightly preview

* passing parameters to pack

* Fixing nuget version

* arguments on the nuget pack

* folder structure

* testing v5

* Using only content

* Removing currentDate

* [Internal] OpenTelemetry: Direct Package update and replacing diagnostic files (#3797)

* Direct Package update and replacing dagnostic files

* Resolve merge conflicts

* Running updateCOnstracts script

* Removed LinqTranslationWithCustomSerializerBaseline file

* Adding isDistributedTracingEnabled flag

* Running update contracts

* Running update contracts

* Running update contracts

* fix test

* Code cleanup for test fix

* Code cleanup for test fix

* Making regex expression readable

* Adding comment for regex expression

---------

Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>

* [Internal] MerlinBot: Adds auto-merge and cleanup automation (#3813)

* Add config changes

* Polishing automerge config

* Update fabricbot.json (#3824)

* [Internal] Upgrade Resiliency: Adds Logic to Validate `Unknown` Replicas along with `Unhealthy`. (#3820)

* Code changes to add aggressive validation logic.

* Code changes to enable aggressive validation for all regions.

* Code changes to pull in msdata cosmos.direct changes related to aggresive validation logic.

* Code changes to make minor cosmetic changes.

* Code changes to address review comments.

* Serialization: Fixes call to CosmosSerializer.FromStream on Gateway mode when EnableContentResponseOnWrite is false (#3814)

* Do not call serializer if ResponseMessage.Content is empty.

* Add unit test

* Update unit tests

* Remove unused usings

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Documentation: Adds documentation covering build pipelines (#3822)

* Add doc

* Move benchmark

* Fixing texts

* Client Encryption: Adds release version of Microsoft.Azure.Cosmos to Microsoft.Azure.Cosmos.Encryption.Custom (#3799)

* cosmos version change

* changing preview to release

* resolving code review comments

---------

Co-authored-by: Santosh Kulkarni <66682828+kr-santosh@users.noreply.github.com>

* SDK 3.33.0 : Adds version bump and changelog (#3823)

* release 3.30.0

* added changelog

* updated changelog

* updated changelog

* suggested change to changelog

* updated changelog

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>

* [Internal] Documentation: Adds msdata/direct Sync-up Guide. (#3828)

* Code changes to add msdata/direct sync-up documentation.

* Code changes to address review comments.

* Code changes to address further review comments.

* Code changes to address minor review comments.

* Removed internal links.

* Query: Adds TRIM string system function support in LINQ (#3833)

* add trim support

* Added some test coverage

* address reviews

---------

Co-authored-by: Minh Le <leminh@microsoft.com>

* Query: Fixes Parsing Error in SQL DOM when CultureInfo is available (#3832)

* add fix

* Add cultural info to test to verify correct behavior

* address pr review to restore to restore culture

* fix comment

---------

Co-authored-by: Minh Le <leminh@microsoft.com>

* Client Encryption: Adds api FetchDataEncryptionKeyWithoutRawKeyAsync and FetchDataEncryptionKeyAsync to get DEK without and with raw key respectively.  (#3809)

* added raw key to MdeEncryption

* adding ray key to Mde Algo

* test case changes

* resolving code review comments

* code optimization to reduce keyvault calls

* removed Microsoft.Data.Encryption.Cryptography nuget package

* added api for dek with raw key

* resolved code review comments

* adding change log

* code review changes

* Initial commit (#3826)

* Query: Adds Computed Property SDK Support (#3761)

* Initial commit

* Restored settings.json changes.

* Update

* Addressed comments; still need to be tested using Emulator.

* Fixes after test run.

* Ignored the computed property tests based on the sync this morning (to allow for preview release).

* Suite0 fixes.

* Test update.

* Suite0 fixes

* [Internal] Samples: Adds OpenTelemetry and Application Insights samples (#3818)

* add opentelemetry and application insights samples

* address pr comments

* [Internal] Query: Added custom serializer coverage tests to ExpressionToSQL.cs (#3722)

* Ensure enum as string is preserved for custom serializer

* Failing test

* Added failing tests

* Updated requested names

* Ignore result of test for now

* Added additional comment on why the test is ignored

* Replaced with sample code

* Remove ignore attribute from tests, documented misbehavior for future use

* Updated comment

---------

Co-authored-by: leminh98 <leminh.ams@gmail.com>

* Query: Added remaining Cosmos Type checking functions to CosmosLinqExtensions (#3724)

* Added the remaining Cosmos Type checking functions to the CosmosLinqExtensions

* Added comments requested

* Updated comment

* Updated baseline

* Improve readability of dictionary initialization

* Aligned with code style guide

* Revert change to baseline

* Executed update baseline script

---------

Co-authored-by: neildsh <35383880+neildsh@users.noreply.github.com>
Co-authored-by: leminh98 <leminh.ams@gmail.com>

* update sdk version and section tags (#3841)

* PackageLicense: Removes PackageLicenseUrl and Adds PackageLicenseFile since PackageLicenseUrl is deprecated (#3847)

* proposal to add PackageLicenseFile since PackageLicenseUrl is deprecated. NuGet/Home#4628

* adding attribute Visible=false

* making ChangeFeedMode.LatestVersion accessible to the public (#3854)

* Code changes to update msdata/direct repo.

---------

Co-authored-by: Matias Quaranta <ealsur@users.noreply.github.com>
Co-authored-by: neildsh <35383880+neildsh@users.noreply.github.com>
Co-authored-by: Sourabh Jain <sourabh.jain.107@gmail.com>
Co-authored-by: Sourabh Jain <sourabhjain@microsoft.com>
Co-authored-by: aavasthy <113193425+aavasthy@users.noreply.github.com>
Co-authored-by: Kiran Kumar Kolli <kirankk@microsoft.com>
Co-authored-by: Aditya <adityasa@users.noreply.github.com>
Co-authored-by: Fabian Meiswinkel <fabian@meiswinkel.com>
Co-authored-by: akotalwar <94020786+akotalwar@users.noreply.github.com>
Co-authored-by: Nalu Tripician <nalutrip@icloud.com>
Co-authored-by: Nalu Tripician <ntripician@microsoft.com>
Co-authored-by: Pramod Valavala <43602528+PramodValavala-MSFT@users.noreply.github.com>
Co-authored-by: Ezra Haleva <115735172+ezrahaleva-msft@users.noreply.github.com>
Co-authored-by: Vivek Ravindran <vivekra@microsoft.com>
Co-authored-by: Prasad Ullal <36418906+prasadu-microsoft@users.noreply.github.com>
Co-authored-by: Philip Thomas <86612891+philipthomas-MSFT@users.noreply.github.com>
Co-authored-by: Arthur Augsten <arthuraugsten@gmail.com>
Co-authored-by: Augsten <aaugsten@harbourvest.com>
Co-authored-by: Ruben Bartelink <ruben@bartelink.com>
Co-authored-by: Santosh Kulkarni <66682828+kr-santosh@users.noreply.github.com>
Co-authored-by: pravengithub <124255233+pravengithub@users.noreply.github.com>
Co-authored-by: DESKTOP-ED57J7H\Prashanth Venkataram <vprashanth1987@hotmail.com>
Co-authored-by: Justine Cocchi <jucocchi@microsoft.com>
Co-authored-by: Abhijeet Mohanty <mabhijeet1995@gmail.com>
Co-authored-by: Achint-Agrawal <45819170+Achint-Agrawal@users.noreply.github.com>
Co-authored-by: Nimit Shah <nimit.svnit@gmail.com>
Co-authored-by: SaurabhSharma-MSFT <38112130+SaurabhSharma-MSFT@users.noreply.github.com>
Co-authored-by: leminh98 <leminh.ams@gmail.com>
Co-authored-by: Minh Le <leminh@microsoft.com>
Co-authored-by: Nalu Tripician <27316859+NaluTripician@users.noreply.github.com>
Co-authored-by: Scott Addie <10702007+scottaddie@users.noreply.github.com>
Co-authored-by: Kevin Pilch <me@pilchie.com>
Co-authored-by: Amaan-Haque <54414760+Amaan-Haque@users.noreply.github.com>
Co-authored-by: Amaan Haque <t-amaanhaque@microsoft.com>
Co-authored-by: Amaan Haque <amaanhaque@microsoft.com>
Co-authored-by: Baltima <130716409+Baltima@users.noreply.github.com>
Co-authored-by: vipulvishal-ms <110802706+vipulvishal-ms@users.noreply.github.com>
Co-authored-by: Erik O'Leary <969938+onionhammer@users.noreply.github.com>
  • Loading branch information
1 parent 94e811f commit 512a737
Show file tree
Hide file tree
Showing 174 changed files with 12,975 additions and 5,324 deletions.
77 changes: 76 additions & 1 deletion .github/fabricbot.json
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,82 @@
}
]
}
},
{
"taskType": "trigger",
"capabilityId": "AutoMerge",
"subCapability": "AutoMerge",
"version": "1.0",
"config": {
"taskName": "PR Automerge",
"allowAutoMergeInstructionsWithoutLabel": false,
"mergeType": "squash",
"deleteBranches": true,
"removeLabelOnPush": true,
"label": "auto-merge",
"requireAllStatuses": false,
"requireSpecificCheckRuns": false,
"usePrDescriptionAsCommitMessage": false,
"minMinutesOpen": "60",
"enforceDMPAsStatus": true
}
},
{
"taskType": "scheduled",
"capabilityId": "ScheduledSearch",
"subCapability": "ScheduledSearch",
"version": "1.1",
"config": {
"frequency": [
{
"weekDay": 1,
"hours": [
9
],
"timezoneOffset": -7
}
],
"searchTerms": [
{
"name": "isOpen",
"parameters": {}
},
{
"name": "isIssue",
"parameters": {}
},
{
"name": "hasLabel",
"parameters": {
"label": "needs-more-information"
}
},
{
"name": "noActivitySince",
"parameters": {
"days": 14
}
},
{
"name": "noAssignees",
"parameters": {}
}
],
"taskName": "Close inactive needs-information",
"actions": [
{
"name": "addReply",
"parameters": {
"comment": "@${issueAuthor} this issue requires more information for the team to be able to help. In case this information is available, please add it and re-open the Issue."
}
},
{
"name": "closeIssue",
"parameters": {}
}
]
}
}
],
"userGroups": []
}
}
8 changes: 4 additions & 4 deletions Directory.Build.props
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<ClientOfficialVersion>3.32.2</ClientOfficialVersion>
<ClientPreviewVersion>3.32.2</ClientPreviewVersion>
<ClientOfficialVersion>3.33.0</ClientOfficialVersion>
<ClientPreviewVersion>3.33.0</ClientPreviewVersion>
<ClientPreviewSuffixVersion>preview</ClientPreviewSuffixVersion>
<DirectVersion>3.30.4</DirectVersion>
<DirectVersion>3.30.8</DirectVersion>
<EncryptionOfficialVersion>2.0.1</EncryptionOfficialVersion>
<EncryptionPreviewVersion>2.0.1</EncryptionPreviewVersion>
<EncryptionPreviewSuffixVersion>preview</EncryptionPreviewSuffixVersion>
<CustomEncryptionVersion>1.0.0-preview04</CustomEncryptionVersion>
<CustomEncryptionVersion>1.0.0-preview05</CustomEncryptionVersion>
<HybridRowVersion>1.1.0-preview3</HybridRowVersion>
<LangVersion>10.0</LangVersion>
<AboveDirBuildProps>$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))</AboveDirBuildProps>
Expand Down
5 changes: 5 additions & 0 deletions Microsoft.Azure.Cosmos.Encryption.Custom/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ Preview features are treated as a separate branch and will not be included in th
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### <a name="1.0.0-preview05"/> [1.0.0-preview05](https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Encryption.Custom/1.0.0-preview05) - 2023-04-27

#### Fixes
- [#3809](https://github.com/Azure/azure-cosmos-dotnet-v3/pull/3809) Adds api FetchDataEncryptionKeyWithoutRawKeyAsync and FetchDataEncryptionKey to get DEK without and with raw key respectively.

### <a name="1.0.0-preview04"/> [1.0.0-preview04](https://www.nuget.org/packages/Microsoft.Azure.Cosmos.Encryption.Custom/1.0.0-preview04) - 2022-08-16

#### Fixes
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,11 +166,25 @@ public async Task InitializeAsync(
this.container = containerResponse.Container;
}

/// <inheritdoc/>
public override async Task<DataEncryptionKey> FetchDataEncryptionKeyWithoutRawKeyAsync(
string id,
string encryptionAlgorithm,
CancellationToken cancellationToken)
{
return await this.FetchDekAsync(id, encryptionAlgorithm, cancellationToken);
}

/// <inheritdoc/>
public override async Task<DataEncryptionKey> FetchDataEncryptionKeyAsync(
string id,
string encryptionAlgorithm,
CancellationToken cancellationToken)
{
return await this.FetchDekAsync(id, encryptionAlgorithm, cancellationToken, true);
}

private async Task<DataEncryptionKey> FetchDekAsync(string id, string encryptionAlgorithm, CancellationToken cancellationToken, bool withRawKey = false)
{
DataEncryptionKeyProperties dataEncryptionKeyProperties = await this.dataEncryptionKeyContainerCore.FetchDataEncryptionKeyPropertiesAsync(
id,
Expand Down Expand Up @@ -200,7 +214,8 @@ public override async Task<DataEncryptionKey> FetchDataEncryptionKeyAsync(
InMemoryRawDek inMemoryRawDek = await this.dataEncryptionKeyContainerCore.FetchUnwrappedAsync(
dataEncryptionKeyProperties,
diagnosticsContext: CosmosDiagnosticsContext.Create(null),
cancellationToken: cancellationToken);
cancellationToken: cancellationToken,
withRawKey);

return inMemoryRawDek.DataEncryptionKey;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ public override async Task<byte[]> DecryptAsync(
string encryptionAlgorithm,
CancellationToken cancellationToken = default)
{
DataEncryptionKey dek = await this.DataEncryptionKeyProvider.FetchDataEncryptionKeyAsync(
DataEncryptionKey dek = await this.DataEncryptionKeyProvider.FetchDataEncryptionKeyWithoutRawKeyAsync(
dataEncryptionKeyId,
encryptionAlgorithm,
cancellationToken);

if (dek == null)
{
throw new InvalidOperationException($"Null {nameof(DataEncryptionKey)} returned from {nameof(this.DataEncryptionKeyProvider.FetchDataEncryptionKeyAsync)}.");
throw new InvalidOperationException($"Null {nameof(DataEncryptionKey)} returned from {nameof(this.DataEncryptionKeyProvider.FetchDataEncryptionKeyWithoutRawKeyAsync)}.");
}

return dek.DecryptData(cipherText);
Expand All @@ -55,14 +55,14 @@ public override async Task<byte[]> EncryptAsync(
string encryptionAlgorithm,
CancellationToken cancellationToken = default)
{
DataEncryptionKey dek = await this.DataEncryptionKeyProvider.FetchDataEncryptionKeyAsync(
DataEncryptionKey dek = await this.DataEncryptionKeyProvider.FetchDataEncryptionKeyWithoutRawKeyAsync(
dataEncryptionKeyId,
encryptionAlgorithm,
cancellationToken);

if (dek == null)
{
throw new InvalidOperationException($"Null {nameof(DataEncryptionKey)} returned from {nameof(this.DataEncryptionKeyProvider.FetchDataEncryptionKeyAsync)}.");
throw new InvalidOperationException($"Null {nameof(DataEncryptionKey)} returned from {nameof(this.DataEncryptionKeyProvider.FetchDataEncryptionKeyWithoutRawKeyAsync)}.");
}

return dek.EncryptData(plainText);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,7 @@ internal async Task<DataEncryptionKey> FetchUnWrappedMdeSupportedLegacyDekAsync(
unwrapResult.DataEncryptionKey);

return new MdeEncryptionAlgorithm(
unwrapResult.DataEncryptionKey,
plaintextDataEncryptionKey,
Data.Encryption.Cryptography.EncryptionType.Randomized);
}
Expand Down Expand Up @@ -378,13 +379,14 @@ internal async Task<DataEncryptionKey> FetchUnWrappedLegacySupportedMdeDekAsync(
internal async Task<InMemoryRawDek> FetchUnwrappedAsync(
DataEncryptionKeyProperties dekProperties,
CosmosDiagnosticsContext diagnosticsContext,
CancellationToken cancellationToken)
CancellationToken cancellationToken,
bool withRawKey = false)
{
try
{
if (string.Equals(dekProperties.EncryptionAlgorithm, CosmosEncryptionAlgorithm.MdeAeadAes256CbcHmac256Randomized))
{
DataEncryptionKey dek = this.InitMdeEncryptionAlgorithm(dekProperties);
DataEncryptionKey dek = this.InitMdeEncryptionAlgorithm(dekProperties, withRawKey);

// TTL is not used since DEK is not cached.
return new InMemoryRawDek(dek, TimeSpan.FromMilliseconds(0));
Expand Down Expand Up @@ -564,7 +566,7 @@ private async Task<EncryptionKeyUnwrapResult> UnWrapDekMdeEncAlgoAsync(
return unwrapResult;
}

internal DataEncryptionKey InitMdeEncryptionAlgorithm(DataEncryptionKeyProperties dekProperties)
internal DataEncryptionKey InitMdeEncryptionAlgorithm(DataEncryptionKeyProperties dekProperties, bool withRawKey = false)
{
if (this.DekProvider.MdeKeyWrapProvider == null)
{
Expand All @@ -576,7 +578,8 @@ internal DataEncryptionKey InitMdeEncryptionAlgorithm(DataEncryptionKeyPropertie
dekProperties,
Data.Encryption.Cryptography.EncryptionType.Randomized,
this.DekProvider.MdeKeyWrapProvider.EncryptionKeyStoreProvider,
this.DekProvider.PdekCacheTimeToLive);
this.DekProvider.PdekCacheTimeToLive,
withRawKey);
}

private async Task<DataEncryptionKeyProperties> ReadResourceAsync(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,19 @@ namespace Microsoft.Azure.Cosmos.Encryption.Custom
public abstract class DataEncryptionKeyProvider
{
/// <summary>
/// Retrieves the data encryption key for the given id.
/// Retrieves the data encryption key for the given id without rawkey. RawKey will be set to null.
/// </summary>
/// <param name="id">Identifier of the data encryption key.</param>
/// <param name="encryptionAlgorithm">Encryption algorithm that the retrieved key will be used with.</param>
/// <param name="cancellationToken">Token for request cancellation.</param>
/// <returns>Data encryption key bytes.</returns>
public abstract Task<DataEncryptionKey> FetchDataEncryptionKeyWithoutRawKeyAsync(
string id,
string encryptionAlgorithm,
CancellationToken cancellationToken);

/// <summary>
/// Retrieves the data encryption key for the given id with RawKey value.
/// </summary>
/// <param name="id">Identifier of the data encryption key.</param>
/// <param name="encryptionAlgorithm">Encryption algorithm that the retrieved key will be used with.</param>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -791,13 +791,6 @@ public override Task<IReadOnlyList<FeedRange>> GetFeedRangesAsync(
return this.container.GetFeedRangesAsync(cancellationToken);
}

public override Task<IEnumerable<string>> GetPartitionKeyRangesAsync(
FeedRange feedRange,
CancellationToken cancellationToken = default)
{
return this.container.GetPartitionKeyRangesAsync(feedRange, cancellationToken);
}

public override FeedIterator GetItemQueryStreamIterator(
FeedRange feedRange,
QueryDefinition queryDefinition,
Expand Down Expand Up @@ -1010,6 +1003,14 @@ public override async Task<FeedResponse<T>> ReadManyItemsAsync<T>(
return this.ResponseFactory.CreateItemFeedResponse<T>(responseMessage);
}

#if ENCRYPTIONPREVIEW
public override Task<IEnumerable<string>> GetPartitionKeyRangesAsync(
FeedRange feedRange,
CancellationToken cancellationToken = default)
{
return this.container.GetPartitionKeyRangesAsync(feedRange, cancellationToken);
}

public override Task<ResponseMessage> DeleteAllItemsByPartitionKeyStreamAsync(
Cosmos.PartitionKey partitionKey,
RequestOptions requestOptions = null,
Expand All @@ -1020,6 +1021,7 @@ public override Task<ResponseMessage> DeleteAllItemsByPartitionKeyStreamAsync(
requestOptions,
cancellationToken);
}
#endif

private async Task<ResponseMessage> ReadManyItemsHelperAsync(
IReadOnlyList<(string id, PartitionKey partitionKey)> items,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ internal sealed class MdeEncryptionAlgorithm : DataEncryptionKey
{
private readonly AeadAes256CbcHmac256EncryptionAlgorithm mdeAeadAes256CbcHmac256EncryptionAlgorithm;

private readonly byte[] unwrapKey;

// unused for MDE Algorithm.
public override byte[] RawKey => null;
public override byte[] RawKey { get; }

public override string EncryptionAlgorithm => CosmosEncryptionAlgorithm.MdeAeadAes256CbcHmac256Randomized;

Expand All @@ -32,7 +34,8 @@ public MdeEncryptionAlgorithm(
DataEncryptionKeyProperties dekProperties,
Data.Encryption.Cryptography.EncryptionType encryptionType,
EncryptionKeyStoreProvider encryptionKeyStoreProvider,
TimeSpan? cacheTimeToLive)
TimeSpan? cacheTimeToLive,
bool withRawKey=false)
{
if (dekProperties == null)
{
Expand All @@ -49,36 +52,39 @@ public MdeEncryptionAlgorithm(
dekProperties.EncryptionKeyWrapMetadata.Value,
encryptionKeyStoreProvider);

ProtectedDataEncryptionKey protectedDataEncryptionKey;
if (cacheTimeToLive.HasValue)
if (!withRawKey)
{
// no caching
if (cacheTimeToLive.Value == TimeSpan.Zero)
{
protectedDataEncryptionKey = new ProtectedDataEncryptionKey(
ProtectedDataEncryptionKey protectedDataEncryptionKey = cacheTimeToLive.HasValue && cacheTimeToLive.Value == TimeSpan.Zero
? new ProtectedDataEncryptionKey(
dekProperties.Id,
keyEncryptionKey,
dekProperties.WrappedDataEncryptionKey)
: ProtectedDataEncryptionKey.GetOrCreate(
dekProperties.Id,
keyEncryptionKey,
dekProperties.WrappedDataEncryptionKey);
}
else
{
protectedDataEncryptionKey = ProtectedDataEncryptionKey.GetOrCreate(
dekProperties.Id,
keyEncryptionKey,
dekProperties.WrappedDataEncryptionKey);
}
this.mdeAeadAes256CbcHmac256EncryptionAlgorithm = AeadAes256CbcHmac256EncryptionAlgorithm.GetOrCreate(
protectedDataEncryptionKey,
encryptionType);
}
else
{
protectedDataEncryptionKey = ProtectedDataEncryptionKey.GetOrCreate(
dekProperties.Id,
keyEncryptionKey,
dekProperties.WrappedDataEncryptionKey);
byte[] rawKey = keyEncryptionKey.DecryptEncryptionKey(dekProperties.WrappedDataEncryptionKey);
PlaintextDataEncryptionKey plaintextDataEncryptionKey = cacheTimeToLive.HasValue && (cacheTimeToLive.Value == TimeSpan.Zero)
? new PlaintextDataEncryptionKey(
dekProperties.Id,
rawKey)
: PlaintextDataEncryptionKey.GetOrCreate(
dekProperties.Id,
rawKey);
this.RawKey = rawKey;
this.mdeAeadAes256CbcHmac256EncryptionAlgorithm = AeadAes256CbcHmac256EncryptionAlgorithm.GetOrCreate(
plaintextDataEncryptionKey,
encryptionType);

}

this.mdeAeadAes256CbcHmac256EncryptionAlgorithm = AeadAes256CbcHmac256EncryptionAlgorithm.GetOrCreate(
protectedDataEncryptionKey,
encryptionType);

}

/// <summary>
Expand All @@ -90,9 +96,11 @@ public MdeEncryptionAlgorithm(
/// <param name="dataEncryptionKey"> Data Encryption Key </param>
/// <param name="encryptionType"> Encryption type </param>
public MdeEncryptionAlgorithm(
byte[] rawkey,
Data.Encryption.Cryptography.DataEncryptionKey dataEncryptionKey,
Data.Encryption.Cryptography.EncryptionType encryptionType)
{
this.RawKey = rawkey;
this.mdeAeadAes256CbcHmac256EncryptionAlgorithm = AeadAes256CbcHmac256EncryptionAlgorithm.GetOrCreate(
dataEncryptionKey,
encryptionType);
Expand Down
Loading

0 comments on commit 512a737

Please sign in to comment.