Issues with Azure Data-lake store applying the permissions.

[thunderfalcon]

Issues with Azure Data-lake store applying the permissions.

The new Az module have issues applying permissions.

Set-AzDataLakeStoreItemAclEntry -AccountName "storename" -Path /-AceType User -Id (ID)-Permissions All -Default -Recurse

AzDatalakeStoreItemAclEntry is throwing error

<# Set-AzDataLakeStoreItemAclEntry : Error in modifying ACL entries default:user::rwx for
path /.
Operation: MODIFYACLENTRIES failed with Unknown Error: The 'User-Agent' header must be modified using the appropriate
property or method.
Parameter name: name Source: System StackTrace: at System.Net.WebHeaderCollection.ThrowOnRestrictedHeader(String
headerName)
at System.Net.WebHeaderCollection.Set(String name, String value)
at Microsoft.Azure.DataLake.Store.WebTransport.AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client,
RequestOptions req, String token, String opMethod, IDictionary`2 customHeaders, Int32 postRequestLength)
at Microsoft.Azure.DataLake.Store.WebTransport.d__22.MoveNext().
Last encountered exception thrown after 5 tries. [The 'User-Agent' header must be modified using the appropriate property or
method.
Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name]
[ServerRequestId:]
At line:1 char:1

• Set-AzDataLakeStoreItemAclEntry -AccountName "" -Pa ...

•   + CategoryInfo          : CloseError: (:) [Set-AzDataLakeStoreItemAclEntry], AdlsException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.DataLakeStore.SetAzureDataLakeStoreItemAclEntry #>
  
  

I reverted back to AzureRm module and everything works smooth.
This fix needs to be deployed.

[markcowl]

@rahuldutta90 Can you take a look?

[rahuldutta90]

@thunderfalcon please refer this: #8141

To use datalake az module you have to use it in netcore powershell (not windows powershell)

[migrande]

This is a blocker for me right now. It's not practical to require netcore PowerShell for the users that will be running the script I'm writing. Can this be resolved?

[rahuldutta90]

@migrande What is the blocker for you in using netcore powershell?

Otherwise you can use AzureRm.DataLakeStore module in windows powershell.

Currently we have a SDK that can fix it, but it is beta version: https://www.nuget.org/packages/Microsoft.Azure.DataLake.Store/1.2.0-alpha , This is alpha not a major release because of a .net issue with one of the core components (whicn wont be fixed).

Powershell team wont allow the alpha version to be merged to major release. WE can do a beta release of the Az.DataLakeStore module. Will that work for you?

[migrande]

@rahuldutta90 We are passing off a script to a client for use in DR scenarios. Requiring that they procure and install netcore PowerShell isn't really viable. Outside of hardcore PowerShell/Azure folks, I don't think netcore PS is widely known.

I could revert these scripts to AzureRM but that presents a development issue. I intentionally don't have the AzureRM modules anymore, and the scripts are already written to use the Az modules. I could switch them back to AzureRM, but that isn't worth it for this one bug. I can have the customer manually apply the Data Lake Store ACLs for now.

However, it's frustrating to come across issues that are a regression in Az from AzureRM. This type of issue should make it to the Breaking Changes page in the Az migration documentation.

[rahuldutta90]

@migrande we have fixed this issue. Apologize for the inconvenience. It will be released as part of next release.

[guntaled]

Was this issue fixed? I'm getting 'The 'User-Agent' header must be modified using the appropriate property or method.' for Get-AzDataLakeStoreChildItem.

[rahuldutta90]

@guntaled The fix was made but it was not released as May build because I missed updating the changelog. I fixed that (#9153) and now the changes will be released as 05-21 build.
I am assuming you will see a change in https://www.powershellgallery.com/packages/Az.DataLakeStore/ after 05-21.

[rahuldutta90]

@guntaled @migrande Can you please try https://www.powershellgallery.com/packages/Az.DataLakeStore/1.2.1

[guntaled]

@rahuldutta90, it works on my local machine, but not on Automation runbooks.

[rahuldutta90]

@guntaled Can you please check the version of Az.DataLakeStore in automation runbooks?

[guntaled]

@rahuldutta90 , it is 1.2.1, and it works. It was another issue this time, but the error it threw was the same, so I just assumed. It seems to all be working now.

Thanks!

[rahuldutta90]

I am closing the issue. @migrande if the new version does not resolve the issue, please reopen it.

[darrenstilesag]

Is this issue supposed to have been fixed by now? I am still getting the same error as previously reported.

[rahuldutta90]

@darrenstilesag this should have been fixed. Which version of az.datalakestore module you are facing issue with?

[darrenstilesag]

@rahuldutta90 i am using version 1.2.3

[darrenstilesag]

@rahuldutta90 any update on this issue? thanks

[rahuldutta90]

@darrenstilesag Do you have other versions installed? Meaning do you have other versions of Az.datalakestore or Azurerm.Datalakestore installed? You can check "Get-Module -Name AzM.DataLakeStore –ListAvailable". Make sure all otehr versions are uninstalled.

After that Can you also paste what error you are facing? Run the commandlet in "-debug" mode

[darrenstilesag]

Hi Rahul, I am now running just version 1.2.6 of az.datalakestore, older and other versions have now been removed. Can I you send me your email address so I can send you the debug output, otherwise I’ll have to redact loads of information before sending (company policy). Thanks Darren

[rahuldutta90]

@darrenstilesag is this the same error as "Unknown Error: The 'User-Agent' header must be modified using the appropriate property or method."? Or is it some thing like "Could not load file or assembly 'System.Buffers, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies."

You do not have to paste all the debug lines. Basically paste the stacktrace from our code.

[darrenstilesag]

@rahuldutta90 error details below.

DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:e06b4728-ec9d-4a4d-9155-2e0499ce8c74.1,lat:10,err:An error occurred while sending the request.,Reqlen:0
,Resplen:0,token_ns:0,sReqId:,path:,qp:op=MODIFYACLENTRIES&aclspec=group%%3Arwx&api-version=2018-09-01, connectionfailure:True

Operation: MODIFYACLENTRIES failed with Unknown Error: An error occurred while sending the request. Source: mscorlib StackTrace: at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.DataLake.Store.WebTransport.d__20.MoveNext().
.
Last encountered exception thrown after 5 tries. [An error occurred while sending the request.,An error occurred while sending the request.,An error occurred while sending the request.,An
error occurred while sending the request.,An error occurred while sending the request.]
[ServerRequestId:]
At line:2 char:1

• Set-AzDataLakeStoreItemAclEntry -Account "" -Path " ...

•   + CategoryInfo          : CloseError: (:) [Set-AzDataLakeStoreItemAclEntry], AdlsException
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.DataLakeStore.SetAzureDataLakeStoreItemAclEntry
  
  

DEBUG: AzureQoSEvent: CommandName - Set-AzDataLakeStoreItemAclEntry; IsSuccess - False; Duration - 00:02:24.7060535; Exception - Microsoft.Azure.DataLake.Store.AdlsException: Error in modifyin
g ACL entries group::rwx for path .
Operation: MODIFYACLENTRIES failed with Unknown Error: An error occurred while sending the request. Source: mscorlib StackTrace: at System.Runtime.ExceptionServices.ExceptionDispatchInfo.
Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.DataLake.Store.WebTransport.d__20.MoveNext().
.
Last encountered exception thrown after 5 tries. [An error occurred while sending the request.,An error occurred while sending the request.,An error occurred while sending the request.,An erro
r occurred while sending the request.,An error occurred while sending the request.]
[ServerRequestId:]
at Microsoft.Azure.DataLake.Store.AdlsClient.d__110.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.DataLake.Store.AdlsClient.ModifyAclEntries(String path, List1 aclSpec, CancellationToken cancelToken) at Microsoft.Azure.Commands.DataLakeStore.Models.DataLakeStoreFileSystemClient.ModifyAcl(String path, String accountName, List1 aclToModify)
at Microsoft.Azure.Commands.DataLakeStore.SetAzureDataLakeStoreItemAclEntry.<>c__DisplayClass46_0.b__2()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(String processMessage, String target, Action action)
at Microsoft.Azure.Commands.DataLakeStore.SetAzureDataLakeStoreItemAclEntry.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord();
DEBUG: Finish sending metric.

[rahuldutta90]

@darrenstilesag this is not the same issue related to this thread. This is something different. I suspect something is wrong with the full account and domain name or connectivity to the ADLS account from wherever you are running this command from. Just to rule out some stuff, please try to apss fqdn(contoso.azuredatalakestore.net) to the "Account". Also try to do ping ("ping contoso.azuredatalakestore.net") and see if its resolvable or not. Also try to run these commandlet from a Azure VM.

Either way I am assuming you wont be able to share a whole lot of details here, so you can open a support ticket on ADLS.

[darrenstilesag]

@rahuldutta90 I can perform ACL changes via Storage Explorer on my PC, because this works should Powershell work? I cannot see contoso.azuredatalakestore.net and I don't have permissions to run these commands on our Azure VM. I changed my code to include the contoso.azuredatalakestore.net and I got the same error. Does this mean a security issue on my side? Thanks