Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import-AzDataLakeStoreItem fails with Operation: GETFILESTATUS failed with Unknown Error: Specified value has invalid CRLF characters. #8141

Closed
kenogr opened this issue Dec 14, 2018 · 13 comments
Assignees
Labels
Data Lake Store Service Attention This issue is responsible by Azure service team.

Comments

@kenogr
Copy link

kenogr commented Dec 14, 2018

Description

Import-AzDataLakeStoreItem fails with Operation: GETFILESTATUS failed with Unknown Error: Specified value has invalid CRLF characters.

Script/Steps for Reproduction

Import-AzDataLakeStoreItem -AccountName "coscondash" -Path "cosmosAuditLog.csv" -Destination "/cooked/cosmosAuditLog.csv" -Force

Module Version

    Directory: C:\Users\v-keg\Documents\WindowsPowerShell\Modules


ModuleType Version    Name                                ExportedCommands                                                                          
---------- -------    ----                                ----------------                                                                          
Binary     1.0.295    Cosmos                              {Get-CosmosVirtualCluster, Get-CosmosCluster, Get-CosmosSDK, Export-CosmosSDK...}         


    Directory: C:\Program Files\WindowsPowerShell\Modules


ModuleType Version    Name                                ExportedCommands                                                                          
---------- -------    ----                                ----------------                                                                          
Script     0.7.0      Az.Aks                              {Get-AzAks, New-AzAks, Remove-AzAks, Import-AzAksCredential...}                           
Script     0.7.0      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServi...
Script     0.7.0      Az.ApiManagement                    {Add-AzApiManagementRegion, Get-AzApiManagementSsoToken, New-AzApiManagementHostnameCon...
Script     0.7.0      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsights, Se...
Script     0.7.0      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-AzAutomat...
Script     0.7.0      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKeys, New-AzBatchAccount...}
Script     0.7.0      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount}                      
Script     0.7.0      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...}      
Script     0.7.0      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiveServ...
Script     0.7.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-AzAvail...
Script     0.7.0      Az.Consumption                      {Get-AzConsumptionBudget, Get-AzConsumptionMarketplace, Get-AzConsumptionPriceSheet, Ge...
Script     0.7.0      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzContainerIn...
Script     0.7.0      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, Remove-A...
Script     0.7.0      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remove-Az...
Script     0.7.0      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Remove-...
Script     0.7.0      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, ...
Script     0.7.0      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecordSet...} 
Script     0.7.0      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGridTopic...
Script     0.7.0      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remove-AzEv...
Script     0.7.0      Az.Insights                         {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...}          
Script     0.7.0      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString, Get-...
Script     0.7.0      Az.KeyVault                         {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertificateOpe...
Script     0.7.0      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountCallbackUrl, Get-AzIntegrat...
Script     0.7.0      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlanU...
Script     0.7.0      Az.MachineLearningCompute           {Get-AzMlOpCluster, Get-AzMlOpClusterKey, Test-AzMlOpClusterSystemServicesUpdateAvailab...
Script     0.7.0      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}                                          
Script     0.7.0      Az.Media                            {Sync-AzMediaServiceStorageKeys, Set-AzMediaServiceKey, Get-AzMediaServiceKeys, Get-AzM...
Script     0.7.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthenticat...
Script     0.7.0      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRules, Get-AzNotificationHubL...
Script     0.7.0      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCustomLo...
Script     0.7.0      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemediatio...
Script     0.7.0      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWo...
Script     0.7.0      Az.Profile                          {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-A...
Script     0.7.0      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecoveryServi...
Script     0.7.0      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCachePatch...
Script     0.7.0      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRelayNamesp...
Script     0.7.0      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssi...
Script     0.7.0      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace, Remov...
Script     0.7.0      Az.ServiceFabric                    {Add-AzServiceFabricApplicationCertificate, Add-AzServiceFabricClientCertificate, Add-A...
Script     0.7.0      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...}                     
Script     0.7.0      Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryption...
Script     0.7.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAcco...
Script     0.7.0      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, New-AzS...
Script     0.7.0      Az.Tags                             {Remove-AzTag, Get-AzTag, New-AzTag}                                                      
Script     0.7.0      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFromEnd...
Script     0.7.0      Az.UsageAggregates                  Get-UsageAggregates                                                                       
Script     0.7.0      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppServiceP...
Script     5.1.2      Azure                               {Get-AzureAutomationCertificate, Get-AzureAutomationConnection, New-AzureAutomationConn...
Binary     2.0.1.16   AzureAD                             {Add-AzureADApplicationOwner, Get-AzureADApplication, Get-AzureADApplicationExtensionPr...
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}                                     
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}                    
Script     3.4.0      Pester                              {Describe, Context, It, Should...}                                                        
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}                              
Script     2.0.0      PSReadline                          {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHandler, Get-P...
Script     21.1.18068 SqlServer                           {Add-RoleMember, Add-SqlAvailabilityDatabase, Add-SqlAvailabilityGroupListenerStaticIp,...
Script     21.0.17224 SqlServer                           {Add-SqlFirewallRule, ConvertFrom-EncodedSqlName, New-SqlAvailabilityGroup, Invoke-Proc...


    Directory: C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules


ModuleType Version    Name                                ExportedCommands                                                                          
---------- -------    ----                                ----------------                                                                          
Manifest   1.0.0.0    AppBackgroundTask                   {Disable-AppBackgroundTaskDiagnosticLog, Enable-AppBackgroundTaskDiagnosticLog, Set-App...
Manifest   2.0.0.0    AppLocker                           {Get-AppLockerFileInformation, Get-AppLockerPolicy, New-AppLockerPolicy, Set-AppLockerP...
Manifest   1.0.0.0    AppvClient                          {Add-AppvClientConnectionGroup, Add-AppvClientPackage, Add-AppvPublishingServer, Disabl...
Manifest   2.0.1.0    Appx                                {Add-AppxPackage, Get-AppxPackage, Get-AppxPackageManifest, Remove-AppxPackage...}        
Script     1.0.0.0    AssignedAccess                      {Clear-AssignedAccess, Get-AssignedAccess, Set-AssignedAccess}                            
Manifest   1.0.0.0    BitLocker                           {Unlock-BitLocker, Suspend-BitLocker, Resume-BitLocker, Remove-BitLockerKeyProtector...}  
Manifest   2.0.0.0    BitsTransfer                        {Add-BitsFile, Complete-BitsTransfer, Get-BitsTransfer, Remove-BitsTransfer...}           
Manifest   1.0.0.0    BranchCache                         {Add-BCDataCacheExtension, Clear-BCCache, Disable-BC, Disable-BCDowngrading...}           
Manifest   1.0.0.0    CimCmdlets                          {Get-CimAssociatedInstance, Get-CimClass, Get-CimInstance, Get-CimSession...}             
Manifest   1.0        ConfigCI                            {Get-SystemDriver, New-CIPolicyRule, New-CIPolicy, Get-CIPolicy...}                       
Manifest   1.0        Defender                            {Get-MpPreference, Set-MpPreference, Add-MpPreference, Remove-MpPreference...}            
Manifest   1.0.1.0    DeliveryOptimization                {Get-DeliveryOptimizationStatus, Get-DeliveryOptimizationPerfSnap, Get-DeliveryOptimiza...
Manifest   1.0.0.0    DirectAccessClientComponents        {Disable-DAManualEntryPointSelection, Enable-DAManualEntryPointSelection, Get-DAClientE...
Script     3.0        Dism                                {Add-AppxProvisionedPackage, Add-WindowsDriver, Add-WindowsCapability, Add-WindowsImage...
Manifest   1.0.0.0    DnsClient                           {Resolve-DnsName, Clear-DnsClientCache, Get-DnsClient, Get-DnsClientCache...}             
Manifest   1.0.0.0    EventTracingManagement              {Start-EtwTraceSession, New-EtwTraceSession, Get-EtwTraceSession, Update-EtwTraceSessio...
Manifest   2.0.0.0    International                       {Get-WinDefaultInputMethodOverride, Set-WinDefaultInputMethodOverride, Get-WinHomeLocat...
Manifest   1.0.0.0    iSCSI                               {Get-IscsiTargetPortal, New-IscsiTargetPortal, Remove-IscsiTargetPortal, Update-IscsiTa...
Script     1.0.0.0    ISE                                 {New-IseSnippet, Import-IseSnippet, Get-IseSnippet}                                       
Manifest   1.0.0.0    Kds                                 {Add-KdsRootKey, Get-KdsRootKey, Test-KdsRootKey, Set-KdsConfiguration...}                
Manifest   1.0.1.0    Microsoft.PowerShell.Archive        {Compress-Archive, Expand-Archive}                                                        
Manifest   3.0.0.0    Microsoft.PowerShell.Diagnostics    {Get-WinEvent, Get-Counter, Import-Counter, Export-Counter...}                            
Manifest   3.0.0.0    Microsoft.PowerShell.Host           {Start-Transcript, Stop-Transcript}                                                       
Manifest   1.0.0.0    Microsoft.PowerShell.LocalAccounts  {Add-LocalGroupMember, Disable-LocalUser, Enable-LocalUser, Get-LocalGroup...}            
Manifest   3.1.0.0    Microsoft.PowerShell.Management     {Add-Content, Clear-Content, Clear-ItemProperty, Join-Path...}                            
Script     1.0        Microsoft.PowerShell.ODataUtils     Export-ODataEndpointProxy                                                                 
Manifest   3.0.0.0    Microsoft.PowerShell.Security       {Get-Acl, Set-Acl, Get-PfxCertificate, Get-Credential...}                                 
Manifest   3.1.0.0    Microsoft.PowerShell.Utility        {Format-List, Format-Custom, Format-Table, Format-Wide...}                                
Manifest   3.0.0.0    Microsoft.WSMan.Management          {Disable-WSManCredSSP, Enable-WSManCredSSP, Get-WSManCredSSP, Set-WSManQuickConfig...}    
Manifest   1.0        MMAgent                             {Disable-MMAgent, Enable-MMAgent, Set-MMAgent, Get-MMAgent...}                            
Manifest   1.0.0.0    MsDtc                               {New-DtcDiagnosticTransaction, Complete-DtcDiagnosticTransaction, Join-DtcDiagnosticRes...
Manifest   2.0.0.0    NetAdapter                          {Disable-NetAdapter, Disable-NetAdapterBinding, Disable-NetAdapterChecksumOffload, Disa...
Manifest   1.0.0.0    NetConnection                       {Get-NetConnectionProfile, Set-NetConnectionProfile}                                      
Manifest   1.0.0.0    NetDiagnostics                      Get-NetView                                                                               
Manifest   1.0.0.0    NetEventPacketCapture               {New-NetEventSession, Remove-NetEventSession, Get-NetEventSession, Set-NetEventSession...}
Manifest   2.0.0.0    NetLbfo                             {Add-NetLbfoTeamMember, Add-NetLbfoTeamNic, Get-NetLbfoTeam, Get-NetLbfoTeamMember...}    
Manifest   1.0.0.0    NetNat                              {Get-NetNat, Get-NetNatExternalAddress, Get-NetNatStaticMapping, Get-NetNatSession...}    
Manifest   2.0.0.0    NetQos                              {Get-NetQosPolicy, Set-NetQosPolicy, Remove-NetQosPolicy, New-NetQosPolicy}               
Manifest   2.0.0.0    NetSecurity                         {Get-DAPolicyChange, New-NetIPsecAuthProposal, New-NetIPsecMainModeCryptoProposal, New-...
Manifest   1.0.0.0    NetSwitchTeam                       {New-NetSwitchTeam, Remove-NetSwitchTeam, Get-NetSwitchTeam, Rename-NetSwitchTeam...}     
Manifest   1.0.0.0    NetTCPIP                            {Get-NetIPAddress, Get-NetIPInterface, Get-NetIPv4Protocol, Get-NetIPv6Protocol...}       
Manifest   1.0.0.0    NetworkConnectivityStatus           {Get-DAConnectionStatus, Get-NCSIPolicyConfiguration, Reset-NCSIPolicyConfiguration, Se...
Manifest   1.0.0.0    NetworkSwitchManager                {Disable-NetworkSwitchEthernetPort, Enable-NetworkSwitchEthernetPort, Get-NetworkSwitch...
Manifest   1.0.0.0    NetworkTransition                   {Add-NetIPHttpsCertBinding, Disable-NetDnsTransitionConfiguration, Disable-NetIPHttpsPr...
Manifest   1.0.0.0    PcsvDevice                          {Get-PcsvDevice, Start-PcsvDevice, Stop-PcsvDevice, Restart-PcsvDevice...}                
Binary     1.0.0.0    PersistentMemory                    {Get-PmemDisk, Get-PmemPhysicalDevice, Get-PmemUnusedRegion, New-PmemDisk...}             
Manifest   1.0.0.0    PKI                                 {Add-CertificateEnrollmentPolicyServer, Export-Certificate, Export-PfxCertificate, Get-...
Manifest   1.0.0.0    PnpDevice                           {Get-PnpDevice, Get-PnpDeviceProperty, Enable-PnpDevice, Disable-PnpDevice}               
Manifest   1.1        PrintManagement                     {Add-Printer, Add-PrinterDriver, Add-PrinterPort, Get-PrintConfiguration...}              
Binary     1.0.11     ProcessMitigations                  {Get-ProcessMitigation, Set-ProcessMitigation, ConvertTo-ProcessMitigationPolicy}         
Script     3.0        Provisioning                        {Install-ProvisioningPackage, Export-ProvisioningPackage, Install-TrustedProvisioningCe...
Manifest   1.1        PSDesiredStateConfiguration         {Set-DscLocalConfigurationManager, Start-DscConfiguration, Test-DscConfiguration, Publi...
Script     1.0.0.0    PSDiagnostics                       {Disable-PSTrace, Disable-PSWSManCombinedTrace, Disable-WSManTrace, Enable-PSTrace...}    
Binary     1.1.0.0    PSScheduledJob                      {New-JobTrigger, Add-JobTrigger, Remove-JobTrigger, Get-JobTrigger...}                    
Manifest   2.0.0.0    PSWorkflow                          {New-PSWorkflowExecutionOption, New-PSWorkflowSession, nwsn}                              
Manifest   1.0.0.0    PSWorkflowUtility                   Invoke-AsWorkflow                                                                         
Manifest   1.0.0.0    ScheduledTasks                      {Get-ScheduledTask, Set-ScheduledTask, Register-ScheduledTask, Unregister-ScheduledTask...
Manifest   2.0.0.0    SecureBoot                          {Confirm-SecureBootUEFI, Set-SecureBootUEFI, Get-SecureBootUEFI, Format-SecureBootUEFI...}
Manifest   2.0.0.0    SmbShare                            {Get-SmbShare, Remove-SmbShare, Set-SmbShare, Block-SmbShareAccess...}                    
Manifest   2.0.0.0    SmbWitness                          {Get-SmbWitnessClient, Move-SmbWitnessClient, gsmbw, msmbw...}                            
Manifest   1.0.0.0    StartLayout                         {Export-StartLayout, Import-StartLayout, Export-StartLayoutEdgeAssets, Get-StartApps}     
Manifest   2.0.0.0    Storage                             {Add-InitiatorIdToMaskingSet, Add-PartitionAccessPath, Add-PhysicalDisk, Add-StorageFau...
Manifest   1.0.0.0    StorageBusCache                     {Clear-StorageBusDisk, Disable-StorageBusCache, Disable-StorageBusDisk, Enable-StorageB...
Manifest   2.0.0.0    TLS                                 {New-TlsSessionTicketKey, Enable-TlsSessionTicketKey, Disable-TlsSessionTicketKey, Expo...
Manifest   1.0.0.0    TroubleshootingPack                 {Get-TroubleshootingPack, Invoke-TroubleshootingPack}                                     
Manifest   2.0.0.0    TrustedPlatformModule               {Get-Tpm, Initialize-Tpm, Clear-Tpm, Unblock-Tpm...}                                      
Binary     2.1.639.0  UEV                                 {Clear-UevConfiguration, Clear-UevAppxPackage, Restore-UevBackup, Set-UevTemplateProfil...
Manifest   2.0.0.0    VpnClient                           {Add-VpnConnection, Set-VpnConnection, Remove-VpnConnection, Get-VpnConnection...}        
Manifest   1.0.0.0    Wdac                                {Get-OdbcDriver, Set-OdbcDriver, Get-OdbcDsn, Add-OdbcDsn...}                             
Manifest   1.0.0.0    WindowsDeveloperLicense             {Get-WindowsDeveloperLicense, Unregister-WindowsDeveloperLicense, Show-WindowsDeveloper...
Script     1.0        WindowsErrorReporting               {Enable-WindowsErrorReporting, Disable-WindowsErrorReporting, Get-WindowsErrorReporting}  
Manifest   1.0.0.0    WindowsSearch                       {Get-WindowsSearchSetting, Set-WindowsSearchSetting}                                      
Manifest   1.0.0.0    WindowsUpdate                       Get-WindowsUpdateLog                                                                      
Manifest   1.0.0.2    WindowsUpdateProvider               {Get-WUAVersion, Get-WULastInstallationDate, Get-WULastScanSuccessDate, Get-WUIsPending...


    Directory: C:\Program Files (x86)\Microsoft Azure Information Protection\Powershell


ModuleType Version    Name                                ExportedCommands                                                                          
---------- -------    ----                                ----------------                                                                          
Binary     1.37.19.0  AzureInformationProtection          {Clear-RMSAuthentication, Get-RMSFileStatus, Get-RMSServer, Get-RMSServerAuthentication...

Environment Data

Name                           Value                                                                                                                
----                           -----                                                                                                                
PSVersion                      5.1.17763.134                                                                                                        
PSEdition                      Desktop                                                                                                              
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                              
BuildVersion                   10.0.17763.134                                                                                                       
CLRVersion                     4.0.30319.42000                                                                                                      
WSManStackVersion              3.0                                                                                                                  
PSRemotingProtocolVersion      2.3                                                                                                                  
SerializationVersion           1.1.0.1  

Debug Output

DEBUG: 1:58:39 PM - ImportAzureDataLakeStoreItem begin processing with ParameterSet 'NoDiagnosticLogging'.
DEBUG: 1:58:39 PM - using account id '4c77b514-8e34-4e1a-89eb-feea0952c51b'...
FailedTransfer: /cooked/cosmosAuditLog.csv Error in creating file /cooked/cosmosAuditLog.csv.
Operation: CREATE failed with   Unknown Error: Specified value has invalid CRLF characters.
Parameter name: value Source: System StackTrace:    at System.Net.WebHeaderCollection.CheckBadChars(String name, Boolean isHeaderValue)
   at System.Net.WebHeaderCollection.Set(String name, String value)
   at Microsoft.Azure.DataLake.Store.WebTransport.AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client, RequestOptions req, String token,
 String opMethod, IDictionary`2 customHeaders, Int32 postRequestLength)
   at Microsoft.Azure.DataLake.Store.WebTransport.MakeSingleCall(String opCode, String path, ByteBuffer requestData, ByteBuffer responseData, QueryPa
rams qp, AdlsClient client, RequestOptions req, OperationResponse resp, IDictionary`2 customHeaders).
Last encountered exception thrown after 5 tries. [The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value]
[ServerRequestId:]
DEBUG: [Common.Authentication]: Authenticating using Account: '4c77b514-8e34-4e1a-89eb-feea0952c51b', environment: 'AzureCloud', tenant: '72f988bf-86
f1-41af-91ab-2d7cd011db47'
DEBUG: 2018-12-14 13:58:40.1321|DEBUG|adls.dotnet.FileTransfer|FileTransfer.Uploader, Src: C:\local\OneDrive\_Slalom\Projects\Microsoft\ADL\Dashboard
\CosmosJobs\cosmosAuditLog.csv, Dest: /cooked/cosmosAuditLog.csv, Threads: 128, TrackingProgress: True, OverwriteIfExist: True
DEBUG: 2018-12-14 13:58:40.1571|DEBUG|adls.dotnet.FileTransfer|FileTransfer.FileProduced, Name: C:\local\OneDrive\_Slalom\Projects\Microsoft\ADL\Dash
board\CosmosJobs\cosmosAuditLog.csv, Dest: /cooked/cosmosAuditLog.csv, Length: 112818156, Chunks: 0
DEBUG: 2018-12-14 13:58:40.2181|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:94da3850-1f7c-4e00-a42d-5b9922899662.0,lat:27,err:The 'User-
Agent' header must be modified using the appropriate property or method.
Parameter name: name,Reqlen:0,Resplen:0,token_ns:4,sReqId:,path:/cooked/cosmosAuditLog.csv,qp:op=CREATE&overwrite=True&leaseid=6c55c439-a490-4b26-921
7-56eff428f62e&filesessionid=6c55c439-a490-4b26-9217-56eff428f62e&CreateParent=True&write=true&syncFlag=DATA&api-version=2018-05-01
DEBUG: 2018-12-14 13:58:41.2209|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:94da3850-1f7c-4e00-a42d-5b9922899662.1,lat:0,err:Specified v
alue has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/cooked/cosmosAuditLog.csv,qp:op=CREATE&overwrite=True&leaseid=6c55c439-a490-4b26-92
17-56eff428f62e&filesessionid=6c55c439-a490-4b26-9217-56eff428f62e&CreateParent=True&write=true&syncFlag=DATA&api-version=2018-05-01
DEBUG: 2018-12-14 13:58:45.2216|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:94da3850-1f7c-4e00-a42d-5b9922899662.2,lat:0,err:Specified v
alue has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/cooked/cosmosAuditLog.csv,qp:op=CREATE&overwrite=True&leaseid=6c55c439-a490-4b26-92
17-56eff428f62e&filesessionid=6c55c439-a490-4b26-9217-56eff428f62e&CreateParent=True&write=true&syncFlag=DATA&api-version=2018-05-01
DEBUG: 2018-12-14 13:59:01.2221|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:94da3850-1f7c-4e00-a42d-5b9922899662.3,lat:0,err:Specified v
alue has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/cooked/cosmosAuditLog.csv,qp:op=CREATE&overwrite=True&leaseid=6c55c439-a490-4b26-92
17-56eff428f62e&filesessionid=6c55c439-a490-4b26-9217-56eff428f62e&CreateParent=True&write=true&syncFlag=DATA&api-version=2018-05-01
DEBUG: 2018-12-14 14:00:05.2233|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:94da3850-1f7c-4e00-a42d-5b9922899662.4,lat:0,err:Specified v
alue has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/cooked/cosmosAuditLog.csv,qp:op=CREATE&overwrite=True&leaseid=6c55c439-a490-4b26-92
17-56eff428f62e&filesessionid=6c55c439-a490-4b26-9217-56eff428f62e&CreateParent=True&write=true&syncFlag=DATA&api-version=2018-05-01
DEBUG: 2018-12-14 14:00:05.2303|DEBUG|adls.dotnet.FileTransfer.Job|FileTransfer.CopyFile, Source: C:\local\OneDrive\_Slalom\Projects\Microsoft\ADL\Da
shboard\CosmosJobs\cosmosAuditLog.csv, Dest: /cooked/cosmosAuditLog.csv, Offset: 0, Length: 112818156, Lat: 85065, JobStatus: Failed, Error: Error in
 creating file /cooked/cosmosAuditLog.csv.
Operation: CREATE failed with   Unknown Error: Specified value has invalid CRLF characters.
Parameter name: value Source: System StackTrace:    at System.Net.WebHeaderCollection.CheckBadChars(String name, Boolean isHeaderValue)
   at System.Net.WebHeaderCollection.Set(String name, String value)
   at Microsoft.Azure.DataLake.Store.WebTransport.AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client, RequestOptions req, String token,
 String opMethod, IDictionary`2 customHeaders, Int32 postRequestLength)
   at Microsoft.Azure.DataLake.Store.WebTransport.MakeSingleCall(String opCode, String path, ByteBuffer requestData, ByteBuffer responseData, QueryPa
rams qp, AdlsClient client, RequestOptions req, OperationResponse resp, IDictionary`2 customHeaders).
Last encountered exception thrown after 5 tries. [The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value]
[ServerRequestId:]
/cooked/cosmosAuditLog.csv
DEBUG: AzureQoSEvent: CommandName - Import-AzDataLakeStoreItem; IsSuccess - True; Duration - 00:01:25.6064958; Exception - ;
DEBUG: Finish sending metric.
DEBUG: 2:00:06 PM - ImportAzureDataLakeStoreItem end processing.
DEBUG: 2:00:06 PM - ImportAzureDataLakeStoreItem end processing.
@markcowl markcowl added Data Lake Store Service Attention This issue is responsible by Azure service team. labels Dec 14, 2018
@markcowl
Copy link
Member

@rahuldutta90 Can you please take a look

@rahuldutta90
Copy link
Contributor

@kenogr This is a problem with Httpwebrequest class used by our SDK. Setting useragent for httpwebrequest is different across netframework and netcore:

NET framework: webReq.UserAgent = client.GetUserAgent();

net standard webReq.Headers["User-Agent"] = client.GetUserAgent()

If you try to later thing in net framework you get the error you are getting above.

When you use this az module which is using the netstandard dll of our SDK. When you use that from windows powershell, its trying to use net standard dll on netframework which gives this error.

Solution:
I tested this in net core powershell. It runs fine.
We are moving from httpwebrequest to httpclient which will probably fix the issue.

So basically if you are using windows powershell use azurerm or else use Az from powershell netcore.

@rahuldutta90
Copy link
Contributor

@kenogr I also hit this, i bypassed this on my local build.

@cormacpayne Looks like this issue: #7462 is hitting in netcore.

It also hit for me. Based on my last investigation and I pointed out that this wont be hit for spi path, because we convert it to uri. However we are hitting this for spi path also, in netcore which I do not understand. Can you please investigate? I think we need the trailing slash fix sooner since adls module is broken in netcore.

@kenogr
Copy link
Author

kenogr commented Dec 17, 2018

Yes, I think you're right based upon the error. For my SPI, I also tried setting the reply URL's just in case -- doesn't work on any variant. Looks like #7462 is root cause here.

@rahuldutta90
Copy link
Contributor

rahuldutta90 commented Dec 18, 2018

@kenogr here is another mitigation:

$testEnv1="ADLSEnv"
   Add-AzEnvironment -Name $testEnv1 `
    -PublishSettingsFileUrl 'http://go.microsoft.com/fwlink/?LinkID=301775' `
    -ServiceEndpoint 'https://management.core.windows.net/' `
    -ManagementPortalUrl 'http://go.microsoft.com/fwlink/?LinkId=254433' `
    -ActiveDirectoryEndpoint 'https://login.windows.net/' `
    -ActiveDirectoryServiceEndpointResourceId 'https://management.core.windows.net/' `
    -ResourceManagerEndpoint 'https://management.azure.com/' `
    -GalleryEndpoint 'https://df.gallery.azure.com/' `
    -GraphEndpoint 'https://graph.windows.net/' `
    -AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix 'azuredataanalytics.net' `
    -AzureDataLakeStoreFileSystemEndpointSuffix 'azuredatalakestore.net' `
    -DataLakeEndpointResourceId 'https://datalake.azure.net/'

Login-AzAccount -ServicePrincipal -Credential $psCred -TenantId $tenantid -Environment $testEnv1
Test-AzDataLakeStoreItem -AccountName rdutta2 -Path "/" -Debug

Make sure you close and restart the powershell session before doing it.

@kenogr
Copy link
Author

kenogr commented Dec 18, 2018

Thanks @rahuldutta90. Mitigation works perfectly. Looking forward to deployment of fix.

@naveenaggarwal
Copy link

@rahuldutta90 I am still getting the same error message after Mitigation.

PS C:> Test-AzDataLakeStoreItem -AccountName testaccountName -Path "/" -Debug
DEBUG: 12:49:35 PM - TestAzureDataLakeStoreItem begin processing with ParameterSet '__AllParameterSets'.

Confirm
Continue with this operation?
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): A
DEBUG: 12:49:40 PM - using account id 'xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx'...
DEBUG: [Common.Authentication]: Authenticating using Account: 'xxxxxxxx-xxxx-xxxxxx-xxxxxxxxx', environment: 'AzureCloud', tenant: 'tenantid'
DEBUG: 2019-01-04 12:49:40.4633|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:59a22fa6-2a99-4391-aa50-12457f872db8.0,lat:40,err:The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,Reqlen:0,Resplen:0,token_ns:3,sReqId:,path:/,qp:op=GETFILESTATUS&tooid=True&api-version=2018-05-01
DEBUG: 2019-01-04 12:49:41.4807|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:59a22fa6-2a99-4391-aa50-12457f872db8.1,lat:0,err:Specified value has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/,qp:op=GETFILESTATUS&tooid=True&api-version=2018-05-01
DEBUG: 2019-01-04 12:49:45.4823|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:59a22fa6-2a99-4391-aa50-12457f872db8.2,lat:0,err:Specified value has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/,qp:op=GETFILESTATUS&tooid=True&api-version=2018-05-01
DEBUG: 2019-01-04 12:50:01.4837|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:59a22fa6-2a99-4391-aa50-12457f872db8.3,lat:0,err:Specified value has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/,qp:op=GETFILESTATUS&tooid=True&api-version=2018-05-01
DEBUG: 2019-01-04 12:51:05.4845|DEBUG|adls.dotnet.WebTransport|HTTPRequest,failed,cReqId:59a22fa6-2a99-4391-aa50-12457f872db8.4,lat:0,err:Specified value has invalid CRLF characters.
Parameter name: value,Reqlen:0,Resplen:0,token_ns:0,sReqId:,path:/,qp:op=GETFILESTATUS&tooid=True&api-version=2018-05-01

Confirm
Error in getting metadata for path /.
Operation: GETFILESTATUS failed with Unknown Error: Specified value has invalid CRLF characters.
Parameter name: value Source: System StackTrace: at System.Net.WebHeaderCollection.CheckBadChars(String name, Boolean isHeaderValue)
at System.Net.WebHeaderCollection.Set(String name, String value)
at Microsoft.Azure.DataLake.Store.WebTransport.AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client, RequestOptions req, String token, String opMethod, IDictionary`2 customHeaders, Int32 postRequestLength)
at Microsoft.Azure.DataLake.Store.WebTransport.d__21.MoveNext().
Last encountered exception thrown after 5 tries. [The 'User-Agent' header must be modified using the appropriate property or method.
Parameter name: name,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value,Specified value has invalid CRLF characters.
Parameter name: value]
[ServerRequestId:]
[Y] Yes [A] Yes to All [H] Halt Command [S] Suspend [?] Help (default is "Y"): S

@rahuldutta90
Copy link
Contributor

@naveenaggarwal where are running this Az module? Are you running this on windows powershell or net core powershell? You cannot run this on windows powershell. It needs to be run on net core powershell.

@T4GJuneYao
Copy link

Hi @rahuldutta90
I just migrate to version of az module from AzureRm, using Test-AzDataLakeStoreItem, and it shows the
result like the this:

Test-AzDataLakeStoreItem : Error in getting metadata for path
/global/system_mirror/test.tsv.
Operation: GETFILESTATUS failed with Unknown Error: The 'User-Agent' header
must be modified using the appropriate property or method.

How to get rid of User-Agent error?

@rahuldutta90
Copy link
Contributor

@T4GJuneYao Can you please read through the above recomendations:

First use net core powershell not windows powershell for az modules

And for a authentication bug you have to use following way to authenticate:

$testEnv1="ADLSEnv"
   Add-AzEnvironment -Name $testEnv1 `
    -PublishSettingsFileUrl 'http://go.microsoft.com/fwlink/?LinkID=301775' `
    -ServiceEndpoint 'https://management.core.windows.net/' `
    -ManagementPortalUrl 'http://go.microsoft.com/fwlink/?LinkId=254433' `
    -ActiveDirectoryEndpoint 'https://login.windows.net/' `
    -ActiveDirectoryServiceEndpointResourceId 'https://management.core.windows.net/' `
    -ResourceManagerEndpoint 'https://management.azure.com/' `
    -GalleryEndpoint 'https://df.gallery.azure.com/' `
    -GraphEndpoint 'https://graph.windows.net/' `
    -AzureDataLakeAnalyticsCatalogAndJobEndpointSuffix 'azuredataanalytics.net' `
    -AzureDataLakeStoreFileSystemEndpointSuffix 'azuredatalakestore.net' `
    -DataLakeEndpointResourceId 'https://datalake.azure.net/'
Login-AzAccount -ServicePrincipal -Credential $psCred -TenantId $tenantid -Environment $testEnv1
Test-AzDataLakeStoreItem -AccountName rdutta2 -Path "/" -Debug

This authentication bug will be fixed in upcoming release.

@T4GJuneYao
Copy link

@rahuldutta90 Thanks for replying me. Howerver after downloading the Powershell core 6.0, I have no idea on how it work with the powershell ISE because I tried to run the ps1 file in the Powershell ISE.

When is the upcoming release available? I am thinking of restoring back to AzureRm module now and waiting for the next release if it is too much time consuming on fixing this bug.

@rahuldutta90
Copy link
Contributor

@T4GJuneYao you can run the powershell (ps1) script file in the powershell net core like:

.\test.ps1

If you have functions in powershell file, you can include it and run the function like (testfunc is a function within test.ps1, dot script includes the script):

. .\test.ps1
testfunc

@rahuldutta90
Copy link
Contributor

The audience issue will be fixed in the last release. Until we migrate our sdk to httpclient, az module will only work in netcore powershell.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Data Lake Store Service Attention This issue is responsible by Azure service team.
Projects
None yet
Development

No branches or pull requests

6 participants