Problem using Azure Storage Blob backend #1901

tropnikovvl · 2022-05-16T20:55:37Z

tropnikovvl
May 16, 2022

Describe the bug

Following the configuration I want to set up the backend to Azure Blob.
https://grafana.com/docs/mimir/latest/operators-guide/configuring/reference-configuration-parameters/#blocks_storage
Entering all the parameters I get an error:

level=warn ts=2022-05-16T20:46:47.678358774Z caller=ingester.go:1795 msg="shipper failed to synchronize TSDB blocks with the storage" user=anonymous uploaded=0 err="read metadata for block /data/tsdb/ anonymous/01G379M4GEJBSVGK6ANK4PZXVV: open /data/tsdb/anonymous/01G379M4GEJBSVGK6ANK4PZXVV/meta.json: no such file or directory"

level=error ts=2022-05-17T07:23:53.451557457Z caller=compact.go:618 org_id=anonymous msg="Failed to remove temp folder after failed compaction" dir=/data/tsdb/anonymous/01G38E0N4QVWFQYMCP60HGTK00.tmp-  for-creation err="unlinkat /data/tsdb/anonymous/01G38E0N4QVWFQYMCP60HGTK00.tmp-for-creation: directory not empty"

level=warn ts=2022-05-17T10:35:46.410266894Z caller=ingester.go:1884 msg="TSDB blocks compaction for user has failed" user=anonymous err="compact head: persist head block: rename block dir: rename /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP.tmp-for-creation /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP: input/output error" compactReason=regular

To Reproduce

Steps to reproduce the behavior:

deploy Mimir to Kubernetes with this guide
https://github.com/grafana/helm-charts/tree/main/charts/mimir-distributed
change parametes to azure backend
https://github.com/grafana/helm-charts/blob/main/charts/mimir-distributed/values.yaml#L115

    blocks_storage:
      backend: azure
      azure:
        account_name: accountname
        account_key: 'accountkey'
        container_name: 'containername'
        endpoint_suffix: 'blob.core.windows.net'
      tsdb:
        dir: '/data/tsdb'
        block_ranges_period: ["0h10m0s"]
      bucket_store:
        sync_dir: '/data/tsdb-sync'

Get write error

Expected behavior

Should save data to storage

Environment

Infrastructure: Azure Kubernetes Service, 2 Azure Storage Accounts for tests (with HNS and without HNS), configured firewall on the Storages with access from AKS
Deployment tool: helm

Additional Context

Answered by pracucci

May 20, 2022

I moved the issue to a discussion. We can open specific issue if it will turn out to be a Mimir bug.

From you logs I can see the following errors:

read metadata for block /data/tsdb/ anonymous/01G379M4GEJBSVGK6ANK4PZXVV: open /data/tsdb/anonymous/01G379M4GEJBSVGK6ANK4PZXVV/meta.json: no such file or directory

compact head: persist head block: rename block dir: rename /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP.tmp-for-creation /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP: input/output error

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001: f…

View full answer

tropnikovvl · 2022-05-19T17:50:08Z

tropnikovvl
May 19, 2022
Author

today i got new errors in clean environment

level=error ts=2022-05-19T17:49:37.582206943Z caller=shipper.go:307 org_id=anonymous msg="shipping failed" block=01G3EPX6R8KZC0RD9VKB4C63N3 err="hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001: function not implemented"                                        
level=info ts=2022-05-19T17:49:37.593690436Z caller=shipper.go:334 org_id=anonymous msg="upload new block" id=01G3EPXK2A29C65940XCQYFGXD                                                                 
level=error ts=2022-05-19T17:49:37.733135855Z caller=shipper.go:307 org_id=anonymous msg="shipping failed" block=01G3EPXK2A29C65940XCQYFGXD err="hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPXK2A29C65940XCQYFGXD/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPXK2A29C65940XCQYFGXD/chunks/000001: function not implemented"                                        
level=warn ts=2022-05-19T17:49:37.849137987Z caller=ingester.go:1795 msg="shipper failed to synchronize TSDB blocks with the storage" user=anonymous uploaded=0 err="failed to sync 15 blocks"           
level=info ts=2022-05-19T17:49:45.867785595Z caller=compact.go:578 org_id=anonymous msg="write block" mint=1652982428880 maxt=1652982480000 ulid=01G3EQ0VY8Q7S09M6M48QQZWMS duration=10.819013795s       
level=info ts=2022-05-19T17:49:45.910006741Z caller=head.go:861 org_id=anonymous msg="Head GC completed" duration=11.660595ms                                                                            
level=info ts=2022-05-19T17:49:56.763220038Z caller=compact.go:578 org_id=anonymous msg="write block" mint=1652982481395 maxt=1652982540000 ulid=01G3EQ16PNXJ8XT2BT6RG17C05 duration=10.694097893s       
level=info ts=2022-05-19T17:49:56.866390676Z caller=head.go:861 org_id=anonymous msg="Head GC completed" duration=57.637968ms                                                                            
level=warn ts=2022-05-19T17:49:57.233812262Z caller=ingester.go:1884 msg="TSDB blocks compaction for user has failed" user=anonymous err="WAL truncation in Compact: get segment range: segments are not sequential" compactReason=regular

I no longer understand what Mimir needs to work with blobs in Azure. I went through all kinds of SKU, disabled all firewalls and gave all possible write permissions
but it still doesn't work

1 reply

malnor Sep 7, 2022

@tropnikovvl The documentation in mimir is a bit vague and hard to understand when it comes to managed identities. We had this problem as well and figured out that mimir underneath the hood use thanos (from looking at the traces). Their documentation was a bit more clear. Stating that when you specify the msi_resource it will use the managed identity instead of the storage account key.

If msi_resource is used, authentication is done via system-assigned managed identity. The value for Azure should be https://.blob.core.windows.net.

https://github.com/thanos-io/thanos/blob/main/docs/storage.md#azure

pracucci · 2022-05-20T08:51:29Z

pracucci
May 20, 2022
Maintainer

I moved the issue to a discussion. We can open specific issue if it will turn out to be a Mimir bug.

From you logs I can see the following errors:

read metadata for block /data/tsdb/ anonymous/01G379M4GEJBSVGK6ANK4PZXVV: open /data/tsdb/anonymous/01G379M4GEJBSVGK6ANK4PZXVV/meta.json: no such file or directory

compact head: persist head block: rename block dir: rename /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP.tmp-for-creation /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP: input/output error

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001: function not implemented

This issue is not caused by a problem with your Azure blob storage configure, but an issue with the ingesters local disk. Which disk type are you using for ingesters? Which filesystem? Any special setup about it? The "function not implemented" makes me think there's something very wrong going with it.

0 replies

pracucci · 2022-05-20T08:51:32Z

pracucci
May 20, 2022
Maintainer

I moved the issue to a discussion. We can open specific issue if it will turn out to be a Mimir bug.

From you logs I can see the following errors:

read metadata for block /data/tsdb/ anonymous/01G379M4GEJBSVGK6ANK4PZXVV: open /data/tsdb/anonymous/01G379M4GEJBSVGK6ANK4PZXVV/meta.json: no such file or directory

compact head: persist head block: rename block dir: rename /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP.tmp-for-creation /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP: input/output error

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001: function not implemented

This issue is not caused by a problem with your Azure blob storage configure, but an issue with the ingesters local disk. Which disk type are you using for ingesters? Which filesystem? Any special setup about it? The "function not implemented" makes me think there's something very wrong going with it.

0 replies

pracucci · 2022-05-20T08:56:31Z

pracucci
May 20, 2022
Maintainer

I moved the issue to a discussion. We can open specific issue if it will turn out to be a Mimir bug.

From you logs I can see the following errors:

read metadata for block /data/tsdb/ anonymous/01G379M4GEJBSVGK6ANK4PZXVV: open /data/tsdb/anonymous/01G379M4GEJBSVGK6ANK4PZXVV/meta.json: no such file or directory

compact head: persist head block: rename block dir: rename /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP.tmp-for-creation /data/tsdb/anonymous/01G38S0Z9DAM6KWD7QQ72RHEAP: input/output error

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3EPX6R8KZC0RD9VKB4C63N3/chunks/000001: function not implemented

This issue is not caused by a problem with your Azure blob storage configure, but an issue with the ingesters local disk. Which disk type are you using for ingesters? Which filesystem? Any special setup about it? The "function not implemented" makes me think there's something very wrong going with it.

6 replies

pracucci May 20, 2022
Maintainer

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3GBDFM8RBWHVWRA511BX9EM/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3GBDFM8RBWHVWRA511BX9EM/chunks/000001: function not implemented

This error still looks like an issue with the local disk.

tropnikovvl May 20, 2022
Author

hard link block: hard link file chunks/000001: link /data/tsdb/anonymous/01G3GBDFM8RBWHVWRA511BX9EM/chunks/000001 /data/tsdb/anonymous/thanos/upload/01G3GBDFM8RBWHVWRA511BX9EM/chunks/000001: function not implemented

This error still looks like an issue with the local disk.

i use this driver to integrate with azure storage
https://github.com/kubernetes-sigs/blob-csi-driver
other apps work great

      mount_options:
        - -o allow_other
        - --file-cache-timeout-in-seconds=120
        - --use-attr-cache=true
        - --cancel-list-on-mount-seconds=60
        - -o attr_timeout=120
        - -o entry_timeout=120
        - -o negative_timeout=120
        - --cache-size-mb=1000
        - --use-adls=true

pracucci May 20, 2022
Maintainer

Mimir needs to create a hard link on disk. Is it supported by your volume? Could you try to manually create a test one on your disk, please?

tropnikovvl May 20, 2022
Author

thank you, I will try

tropnikovvl May 22, 2022
Author

Hey
As it turned out, the only entity in Azure that supports hard links is the disk.
Everything else just won't work.
Thank you very much for your help.

ksingh-scogo · 2023-12-22T20:01:43Z

ksingh-scogo
Dec 22, 2023

For community members those who are facing problems configuring Azure Blob with Mimir , use the below config, it worked

(edit by @dimitarvdimitrov: this works as values file for the mimir-distributed helm chart)

mimir:
  structuredConfig:
    common:
      storage:
        backend: azure
        azure:
          account_key: "<YOUR KEY HERE>"
          account_name: <ACKT_NAME>
          endpoint_suffix: "blob.core.windows.net"
    blocks_storage:
      backend: azure ## This line is important here, official docs are missing this
      azure:
        container_name: mimir-dev-blocks
    alertmanager_storage:
      backend: azure ## This line is important here, official docs are missing this
      azure:
        container_name: mimir-dev-alertmanager
    ruler_storage:
      backend: azure ## This line is important here, official docs are missing this
      azure:
        container_name: mimir-dev-ruler

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problem using Azure Storage Blob backend #1901

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Problem using Azure Storage Blob backend #1901

tropnikovvl May 16, 2022

Describe the bug

To Reproduce

Expected behavior

Environment

Additional Context

Replies: 5 comments · 7 replies

tropnikovvl May 19, 2022 Author

malnor Sep 7, 2022

pracucci May 20, 2022 Maintainer

pracucci May 20, 2022 Maintainer

pracucci May 20, 2022 Maintainer

pracucci May 20, 2022 Maintainer

tropnikovvl May 20, 2022 Author

pracucci May 20, 2022 Maintainer

tropnikovvl May 20, 2022 Author

tropnikovvl May 22, 2022 Author

ksingh-scogo Dec 22, 2023

tropnikovvl
May 16, 2022

Replies: 5 comments 7 replies

tropnikovvl
May 19, 2022
Author

pracucci
May 20, 2022
Maintainer

pracucci
May 20, 2022
Maintainer

pracucci
May 20, 2022
Maintainer

pracucci May 20, 2022
Maintainer

tropnikovvl May 20, 2022
Author

pracucci May 20, 2022
Maintainer

tropnikovvl May 20, 2022
Author

tropnikovvl May 22, 2022
Author

ksingh-scogo
Dec 22, 2023