Vault CLI always hangs for a few seconds on Windows

[rageshkrishna]

Describe the bug
Running any vault command in Windows is delayed almost 10 seconds before anything happens. This is true even if run vault --version

To Reproduce
Steps to reproduce the behavior:

1. Run vault --version
2. Wait for ~10 seconds
3. See the output

Expected behavior
There shouldn't be a 10 second delay.

Environment:

• Vault Server Version (retrieve with vault status): Irrelevant; just printing the version triggers the delay. No VAULT_ADDR or any other VAULT envs were set.
• Vault CLI Version (retrieve with vault version): Vault v1.8.2 (aca76f6)
• Client Operating System/Architecture: Client is on Windows 11 / x64

Vault server configuration file(s): Irrelevant

[aphorise]

@rageshkrishna this is likely something specific to your machine - more so if you've not made comparisons relative to some other systems running the same version. What's more the likelihood of it being specific to your system is even more so if you're experiencing the same sort of slowness with all other versions of vault.

For example on my system using PowerShell Prompt (you also didn't mention what you used):

PS C:\> (Measure-Command { vault --version | Out-Default }).ToString()
  # Vault v1.8.2 (aca76f63357041a43b49f3e8c11d67358496959f)
  # 00:00:00.6710139

You may want to give a few systems as well as versions a try and consider closing.

PS - In the future mention the type of system / version of Vault you're using (32bit / 64) as well as some hardware, specific OS (exact versions) and other details as relevant.

[rageshkrishna]

Thanks for the pointers, @aphorise. Here's some more information if it helps:

1. Client 1: Windows 11 (Version 21H2 Build 22000.184), running Vault CLI 64-bit 1.8.2 (aca76f) in cmd inside Windows Sandbox to rule out any other tooling conflicts
2. Client 2: Windows 10 (Version 21H1 Build 19043.1165), running Vault CLI 64-bit 1.8.1 (4b0264) in Powershell 7.1.4

The delay on both of these machines is roughly the same:

(Measure-Command { .\vault --version | Out-Default }).ToString()
Vault v1.8.2 (aca76f63357041a43b49f3e8c11d67358496959f)
00:00:07.8578195

Now, the interesting bit is that on the same machines there is no delay when running in WSL:

$ time vault --version
Vault v1.8.2 (aca76f63357041a43b49f3e8c11d67358496959f)

real    0m0.499s
user    0m0.115s
sys     0m0.102s

$ uname -a
Linux Adenium 5.10.43.3-microsoft-standard-WSL2 #1 SMP Wed Jun 16 23:47:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

I'm wondering if it could be something about my network. I'm running Tailscale, but disconnecting didn't make any difference. I'm also not sure why vault --version would need to go over the network anyway. Any other pointers? Something I can do to debug what in the CLI (if anything) that it's getting stuck on?

[AndrewSav]

In golang the init functions are run before the program run. Each dependency of a project and dependency of dependency and so on can have those init functions. If one runs away we have a slow down. In this particular case it is caused by the aerospike go client combined with a golang bug in the time.sleep function. The net result that on certain machines microseconds turn into milliseconds, and the sleep delay in a loop like in aerospike becomes noticeable.

The good news is that aerospike fixed this issue already, and there is a PR in flight to update Vault to the latest version of aerospike. The issue will disappear then.

For right now, if you do not mind compiling Vault manually (and you do not use aerospike) just comment out the sleep line in the aerospike source code and re-compile, the delay goes away.

If you set an environment variable like this GODEBUG=inittrace=1 before running vault (e.g. in PowerShell $env:GODEBUG="inittrace=1") you can see the timing of inits yourself and can confirm that it's aerospike that is causing it for you:

...
init github.com/hashicorp/vault/helper/builtinplugins @496 ms, 0 ms clock, 5032 bytes, 18 allocs
init github.com/hashicorp/vault/builtin/logical/database @496 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aerospike/aerospike-client-go/pkg/bcrypt @497 ms, 0 ms clock, 384 bytes, 3 allocs
init github.com/aerospike/aerospike-client-go/types @497 ms, 0 ms clock, 528 bytes, 22 allocs
init github.com/aerospike/aerospike-client-go/utils/buffer @498 ms, 0 ms clock, 0 bytes, 0 allocs
init github.com/aerospike/aerospike-client-go/logger @499 ms, 0 ms clock, 128 bytes, 2 allocs
init github.com/aerospike/aerospike-client-go/types/rand @499 ms, 7874 ms clock, 16464 bytes, 514 allocs
init github.com/yuin/gopher-lua/parse @8385 ms, 0 ms clock, 944 bytes, 2 allocs
init github.com/yuin/gopher-lua @8389 ms, 0 ms clock, 12912 bytes, 165 allocs
init github.com/aerospike/aerospike-client-go/internal/lua @8390 ms, 0 ms clock, 1120 bytes, 3 allocs
init github.com/aerospike/aerospike-client-go @8391 ms, 0 ms clock, 272 bytes, 8 allocs
init github.com/aliyun/aliyun-oss-go-sdk/oss @8392 ms, 0 ms clock, 19144 bytes, 3 allocs
init github.com/jackc/pgx/pgtype @8393 ms, 0 ms clock, 15368 bytes, 134 allocs
init github.com/jackc/pgx @8394 ms, 0 ms clock, 45504 bytes, 153 allocs
init github.com/hashicorp/vault/physical/cockroachdb @8395 ms, 0.99 ms clock, 12232 bytes, 11 allocs
...

[rageshkrishna]

Thanks @AndrewSav. I can confirm that this is indeed the issue for me:

init github.com/aerospike/aerospike-client-go/logger @407 ms, 0 ms clock, 128 bytes, 2 allocs
init github.com/aerospike/aerospike-client-go/types/rand @407 ms, 7904 ms clock, 16464 bytes, 514 allocs
init github.com/yuin/gopher-lua/parse @8312 ms, 0 ms clock, 944 bytes, 2 allocs

I'll keep this issue open until I can verify the fix with the 1.9 release.

[rageshkrishna]

@AndrewSav Would you happen to have an ETA for the 1.9 release? In the mean time, as a workaround, I suppose I could just downgrade the CLI on my Windows machines to an older release that isn't affected by this issue.

[AndrewSav]

@rageshkrishna no.

[DanielBoulerice]

I have the same issue with Vault 1.9 on Windows 10. It hangs for around 7.2s everytime I send any command.

[DanielBoulerice]

I had to revert to version 1.7 (here) to make it work faster.

[thomaslevesque]

Same issue here

[nvx]

Just updated to Vault 1.9.1 on Windows which seems to have fixed the issue! (Guessing #12165 did the trick)

[DanielBoulerice]

I confirm: issue has been zapped in 1.9.1.

[heatherezell]

Huzzah! Thank you so much @nvx and @DanielBoulerice for reporting back!