cmd/go: fails when main module is in a 9P network share on Windows Subsystem for Linux

[JulienTant]

What version of Go are you using (go version)?

$ go version
go version go1.14 windows/amd64

Does this issue reproduce with the latest release?

It is the last release

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=on
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\a\AppData\Local\go-build
set GOENV=C:\Users\a\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS= -mod=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\a\go
set GOPRIVATE=
set GOPROXY=direct
set GOROOT=C:\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=Z:\home\a\go\graph\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\a\AppData\Local\Temp\go-build115528494=/tmp/go-build -gno-record-gcc-switches

What did you do?

I'm using Windows Subsystem for Linux 2 and I access my files using a network driver. I tried to use go mod init in a new project and got this error :

a@Hackintosh-Windows MINGW64 /z/home/a/go/graph
$ go mod init graph
go: creating new go.mod: module graph
go: updating go.mod: Lock Z:\home\a\go\graph\go.mod: Fonction incorrecte.

(Fonction incorrecte means incorrect/invalid function)

Then i tried to create it by end with the content :

module graph

go 1.14

Running go list -m (or any command requiring this file actually) gives me this error : go: RLock Z:\home\a\go\graph\go.mod: Fonction incorrecte.

What did you expect to see?

my go.mod file created.

What did you see instead?

An error about Lock/RLock.

More information:

After reinstalling go 1.13.8, i had everything working back.

[ALTree]

Thanks for the report.

The Windows Subsystem for Linux is not a supported platform. The fact that it accidentally worked on Go1.13 does not necessarily mean that it will keep working going forward. We don't have a builder (since it's not a supported platform), so breakage may happen.

I'm closing this issue, but please feel free to comment if you can reproduce this on plain Windows or on Linux; and we'll re-open.

[artsafin]

Can the support of WSL2 be considered or included into roadmap?

WSL2 is becoming a popular platform as it soothes many pains of development under Windows.

[ianlancetaylor]

@artsafin See https://golang.org/wiki/PortingPolicy .

That said, I thought the goal of WSL was to permit running GNU/Linux programs on Windows. It ought to work to run GOOS=linux go build to get a GNU/Linux binary and then run that binary under WSL.

[Wintereise]

That said, I thought the goal of WSL was to permit running GNU/Linux programs on Windows. It ought to work to run GOOS=linux go build to get a GNU/Linux binary and then run that binary under WSL.

The problem here is when your code lives within the WSL2 VM, but you're working on it via the \wsl$ network share in your IDE / toolchain. A lot of us have adopted this since it allows us to code in Windows, but do all testing / verification in Linux.

The go binary fails to lock the needed .mod or .sum files when called with the wsl$ network path and subsequently fails. I suspect this will probably happen with all network shares in Windows.

[JulienTant]

It's just that it's not happening with go 1.13, it would be nice to be able to disable the rlock feature

[ld100]

Another one here with the same problem. This issue makes working in Goland a chore... Each time I add a library to the project I have to copy the project to the Windows drive and download modules there...

The issue is not about WSL2 itself, but about Go being able to work on Windows with files on the share via P9 protocol.

[artsafin]

It's possible to workaround the issue by launching IDE inside WSL connecting it to X server installed on Windows host.

However native support of working on network shares would be much appreciated.

[jaytaph]

Many more people are in need of this. Having an option to disable the file locking would make it work

[gopherbot]

Change https://golang.org/cl/275392 mentions this issue: cmd/go/internal/lockedfile: ignore errors if locks are not supported

[bcmills]

We don't have a builder (since it's not a supported platform), so breakage may happen.

As it just so happens, we now have a WSL builder thanks to @mengzhuo (#17365 (comment)), so I think we can reopen this issue.

[bcmills]

(That said, the cmd/go tests are currently passing on the WSL builder because the tests are not using a network share for the working directory.)

[bcmills]

@ld100

The issue is not about WSL2 itself, but about Go being able to work on Windows with files on the share via P9 protocol.

Note that the 9P protocol does support file-locking — cmd/go's internal/lockedfile package has a Plan 9 implementation and it seems to work reliably. (The Go project's Plan 9 builders are failing frequently at the moment, but the cause of the failures does not appear to be related to file-locking.)

So it seems that the root problem here is that the Windows filesystem APIs are incomplete for the 9P filesystem, for which @kamijin-fanta has helpfully filed microsoft/WSL#5762.

[bcmills]

Having an option to disable the file locking would make it work

The go command locks files in order to prevent data corruption in both the Go module cache (#26794) and the main module's go.mod and go.sum files.

We had originally only used file locks in the module cache, instead preferring atomic renames for the go.mod and go.sum updates. However, the rename approach resulted in spurious changes to file permissions (#34634) and spurious errors on Windows due to certain idiosyncrasies of the Windows file APIs (#32188, #36568). So in Go 1.14 we switched to using file locks for the go.mod and go.sum updates as well.

[jaytaph]

(That said, the cmd/go tests are currently passing on the WSL builder because the tests are not using a network share for the working directory.)

Is there a way to get involved in setting this up? I do have the time for it, but I don't know how to contribute to this.

[bcmills]

Honestly, this seems like a niche enough use-case that I would be satisfied with a test that only fails locally. (This particular part of cmd/go seems pretty stable and is unlikely to regress.)

That said, since this is an upstream bug (an implementation deficiency in the WSL2 filesystem) and there is what seems like a straightforward workaround in the interim (use a working directory and GOMODCACHE outside of the 9P share), I'm not sure that it's worth adding a lot of complexity to cmd/go itself to deal with this case.

[jaytaph]

I also do think that adding more complexity than needed is not a good thing. However, PR #42994 seems a good way to deal with this issue (and others) when locking is not available (for some reason). Once (IF!) wsl2 implements plan9 locking correctly, no need for any changes would be needed, and we would be safe for other types of file-systems that will be thrown in the wsl (or other system) pool that we do not know of.

[bcmills]

@jaytaph

Once (IF!) wsl2 implements plan9 locking correctly, no need for any changes would be needed,

We already should not require any changes once the Windows 9P implementation supports LockFileEx correctly.

and we would be safe for other types of file-systems that will be thrown in the wsl (or other system) pool that we do not know of.

Given that ignoring file-locking errors can lead to silent corruption of the affected files, “safe” is definitely not the word I would use. 😒

[kamijin-fanta]

Please tell me, where can I find the test code for plan9? I thought it wasn't working properly in plan9.

[bcmills]

@kamijin-fanta, the tests in lockedfile_test.go and transform_test.go should run on Plan 9 the same as on every other platform.

There is an additional integration test in mod_concurrent.txt that I believe also runs on Plan 9.

[bcmills]

(The last Plan 9 file-locking bug I'm aware of was #38719, which we fixed by using lockedfile.Read in even more places. 😅)

[kamijin-fanta]

Thanks to you, I understand that Plan 9 implements a lock that doesn't use filelock.Lock here.
https://github.com/golang/go/blob/go1.15.6/src/cmd/go/internal/lockedfile/lockedfile_plan9.go

If the file system does not support locking, can it be implemented using lock files? Create a lock file in the TMP directory, etc. to control the lock. I'm thinking of implementing apt like /var/lib/apt/lists/lock. But this is an ad hoc fix for only WSL.

[pawelqs]

Hi!
I just came here due to the same problem as described on StackOverflow. @bcmills, you have mentioned that on Unix systems you are using flock. I see flock installed on my system, but the problem persists.

$ go mod init example/hello
go: creating new go.mod: module example/hello
go: updating go.mod: Lock /home/pkus/temp/hello/go.mod: function not implemented

How can I resolve it?
Thanks!

[rminnich]

I am now having this problem too: using github.com/u-root/cpu, i.e. a 9P2000 mount on Linux, and the gvisor 9P package, which does not support Tlock.

While the use of flock may be fine on a local file system, depending on file locks to work on any network file system has always been a hazardous undertaking; even the most simple search for rpc.lockd bugs is revealing.

I think Go is being a bit optimistic here about how effective an flock is.

I'd suggest that the build command continue if the error is "not implemented." There is little else that can be done in that case, and it is seems sensible to me to, at most, print a warning, but not actually exit with a fatal error. It may not be possible for people to swap out their kernel or their network file system protocol. I'm happy to submit a PR if that is acceptable.

[rminnich]

Ah somebody beat me to it, I'll take a look at that PR. Again, the situation in my case is a small IoT running Go, over 9P2000. The 9P2000 Go package I'm using does not have Tlock support atm, and it is literally a single user situation, in the IoT case.