go's module cache not working with redis on my Mac.

[henrjk]

I notice the go redis build test is failing when run locally on my Mac:

go test -run TestTaskODSBuildGo/build_go_app_with_redis_sidecar$ github.com/opendevstack/pipeline/test/tasks -v -timeout 60m
...
=== RUN   TestTaskODSBuildGo
    check.go:42: Trying to reach the required services...
    check.go:49: Nexus reached successfully.
    check.go:49: SonarQube reached successfully.
2022/03/15 11:55:22 Create namespace hygfctml to deploy to
2022/03/15 11:55:22 Create persistent volume hygfctml
2022/03/15 11:55:22 Create persistent volume claim task-pv-claim
...
        serviceaccount/pipeline patched
=== RUN   TestTaskODSBuildGo/build_go_app_with_redis_sidecar
...
2022/03/15 11:55:31 Type: Normal, Message: Started container sidecar-sidecar
2022/03/15 11:55:32 ---------------------- Logs from step-build-go-binary -------------------------
+ set -e
+ echo -n ''
+ cache_build_key=go-linux-amd64
+ copy-build-if-cached --cache-build=false --cache-build-key=go-linux-amd64 --cache-location-used-path=/tekton/results/build-reused-from-location --working-dir=. --output-dir=docker --debug=false
Build skipping is not enabled. Continuing with a regular build (CACHE_BUILD==false)
+ build-go --working-dir=. --enable-cgo=false --go-os=linux --go-arch=amd64 --pre-test-script= --output-dir=docker --debug=false
Working on Go module in /workspace/source ...
go version go1.16.12 linux/amd64
INFO: Using gomodule cache on repo pvc
GOMODCACHE=/workspace/source/.ods-cache/deps/gomod
Filesystem      Size  Used Avail Use% Mounted on
grpcfuse        466G  393G   46G  90% /workspace/source
Checking format ...
Linting ...
golangci-lint has version 1.41.1 built from a2074809 on 2021-06-19T16:01:50Z
OK
Testing ...
Filesystem      Size  Used Avail Use% Mounted on
grpcfuse        466G  393G   46G  90% /workspace/source
=== RUN   TestExample
key value
key2 does not exist
--- PASS: TestExample (0.00s)
PASS
coverage: 55.6% of statements
ok  	go-redis	0.015s	coverage: 55.6% of statements
Building ...
Checking for sonar-project.properties ...
+ '[' false == true ']'
2022/03/15 11:57:32 Waiting for container step-scan-with-sonar from pod ods-build-go-with-sidecar-taskrun-fgpoenlb-pod-mqxg7 to be ready...
    helper.go:123: Status: True
    helper.go:124: Reason: Succeeded
    helper.go:125: Message: All Steps have completed executing
    run.go:167: unlinkat /Users/henrichkraemer/code/bix/ods/ods-pipeline/test/testdata/workspaces/workspace-692217962/.ods-cache/deps/gomod/github.com/cespare/xxhash/v2@v2.1.1/xxhash_amd64.go: permission denied
=== CONT  TestTaskODSBuildGo
    common_test.go:169: Deleting namespace hygfctml
    common_test.go:169: Deleting persistent volume with name hygfctml
--- FAIL: TestTaskODSBuildGo (132.17s)
    --- FAIL: TestTaskODSBuildGo/build_go_app_with_redis_sidecar (128.00s)
FAIL
FAIL	github.com/opendevstack/pipeline/test/tasks	132.206s
FAIL

While this in the context of working on PR #462 I believe it is related to the addition of the global dependency cache. It appears to not have this issue in the GitHub build.

Another issue I noticed is that after using go task tests locally make clear-tmp-workspaces it fails with permission issues as the go module cache uses read only permissions by default.

[michaelsauter]

Am I getting this right that the test failure is related to unsuccessful cleanup? So this would be the same issue as with make clear-tmp-workspaces, no?

[henrjk]

I believe not. This happens after doing a fixed local clear-tmp-workspaces also. But perhaps there is an issue in case the tests aim to remove stuff and fails. But then it should fail for other go tests as well or?

[michaelsauter]

But then it should fail for other go tests as well or?

Others do not use any 3rd party dependencies, so maybe do not trigger the problem.

On that note, @kuebler @henninggross I just realise that your changes in #404 removed the Redis test, which now means we do not have a test exercising integration of external Go modules. I think we should expand the Go sample app to download a simple Go module to bring back that confidence. We could potentially use https://pkg.go.dev/rsc.io/quote, it is used in various demos etc., and "maintained" by the Go lead so should be reliable.

[michaelsauter]

I can reproduce this locally, and as far as I can tell it's because the global caching prevents all cleanup later due to insufficient permissions. I'm working on a fix.