Skip to content

Commit

Permalink
testscript: remove our code coverage mechanism thanks to Go 1.20
Browse files Browse the repository at this point in the history
Our code was a fairly hacky version of what Go 1.20 does for us,
since we had to externally reach into the testing internals
to do the right thing before and after each program execution.

With Go 1.20, all we actually need to do is ensure that the GOCOVERDIR
environment variable is forwarded properly. With that, test binaries
will know how to produce multiple coverage profiles, and "go test" will
know how to collect and merge them.

We could keep our hacky workaround for the sake of "deep" coverage
information for Go 1.19, but that doesn't seem worthwhile.
The old mechanism caused test flakes like rogpeppe#130,
is incompatible with Go 1.20, and is overall worse than what Go 1.20 can do.
If a user wants code coverage with the latest testscript version, they
can use Go 1.20. If they are stuck on Go 1.19 and need code coverage,
I imagine they can also stick to a slightly older testscript version.

On a main package, the old testscript with Go 1.19 reports:

	PASS
	coverage: 8.0% of statements
	total coverage: 90.1% of statements
	ok	mvdan.cc/sh/v3/cmd/shfmt	0.063s

The new testscript with Go 1.20 reports:

	PASS
		mvdan.cc/sh/v3/cmd/shfmt	coverage: 90.1% of statements
	ok	mvdan.cc/sh/v3/cmd/shfmt	0.047s

Fixes rogpeppe#130, as the RemoveAll call is no longer present.
Fixes rogpeppe#161, as the API is now deprecated.
Fixes rogpeppe#199, as "go test -coverprofile" now works on Go 1.20.
  • Loading branch information
mvdan committed Feb 8, 2023
1 parent e3815af commit 26086ed
Show file tree
Hide file tree
Showing 4 changed files with 9 additions and 410 deletions.
15 changes: 6 additions & 9 deletions gotooltest/testdata/cover.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
unquote scripts/exec.txt

[darwin] skip 'Pending a fix for github.com/rogpeppe/go-internal/issues/130'

# The module uses testscript itself.
# Use the checked out module, based on where the test binary ran.
go mod edit -replace=github.com/rogpeppe/go-internal=${GOINTERNAL_MODULE}
Expand All @@ -10,18 +8,17 @@ go mod tidy
# First, a 'go test' run without coverage.
go test -vet=off
stdout 'PASS'
! stdout 'total coverage'
! stdout 'coverage'

# Then, a 'go test' run with -coverprofile.
# Assuming testscript works well, this results in the basic coverage being 0%,
# since the test binary does not directly run any non-test code.
# The total coverage after merging profiles should end up being 100%,
# as long as all three sub-profiles are accounted for.
# The total coverage after merging profiles should end up being 100%.
# Marking all printlns as covered requires all edge cases to work well.
# Go 1.20 learned to produce and merge multiple coverage profiles,
# so versions before then report a shallow 0% coverage.
go test -vet=off -coverprofile=cover.out -v
stdout 'PASS'
stdout '^coverage: 0\.0%'
stdout '^total coverage: 100\.0%'
[go1.20] stdout 'coverage: 100\.0%'
[!go1.20] stdout 'coverage: 0\.0%'
! stdout 'malformed coverage' # written by "go test" if cover.out is invalid
exists cover.out

Expand Down
280 changes: 0 additions & 280 deletions testscript/cover.go

This file was deleted.

Loading

0 comments on commit 26086ed

Please sign in to comment.