Querying managed Prometheus metrics error: json: cannot unmarshal object into Go struct field HttpBody.data of type string

[veggiemonk]

Maybe it is something I did wrong, I couldn't find an example on how to query Google Managed Prometheus programmatically.

The error occurs because the Data field of HttpBody of type string which cause decoding JSON to fail while the request has succeeded (status 200). See https://github.com/googleapis/google-api-go-client/blob/main/monitoring/v1/monitoring-gen.go#L1518
After checking the header of the response, Content-Type is application/json

Could you recommend a way to query metrics without a front-end ?

Much appreciated

Environment details

• Programming language: Go
• OS: linux/amd64
• Language runtime version: go1.21.5
• Package version: google.golang.org/api v0.154.0

Steps to reproduce

1. Replace "XXX_project_id_XXX" with a project ID where Google Managed Prometheus is enabled.

package main

import (
	"context"
	"fmt"
	"testing"
	"time"

	monv1 "google.golang.org/api/monitoring/v1"
)

func main() {
	fmt.Println("Hello, 世界")
}

func TestPromMetrics(t *testing.T) {
	ctx := context.Background()
	svc, err := monv1.NewService(ctx)
	if err != nil {
		t.Fatal(err)
	}
	s := monv1.NewProjectsLocationPrometheusApiV1Service(svc)
	q := &monv1.QueryRangeRequest{
		Query:   "agones_gameservers_total",
		Start:   time.Now().UTC().Add(-5 * time.Minute).Format(time.RFC3339),
		End:     time.Now().UTC().Format(time.RFC3339),
		Step:    "60s",
		Timeout: "60s",
	}
	resp, err := s.QueryRange("projects/XXX_project_id_XXX", "global", q).Do()
	if err != nil {
		t.Fatal(err, resp)
	}
	t.Logf("%#v", resp)
}

2. Set up Application Default Credentials
3. Run go test -run TestPromMetrics

The full error message:

--- FAIL: TestPromMetrics (0.23s)
    metrics_test.go:39: json: cannot unmarshal object into Go struct field HttpBody.data of type string <nil>
FAIL
exit status 1

[veggiemonk]

By forming the query manually it works:

package main

import (
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
	"net/url"
	"path"
	"testing"
	"time"

	monv1 "google.golang.org/api/monitoring/v1"
	"google.golang.org/api/option"
	apihttp "google.golang.org/api/transport/http"
)

func main() {
	fmt.Println("Hello, 世界")
}

const projectID = "XXX"

func TestPromMetrics(t *testing.T) {
	opts := []option.ClientOption{
		option.WithScopes("https://www.googleapis.com/auth/monitoring.read"),
	}
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	targetx := fmt.Sprintf("https://monitoring.googleapis.com/v1/projects/%s/location/global/prometheus", projectID)
	transport, err := apihttp.NewTransport(ctx, http.DefaultTransport, opts...)
	if err != nil {
		t.Fatal("create HTTP transport", "err", err)
	}
	client := http.Client{Transport: transport}
	u, err := url.Parse(targetx)
	if err != nil {
		t.Fatal("parse target URL", "err", err)
	}
	u.Path = path.Join(u.Path, "/api/v1/query_range")
	q := &monv1.QueryRangeRequest{
		Query:   "up",
		Start:   time.Now().UTC().Add(-5 * time.Minute).Format(time.RFC3339),
		End:     time.Now().UTC().Format(time.RFC3339),
		Step:    "60s",
		Timeout: "60s",
		// ForceSendFields: nil,
		// NullFields:      nil,
	}
	body, err := json.Marshal(q)
	if err != nil {
		t.Fatal("marshal query", "err", err)
	}

	newReq, err := http.NewRequestWithContext(ctx, http.MethodPost, u.String(), bytes.NewReader(body))
	resp, err := client.Do(newReq)
	if err != nil {
		t.Fatal("do request", "err", err)
	}
	defer resp.Body.Close()
	res, err := io.ReadAll(resp.Body)
	if err != nil {
		t.Fatal("read response", "err", err)
	}
	t.Log(string(res))
}

[quartzmo]

@veggiemonk,

Thank you for reporting this issue.

I couldn't find an example on how to query Google Managed Prometheus programmatically.

I looked through the samples in https://github.com/GoogleCloudPlatform/golang-samples/tree/main/monitoring but couldn't find anything for queries.

[quartzmo]

@veggiemonk,

Thank you for providing the manual workaround.

Question: Can https://pkg.go.dev/cloud.google.com/go/monitoring be used for this use case?

Unfortunately, due to the maintenance mode status of this project, I won't be able to prioritize reproducing this issue right now. If you can still reproduce it, can you by any chance provide a deeper analysis of where the json error is occurring in the client library and what the fix might be?

[veggiemonk]

@quartzmo

Question: Can https://pkg.go.dev/cloud.google.com/go/monitoring be used for this use case?

That package is for dealing with metrics stored in Cloud Operations as far as I understood. The metrics I used are store in Google Managed Prometheus which, apparently, is a different APIs.

can you by any chance provide a deeper analysis of where the json error is occurring in the client library and what the fix might be?

The error occurs because the Data field of HttpBody of type string which cause decoding JSON to fail while the request has succeeded (status 200). See https://github.com/googleapis/google-api-go-client/blob/main/monitoring/v1/monitoring-gen.go#L1518
The field should be of type []byte as the error states:
json: cannot unmarshal object into Go struct field HttpBody.data of type string

That's the fix as far as I can tell.

[quartzmo]

@veggiemonk Thank you for the link to the incorrect type. This is helpful, I will continue to investigate.

[codyoss]

Closing as this issue has had not recent traffic, feel free to reopen if you are still having this problem.

[hanneshayashi]

Just ran into this issue today so this is still relevant. I would love to use the new /apiv3/v2 for this but it doesn't expose anything that can be used to actually run PromQL. I don't even use managed Prometheus but I still need to query the monitoring API with a PromQL query for my use case.

[quartzmo]

@hanneshayashi Are you also using QueryRange? Can you try running a PromQL query with Query and QueryExemplars if it's not too much trouble, to see if the same decoding problem ("HttpBody.data of type string") occurs?

[hanneshayashi]

Yes, I am using QueryRange and I also ended up using a similar workaround to the above. I will try the other methods as well but I won't be able to before Monday.

[hanneshayashi]

@quartzmo I just tried Query and QueryExemplars and they both returned the same error:

2024/08/19 08:34:53 json: cannot unmarshal array into Go struct field HttpBody.data of type string

[quartzmo]

@hanneshayashi Thanks for verifying that. I doubt there will be any quick fix for this issue, but I will continue to investigate.

[hanneshayashi]

@quartzmo I thought @veggiemonk already identified the cause in his original comment with Data being of type string instead of []byte at

google-api-go-client/monitoring/v1/monitoring-gen.go

Line 1518 in 08bc733

Data string `json:"data,omitempty"`

(this links to v0.154.0, the current position should be

google-api-go-client/monitoring/v1/monitoring-gen.go

Line 1414 in d994b67

Data string `json:"data,omitempty"`

for v0.193.0)

[quartzmo]

monitoring-gen.go is generated from the API definition in google-api-go-client/monitoring/v1/monitoring-api.json, which in turn is derived from the discovery document published by the product team. There are other generated files for other APIs which share the same Data type mapping. A fix can be made for the generator if we can establish that the type mapping is incorrect for all usages (or a conditional fix for some usages), and that's what needs further investigation.

[codyoss]

Have not looked into the full details, but this seems directly related to #346

[codyoss]

Looking into this a little more a quick fix for this would be to change HttpBody.Data to any instead of string. We can't use []byte because the returned data is a json object that does not map to bytes. i.e. json: cannot unmarshal object into Go struct field HttpBody.data of type []uint8

[quartzmo]

The following RPC methods all share the HttpBody response type. Will it be correct to apply the fix of changing HttpBody.Data from string to any for all of them?

• Labels
• Query
• QueryExemplars
• QueryRange
• Series
• Values
• MetadataList

[codyoss]

This seems correct to me. proceeding with PR