pd panic which lead to lightning import failed

[Lily2025]

Bug Report

What did you do?

1、lightning import

    - storage: s3://benchmark/location-dumpling
      s3Endpoint: http://minio.pingcap.net:9000
      db: location
      options: --enable-checkpoint false
      config: |-
        [lightning]
        max-error = 9999999999
        [tikv-importer]
        duplicate-resolution = "record"
        incremental-import = true
      crc32Checksum: 2645404411
      adminCheck:
      - table: AreaMeshLake
        indexes:
        - areameshlakeid
      backend: local
      sortedKvDir: /tmp/sorted-kv-dir
    - storage: s3://brie/lightning-mhy-100G-csv
      s3Endpoint: http://minio.pingcap.net:9001
      db: sysbench
      backend: local
      sortedKvDir: /tmp/sorted-kv-dir
      options: "--enable-checkpoint false"
      config: "[[mydumper.files]]\npattern = \"sysbench-schema-create.sql\"\nschema = \"sysbench\"\ntype = \"schema-schema\"\n[[mydumper.files]]\npattern = \"sysbench.user_data1-schema.sql\"\nschema = \"sysbench\"\ntable = \"user_data1\"\ntype = \"table-schema\"\n[[mydumper.files]]\npattern = \"sysbench.user_data1.000000[0-3][0-9][0-9].csv\"\nschema = \"sysbench\"\ntable = \"user_data1\"\ntype = \"csv\""
      crc32Checksum: 619903217

What did you expect to see?

1、no panic
2、lightning import can success

What did you see instead?

pd panic
lightning import failed

tidb lightning encountered error: [Lightning:PreCheck:ErrCheckClusterRegion]check tikv cluster region error: check tasks exclusively failed: get http://tc-pd-1.tc-pd-peer.ha-test-lightning-tps-3570388-1-187.svc:2379/pd/api/v1/regions/check/empty-region http status code != 200, message PANIC: runtime error: invalid memory address or nil pointer dereference
goroutine 447078 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:159 +0xad
panic({0x2b1b140?, 0x4d5cdc0?})
/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/tikv/pd/pkg/core.(*RegionInfo).Clone(0x0, {0x0, 0x0, 0xc00a3f8380?})
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/core/region.go:262 +0x32
github.com/tikv/pd/pkg/statistics.(*RegionStatistics).GetRegionStatsByType(0xc006ef3d60, 0x100000?)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/statistics/region_collection.go:122 +0x1c9
github.com/tikv/pd/server/cluster.(*RaftCluster).GetRegionStatsByType(...)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/cluster/cluster.go:2242
github.com/tikv/pd/server.(*Handler).GetRegionsByType(0x1?, 0xa3f8468?)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/handler.go:395 +0x72
github.com/tikv/pd/server/api.(*regionsHandler).getRegionsByType(0xc0010aeb60, {0x7f8e1e989860?, 0xc008ac9068}, 0x20?, 0xc00e40db00)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/region.go:553 +0x45
github.com/tikv/pd/server/api.(*regionsHandler).GetEmptyRegions(...)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/region.go:643
github.com/tikv/pd/server/api.(*serviceMiddlewareBuilder).createHandler.WrapFunc.func1({0x7f8e1e989860, 0xc008ac9068}, 0x2bc41c0?, 0xc00c54eda0)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:56 +0x2b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x0?, {0x7f8e1e989860?, 0xc008ac9068?}, 0x374f6c8?, 0xc00a3f86d8?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x2d
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e3920?, 0xc0010af3d0?}, 0xc0010e8870?}, {0x7f8e1e989860, 0xc008ac9068}, 0xc00a3f86d8?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/tikv/pd/server/api.(*rateLimitMiddleware).ServeHTTP(0xc00eb03400?, {0x7f8e1e989860?, 0xc008ac9068?}, 0xc00a3f8800?, 0xf6e92f?)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/middleware.go:170 +0x32c
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e4140?, 0xc00078a548?}, 0xc0010e8858?}, {0x7f8e1e989860, 0xc008ac9068}, 0xc00e182fe0?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/tikv/pd/server/api.(*auditMiddleware).ServeHTTP(0xc00078a540, {0x7f8e1e989860, 0xc008ac9068}, 0xc00e40db00, 0xc00c54ed60)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/middleware.go:150 +0x56a
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e4160?, 0xc00078a540?}, 0xc0010e8840?}, {0x7f8e1e989860, 0xc008ac9068}, 0x39d8d01?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/tikv/pd/server/api.(*requestInfoMiddleware).ServeHTTP(0x8?, {0x7f8e1e989860, 0xc008ac9068}, 0xc00e40da00, 0xc00c54ed40)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/middleware.go:76 +0x31a
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e4180?, 0xc00078a538?}, 0xc0010e8828?}, {0x7f8e1e989860, 0xc008ac9068}, 0xfc1901?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0010eaea0, {0x7f8e1e989860?, 0xc008ac9060}, 0x4e84780?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x114
github.com/tikv/pd/server/api.clusterMiddleware.Middleware-fm.clusterMiddleware.Middleware.func1({0x7f8e1e989860?, 0xc008ac9060}, 0xc00ca8d9b0?)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/server/api/middleware.go:99 +0xb8
net/http.HandlerFunc.ServeHTTP(0xc00e40d700?, {0x7f8e1e989860?, 0xc008ac9060?}, 0x2bc41c0?)
/usr/local/go/src/net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0008420c0, {0x7f8e1e989860, 0xc008ac9060}, 0xc00e40d600)
/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x1c5
github.com/tikv/pd/server/api.NewHandler.Wrap.func7({0x7f8e1e989860, 0xc008ac9060}, 0x2bc41c0?, 0xc00c54ed20)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x45
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x4e474a0?, {0x7f8e1e989860?, 0xc008ac9060?}, 0x2dec8c0?, 0xc00f962af8?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x2d
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e3920?, 0xc001175068?}, 0xc0011750b0?}, {0x7f8e1e989860, 0xc008ac9060}, 0x18?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*redirector).ServeHTTP(0xc0011b12c0, {0x7f8e1e989860, 0xc008ac9060}, 0xc00e40d600, 0xc00c54ed00)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/utils/apiutil/serverapi/middleware.go:150 +0xeb
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e3900?, 0xc0011b12c0?}, 0xc001175098?}, {0x7f8e1e989860, 0xc008ac9060}, 0xc00eb03400?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/tikv/pd/pkg/utils/apiutil/serverapi.(*runtimeServiceValidator).ServeHTTP(0x8?, {0x7f8e1e989860, 0xc008ac9060}, 0x0?, 0xc00c54ece0)
/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/pd/pkg/utils/apiutil/serverapi/middleware.go:44 +0xb9
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e42c0?, 0xc00119c580?}, 0xc001175080?}, {0x7f8e1e989860, 0xc008ac9060}, 0x2a10a40?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc0011c6030, {0x7f8e1e989860?, 0xc008ac9058}, 0x2cc8e80?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x114
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0008423c0, {0x7f8e1e989860, 0xc008ac9058}, 0xc00e40d400)
/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x1c5
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000842000, {0x7f8e1e989860, 0xc008ac9058}, 0xc00e40d200)
/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210 +0x1c5
github.com/urfave/negroni.(*Negroni).UseHandler.Wrap.func1({0x7f8e1e989860, 0xc008ac9058}, 0x2bc41c0?, 0xc00c54ecc0)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:46 +0x45
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0xf304a8?, {0x7f8e1e989860?, 0xc008ac9058?}, 0x1344516?, 0x2e24c38?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:29 +0x2d
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e3920?, 0xc0011751d0?}, 0xc001175200?}, {0x7f8e1e989860, 0xc008ac9058}, 0x2bc41c0?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/urfave/negroni.(*Recovery).ServeHTTP(0x8?, {0x7f8e1e989860?, 0xc008ac9058?}, 0x0?, 0xc00a3f99a0?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/recovery.go:193 +0x7b
github.com/urfave/negroni.middleware.ServeHTTP({{0x39e3c40?, 0xc0004133b0?}, 0xc0011751e8?}, {0x7f8e1e989860, 0xc008ac9058}, 0xc00e182f80?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:38 +0xa8
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc000840240, {0x39f3f60?, 0xc00684b420}, 0xc00ca8d4a0?)
/go/pkg/mod/github.com/urfave/negroni@v0.3.0/negroni.go:96 +0x114
net/http.(*ServeMux).ServeHTTP(0x39f3f60?, {0x39f3f60, 0xc00684b420}, 0xc00e40d200)
/usr/local/go/src/net/http/server.go:2514 +0x142
go.etcd.io/etcd/embed.(*accessController).ServeHTTP(0xc001ae1650, {0x39f3f60, 0xc00684b420}, 0xc00e40d200)
/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/embed/serve.go:381 +0x4e5
net/http.serverHandler.ServeHTTP({0xc012890a50?}, {0x39f3f60?, 0xc00684b420?}, 0x6?)
/usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc00b6f9dd0, {0x39ffc38, 0xc000786d50})
/usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 396
/usr/local/go/src/net/http/server.go:3086 +0x5cb

What version of PD are you using (pd-server -V)?

./pd-server -V
Release Version: v7.6.0-alpha
Edition: Community
Git Commit Hash: 3f1a688
Git Branch: heads/refs/tags/v7.6.0-alpha
UTC Build Time: 2023-10-23 12:20:17

[Lily2025]

/severity critical

[lhy1024]

after #6804, we get region from RegionInfoProvider, where region maybe nil.