Message Sync No data

[harveyff]

Hi, team

Thanks for your great work.

I am trying to sync hubble in k8s, after more than ten hours, looking at the log it seems to have completed the sync, but in the monitoring view, Message Sync is always No Data.

Logs:

{"message":"{\"level\":30,\"time\":1721878200024,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"GossipNode\",\"bootstrapAddrs\":[\"/dns/nemes.farcaster.xyz/tcp/2282\",\"/dns/hoyt.farcaster.xyz/tcp/2282\",\"/dns/lamia.farcaster.xyz/tcp/2282\",\"/dns/bootstrap.neynar.com/tcp/2282\"],\"msg\":\"Bootstrapping Gossip Node\"}\n","timestamp":"2024-07-25T03:30:00.024Z","type":"out","process_id":0,"app_name":"hubble"}
{"level":50,"time":1721878200024,"pid":30,"hostname":"farcasterhubble-569864c58-fn4dr","component":"GossipNodeWorker","err":{"type":"AggregateError","message":"All promises were rejected","stack":"AggregateError: All promises were rejected","aggregateErrors":[]},"msg":"Failed to connect to peer at address: /dns/nemes.farcaster.xyz/tcp/2282"}
{"level":50,"time":1721878200025,"pid":30,"hostname":"farcasterhubble-569864c58-fn4dr","component":"GossipNodeWorker","err":{"type":"AggregateError","message":"All promises were rejected","stack":"AggregateError: All promises were rejected","aggregateErrors":[]},"msg":"Failed to connect to peer at address: /dns/hoyt.farcaster.xyz/tcp/2282"}
{"level":50,"time":1721878200025,"pid":30,"hostname":"farcasterhubble-569864c58-fn4dr","component":"GossipNodeWorker","err":{"type":"AggregateError","message":"All promises were rejected","stack":"AggregateError: All promises were rejected","aggregateErrors":[]},"msg":"Failed to connect to peer at address: /dns/lamia.farcaster.xyz/tcp/2282"}
{"level":50,"time":1721878200025,"pid":30,"hostname":"farcasterhubble-569864c58-fn4dr","component":"GossipNodeWorker","err":{"type":"AggregateError","message":"All promises were rejected","stack":"AggregateError: All promises were rejected","aggregateErrors":[]},"msg":"Failed to connect to peer at address: /dns/bootstrap.neynar.com/tcp/2282"}
{"message":"{\"level\":40,\"time\":1721878200026,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"PeriodicPeerCheckScheduler\",\"err\":{\"type\":\"HubError\",\"message\":\"could not connect to any bootstrap nodes\",\"stack\":\"HubError: could not connect to any bootstrap nodes\\n    at GossipNode.bootstrap (file:///home/node/app/apps/hubble/src/network/p2p/gossipNode.ts:684:18)\\n    at PeriodicPeerCheckScheduler.doJobs (file:///home/node/app/apps/hubble/src/network/p2p/periodicPeerCheck.ts:48:20)\",\"errCode\":\"unavailable\",\"presentable\":false,\"name\":\"HubError\"},\"msg\":\"No Connected Peers\"}\n","timestamp":"2024-07-25T03:30:00.026Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878202387,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"FNameRegistryEventsProvider\",\"msg\":\"Fetched 0 fname events upto 622444\"}\n","timestamp":"2024-07-25T03:30:02.387Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878202590,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139712,\"msg\":\"new block: 123139712\"}\n","timestamp":"2024-07-25T03:30:02.590Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878206611,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139714,\"msg\":\"new block: 123139714\"}\n","timestamp":"2024-07-25T03:30:06.611Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878207639,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"FNameRegistryEventsProvider\",\"msg\":\"Fetched 0 fname events upto 622444\"}\n","timestamp":"2024-07-25T03:30:07.639Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878209049,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"GossipContactInfo\",\"msg\":\"starting gossip contact info job\"}\n","timestamp":"2024-07-25T03:30:09.049Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878209051,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"Hub\",\"rpcAddress\":\"47.238.88.101\",\"rpcPort\":2283,\"msg\":\"gossiping contact info\"}\n","timestamp":"2024-07-25T03:30:09.051Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878210641,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139716,\"msg\":\"new block: 123139716\"}\n","timestamp":"2024-07-25T03:30:10.641Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878212899,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"FNameRegistryEventsProvider\",\"msg\":\"Fetched 0 fname events upto 622444\"}\n","timestamp":"2024-07-25T03:30:12.899Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878214665,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139718,\"msg\":\"new block: 123139718\"}\n","timestamp":"2024-07-25T03:30:14.665Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878218144,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"FNameRegistryEventsProvider\",\"msg\":\"Fetched 0 fname events upto 622444\"}\n","timestamp":"2024-07-25T03:30:18.144Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878218700,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139720,\"msg\":\"new block: 123139720\"}\n","timestamp":"2024-07-25T03:30:18.700Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878222724,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139722,\"msg\":\"new block: 123139722\"}\n","timestamp":"2024-07-25T03:30:22.724Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878223389,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"FNameRegistryEventsProvider\",\"msg\":\"Fetched 0 fname events upto 622444\"}\n","timestamp":"2024-07-25T03:30:23.389Z","type":"out","process_id":0,"app_name":"hubble"}
{"message":"{\"level\":30,\"time\":1721878226741,\"pid\":30,\"hostname\":\"farcasterhubble-569864c58-fn4dr\",\"component\":\"L2EventsProvider\",\"blockNumber\":123139724,\"msg\":\"new block: 123139724\"}\n","timestamp":"2024-07-25T03:30:26.741Z","type":"out","process_id":0,"app_name":"hubble"}

Monitoring

Info API
curl https://xx.com/v1/info?dbstats=1

{"version":"1.13.8","isSyncing":false,"nickname":"Farcaster Hub","rootHash":"7f1ec44f8a881d93324174dae2bc6c3e9145373b","dbStats":{"numMessages":489445060,"numFidEvents":808038,"numFnameEvents":621479,"approxSize":152020335378},"peerId":"12D3KooWN5GVyXwo7SCWytrEuPzDXkjGWsjXkqv1sJmXEWGSmWpb","hubOperatorFid":0}