NoSuchKeyset Error On Sequencer Nodes and Full Nodes. Anytrust chain.

[yohanelly95]

Describe the bug
I get the error NoSuchKeyset on the sequencer node. When I have set a valid keyset (das_bls, das_bls.pub) and used the correct nodeConfig.json as well. At first, it was happening whenever I ran a Full Node. Now it's happening on the sequencer node as well. There is a single DAS which is an archival DAS.

To Reproduce

• Create a bls key
• Call setValidKeysetPrepareTransactionRequest and send the txn.
• Update the nodeConfig.json
• Start the DAS (using S3 bucket)
• Start sequencer node
• Start a full node
• Observe errors

Expected behavior

• No errors like NoSuchKeyset(0x1c4b4d38f1f5396ecb0beee91f91df91dde26d647ca7ceaa63deb8a002e0616f)
• Unsure what keyset this even is, I can share the orbit contract addresses, verified that this keyset is invalid. It does not exist anywhere on my setup nor have I set this keyset hash on-chain.

Screenshots

nitro-1  | ERROR[06-20|11:48:41.718] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x1c4b4d38f1f5396ecb0beee91f91df91dde26d647ca7ceaa63deb8a002e0616f)"
nitro-1  | ERROR[06-20|11:48:41.718] Error while recording                    err="execution reverted: NoSuchKeyset(0x1c4b4d38f1f5396ecb0beee91f91df91dde26d647ca7ceaa63deb8a002e0616f)" status=2
nitro-1  | WARN [06-20|11:48:41.753] Recording for validation failed, retrying.. pos=36
nitro-1  | INFO [06-20|11:48:41.753] rpc response                             method=validation_validate logId=15535 err="context canceled"   result="{\"BlockHash\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"SendRoot\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"Batch\":0,\"PosInBatch\":0}" attempt=0 args="[{\"Id\":41,\"HasDelayedMsg\":false,\"DelayedMsgNr\":0,\"PreimagesB64\":{\"0\":{\"5tmRWoFon8a4SzT0QL65GVgBHrbwXfBG3W2PY+lKNBo=\":\"+QFxoBHPjkg7l3Wi0f160UoUDL99kyw1HmszBVvxe5EGMYlRoHDqOOt1w5yW4UdQN37tvhksFDhzrrPRDwtMzImuNRXmoJ8HKIBRIrDK3IvYUkRgxRd9fIjCOWobJ/34ldQXUeRCgICgcsGFp9ERPPdQvZCZA7zk4bdRc31bAnK+spYKVyRYthagZN4H9I5sBAwRSlPkik8GHTb1scsdW/T4VrqwEcDxPAGAoBw7U4wREPnVpUQOjcCwtUgkPc0NaAYZba8w9f3M1KiToFM+ZA/K48nrgI/goFzFAo5u3BYT2IC+i/T1QECaCgO4oAIyU8hHRgdXVKwfE8LaBflR+QfNt4Zw0jhuU+W3nZI7gKDb35mR59ltEBmLcUmjcY8+8VV5Xq15A+TnvIbpbsxa7ICg/PY8HNGnS2dy2L/41+dIq7TO/tAYsNx2ePSPGNHkcYig3pwGfrE8Y7NUbuXry/uUt1JKRZ27OIYSq4H8f1yLztqA\",\"29+ZkefZbRAZi3FJo3GPPvFVeV6teQPk57yG6W7MWuw=\":\"+EOgIDiCgRJhbI8QAZAI5U2U/brCatoZSYPb5flPQR/BjxWhoDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAw\",\"49rO9TF+jZzrFCb4an5Jn8M5ZWEUUHZBwvdiO4JJLxI=\":\"+FGAgICAoEDjWGqBSWWRd0D1Pn0bGEyci8rRpOiw2uw9GOEDDzqYgICAoAB1sGNyD0G9ZMMBz6aG3LsrF6rzbkNhWyvyPaakgFf4gICAgICAgIA=\",\"OMpN+bNBVL+6uTyOm1FcZuGj1tJjub8iUNaTu20RV0I=\":\"+FGAgICgyyOAwBqtMRCFj1nPjs17i25no1kQsw91aR0KU9/1nJSAgICg..mROdRmPLSFoxJ0cQrKXfrfHehoHRydWUsIkFsbG93RGVidWdQcmVjb21waWxlcyI6ZmFs\",\"u6FJup/VkNpgc25lJo165FFdkgXsfhD5TILCv62vaHA=\":\"+HGgDzPiMM/gcXdYX68OKAIbEdikuuUVAJk2AGWKdqLl1YmAgICAgICAgICAoH0vdvXJN9HEaQuN9bcztoLuDtA9ADSn/wcAXOJAjTdagICAoMVAix51R5kTws7sz4nRn3dLT5GX7NWNL0aRNWugs5WhgA==\",\"vJogyNE/V6GQbCHVu00d0x83LfWOasVed3pmCLgK4sQ=\":\"/vpGVdkCqd1ZhzxpXDoxyOZdImu1ujTDf5GeYcNXe937\",\"Uh8MVrcDodVb+0faoqU6JjxCyamsh1O4Oq4nYeJunq4=\":\"4qAgEr3YLiIfen37rrBoFjCKfYxwBO4G6+jvvNiRdrtqZhQ=\"}},\"BatchInfo\":[{\"Number\":7,\"DataB64\":\"AAAAAGZux+kAAAAAZnQb+QAAAAAAXVvIAAAAAABdzXQAAAAAAAAAFIg8miDI0T9XoZBsIdW7TR3THzct9Y5qxV53emYIuArixKJ7DOEJAuDqO+wWipkIdDoT+u/wcr54RUQPtW5gvin4AAAAAGaH1GgBAAAAAAAAAAEIAL0wAWF+THWij+ujhWrxEXL3qh7TZz5xPczQqQRAYbkGgV9009xc4Vob+5D8kZoXmgWv/1cPO3+2EN3W7tAjeGUBS1KD44yGP/XayDXr5QDqErfDWASKhnhQKRpc2Ws=\"}],\"DelayedMsgB64\":\"\",\"StartState\":{\"BlockHash\":\"0x87d32fba67440b8ac13e9d3a4f327775ffd5df6412364feb879b1126f9050a2e\",\"SendRoot\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"Batch\":7,\"PosInBatch\":5}}, \"0x8b104a2e80ac6165dc58b9048de12f301d70b02a0ab51396c22b4b4b802a16a4\"]"

I see the above logs on my sequencer node continuously now.

When I run a Full node this is what I get:

Completely different keyset hash. Using the same nodeConfig.json.

My das-config on the nodeConfig.json file:

 "data-availability": {
      "enable": true,
      "sequencer-inbox-address": "0x89Cd084622c2233b3166cAD9C82778DfeD5A0Eb4",
      "parent-chain-node-url": "RPC",
      "rest-aggregator": {
        "enable": true,
        "urls": [
          "http://EXTERNAL_DAS_IP:9877"
        ]
      },
      "rpc-aggregator": {
        "enable": true,
        "assumed-honest": 1,
        "backends": "[{\"url\":\"http://EXTERNAL_DAS_IP:9876\",\"pubkey\":\"VALID_PUB_KEY\",\"signermask\":1}]"
      }
    }
  }

[yohanelly95]

What I see on full nodes:

INFO [06-21|05:33:37.775] rpc response                             method=eth_call logId=37 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:37.775] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:37.775] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:39.660] rpc response                             method=eth_call logId=48 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:39.661] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:39.661] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:41.492] rpc response                             method=eth_call logId=59 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:41.493] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:41.493] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:43.324] rpc response                             method=eth_call logId=70 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:43.324] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:43.324] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:45.133] rpc response                             method=eth_call logId=81 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:45.133] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:45.133] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:47.026] rpc response                             method=eth_call logId=92 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:47.027] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:47.027] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
INFO [06-21|05:33:48.844] rpc response                             method=eth_call logId=103 err="execution reverted" result="\"0x\"" attempt=0 args="[{\"from\":\"0x0000000000000000000000000000000000000000\",\"input\":\"0x258f04954722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\",\"to\":\"0x89cd084622c2233b3166cad9c82778dfed5a0eb4\"}, \"latest\"]" errorData="\"0x00f20c5d4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93\""
ERROR[06-21|05:33:48.844] Couldn't fetch DAS batch contents        err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"
WARN [06-21|05:33:48.844] error reading inbox                      err="execution reverted: NoSuchKeyset(0x4722832035aff64c8e9f940afdb0bd42b35b5b7297f13091a3e1d1a03b4f9d93)"

[yohanelly95]

Note: This only occurs when using a custom BLS signature