Eth2Sub transactions are removed from txpool

[svyatonik]

Seems like after last Substrate update, there's a problem with eth2sub syncing. Some transactions are removed from tx pool with ExhaustsResources error. Relay currently do not watch for submitted transactions => because nothing is updated in Substrate state, it waits 5 minutes before 'restarting' itself and resubmitting same headers again.

I'm not sure, if that's the problem with:

1. our configuration - maybe it works that way with #[weight = 0]? something is missed from the runtime;
2. some already-fixed Substrate issue - probably ref update will solve this;
3. not-yet-fixed Substrate issue.

In any case, dropping transactions from pool with that error seems strange - the pool is empty (there's at most 4 signed transactions there during eth2sub), so which resources are exhausted?

Required for #91

log

[svyatonik]

Just got the same error when submitting transaction:

Bridge: [2020-05-18 16:13:36] 2020-05-18 16:13:36 +0300 DEBUG bridge Submitting 32 header(s) to Substrate node: "[HeaderId(129, 0x98ce2ef50ee57ddd6183477ef23791851faef7cdb3c7713c28d9bdf05ae17c00) ... HeaderId(160, 0xa70e553b05ee5cfa9a1042c12e9799038612b575f8e7de73945d7736c63c5197)]"

Substrate: [2020-05-18 16:13:36] 2020-05-18 16:13:36.335 http.worker30 DEBUG rpc  State nonce for 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY: 5

Substrate: [2020-05-18 16:13:36] 2020-05-18 16:13:36.336 http.worker30 DEBUG rpc  Response: {"jsonrpc":"2.0","result":5,"id":270}.

Substrate: [2020-05-18 16:13:36] 2020-05-18 16:13:36.379 http.worker30 DEBUG rpc  Response: {"jsonrpc":"2.0","error":{"code":1010,"message":"Invalid Transaction","data":"ExhaustsResources"},"id":271}.

Bridge: [2020-05-18 16:13:36] 2020-05-18 16:13:36 +0300 ERROR bridge Error submitting headers to Substrate node: ResponseRetrievalFailed(RequestError(Error { code: MethodError(1010), message: "Invalid Transaction", data: Some(String("ExhaustsResources")) }))

Parity: [2020-05-18 16:13:37] 2020-05-18 16:13:37  Syncing    #4293 0x0af4…becc   216.20 blk/s    0.0 tx/s    0.0 Mgas/s      0+ 1289 Qed     #5588    1/25 peers   1 MiB chain 6 MiB db 2 MiB queue 923 KiB sync  RPC:  0 conn,   25 req/s, 1780 µs

[tomusdrw]

What chain spec are you running? I'd assume that some of the block weight parameters are misconfigured and every transaction just ends up being over the limit.
I didn't find any special casing for 0-weight transactions.

[svyatonik]

I'm running this script, which syncs Kovan headers into our local (bridge local) node:

#!/bin/bash
set -e

rm -rf substrate.db
rm -rf parity.db

cargo build --manifest-path=../parity-bridges-common/Cargo.toml -p bridge-node
cp ../parity-bridges-common/target/debug/bridge-node .
cargo build --manifest-path=../parity-bridges-common/Cargo.toml -p ethereum-poa-relay
cp ../parity-bridges-common/target/debug/ethereum-poa-relay .

unbuffer ./parity -d parity.db --chain kovan --no-warp --no-persistent-txqueue --jsonrpc-apis=all 2>&1 | unbuffer -p gawk '{ print strftime("Parity: [%Y-%m-%d %H:%M:%S]"), $0 }' | unbuffer -p tee parity.log&
RUST_LOG=runtime=debug,rpc=debug,txpool=trace unbuffer ./bridge-node --execution=Native --dev -d parity.db 2>&1 | unbuffer -p gawk '{ print strftime("Substrate: [%Y-%m-%d %H:%M:%S]"), $0 }' | unbuffer -p tee substrate.log&
sleep 10
RUST_LOG=bridge=trace unbuffer ./ethereum-poa-relay eth-to-sub 2>&1 | unbuffer -p gawk '{ print strftime("Bridge: [%Y-%m-%d %H:%M:%S]"), $0 }' | unbuffer -p tee bridge.log&

[svyatonik]

What chain spec are you running? I'd assume that some of the block weight parameters are misconfigured and every transaction just ends up being over the limit.
I didn't find any special casing for 0-weight transactions.

The same transaction is included into the block after resubmitting. And it is only dropped in ~20% cases. So it seems there's some other issue than simply being over limit. Also - what's the limit you're talking about? Max block weight? It is much larger than zero

[tomusdrw]

Should be solved by: paritytech/substrate#6067

[svyatonik]

Yeah, thanks! Will need to update Substrate ref.