[Bug] Some subgraphs can't be synced under Erigon V3 and OP_Erigon

[trader-payne]

Bug report

Optimism subgraphs with OP_Erigon stack (based on E2):

Qmcph8jr4WpHuqKU8WnmUmDZrmG1sbZ6RTZCt5DZaDEG8C
QmVngsyJ6dS8uSLyQ3JSKc9oyK68XMXB6iw5iR2wLezsE9
Qmf9i13TJknQwcamcLbb75hEhbf3nqDi321XWhtFBj3P5s
QmSeJkpvWGcm9XceLc7qV9kKbc2pmo4xyfcfEd8GLgpVDw
QmdN4vQir9KZj7rasEtb7WhKJvy28BDdHBko3btS31LqW5
QmSBKLdZMeTVrnAWsjagoWbXSd3qLC5NriGEWANQFhu63b
QmWTY7MvjLsALSXJJpHj7haGeWc8KxZ845YC1DJT7W9uAm
Qmc7RUfYjqSoumraPNAHYyvnEV12cQNZsn2hD1xzsyy57D

Jul 01 18:27:55.352 DEBG Trying again after eth_call RPC call for block #106102561 (b43e9fe96bcaa0298870739064f03ea7c5bd6e454ecd14c2ebfd85378b0ecd18) failed (attempt #8) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 363927226539702117 want 365742326500000000", data: None }))), provider: archive-optimism, sgd: 1198, subgraph_id: QmSeJkpvWGcm9XceLc7qV9kKbc2pmo4xyfcfEd8GLgpVDw, component: SubgraphInstanceManager

Jul 01 18:28:07.683 DEBG Trying again after eth_call RPC call for block #106102561 (b43e9fe96bcaa0298870739064f03ea7c5bd6e454ecd14c2ebfd85378b0ecd18) failed (attempt #6) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 363927226539702117 want 365742326500000000", data: None }))), provider: archive-optimism, sgd: 673, subgraph_id: Qmf9i13TJknQwcamcLbb75hEhbf3nqDi321XWhtFBj3P5s, component: SubgraphInstanceManager

Base Subgraphs with OP_Erigon (based on E2):

QmW7M6xQiJ1APXfpBNXYaArH2QsNBV8xpESqLxWNKvnqks
QmPWDTWRfS8XewMJhxXoQiZX6RkrUMYH7BFnnB846v6wzR
QmTTXzn6iBcAX2Z72yJw6e64yfCX6c5HcTpKaRacfTLa7g
QmbufMKR6DbFDmxESCDKMYxSMVrZJQGn2is9EmosctKX64
QmUbg8H3MJoFrmJk5dVSuY4sYF3VVwuizoajqVgHBR6a5B

Jul 01 18:55:55.992 DEBG Trying again after eth_call RPC call for block #1750275 (c3fa42fc80a8cb6093fd9417b7b9a7d6aab4e19bf21d341ccb976eeb9091b6d9) failed (attempt #4) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 0 want 2500000000", data: None }))), provider: archive-base, sgd: 1108, subgraph_id: QmW7M6xQiJ1APXfpBNXYaArH2QsNBV8xpESqLxWNKvnqks, component: SubgraphInstanceManager

Jul 01 18:56:29.978 DEBG Trying again after eth_call RPC call for block #2436025 (a72ceae3ebfb81a9bf830e3d436fcd1a7964e5ff069ddca7b8373b0b5bf5d738) failed (attempt #4) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 16615255275724386 want 21737500450000000", data: None }))), provider: archive-base, sgd: 1081, subgraph_id: QmPWDTWRfS8XewMJhxXoQiZX6RkrUMYH7BFnnB846v6wzR, component: SubgraphInstanceManager

Jul 01 18:57:22.565 DEBG Trying again after eth_call RPC call for block #2436025 (a72ceae3ebfb81a9bf830e3d436fcd1a7964e5ff069ddca7b8373b0b5bf5d738) failed (attempt #4) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 16615255275724386 want 21737500450000000", data: None }))), provider: archive-base, sgd: 1152, subgraph_id: QmTTXzn6iBcAX2Z72yJw6e64yfCX6c5HcTpKaRacfTLa7g, component: SubgraphInstanceManager

Jul 01 18:57:44.363 DEBG Trying again after eth_call RPC call for block #2436035 (c4f82b0fec98dd5c1de5859d56d554f1a267049f76d08dfe3cb54c5f1cb04da1) failed (attempt #4) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "insufficient funds for gas * price + value: address 0x0000000000000000000000000000000000000000 have 16615255275724386 want 47559166600000000", data: None }))), provider: archive-base, sgd: 1063, subgraph_id: QmbufMKR6DbFDmxESCDKMYxSMVrZJQGn2is9EmosctKX64, component: SubgraphInstanceManager

Gnosis subgraphs synced with Erigon V3:

QmWHxCtPAktZnfqnCEgPcuSesv4oDQssjgyzMvCAE4RQqr
QmUbSegbKWhJ4N98iRjfEpMiogGLF2cN77Uqqbd5NKm6g7
QmS2S1qgZbM5Cr3Q6DZNcLuHnCavzqbvBFcPksNMCeoDEN
QmacRwmkz84Uz5zW4xAsyqzTYBPy8xGNoY6n4c11P8sGjF
QmNafaHCm1PHNhR5LD9raptJiB6KDxW3Yc7pYpj4ZqjHup

Jul 01 18:50:20.919 DEBG Trying again after eth_getTransactionReceipt RPC call for transaction 0x2de66204139cc8e1c5b81a3f502c6d15e9c3b5c3ec328836161a4d714a68fe7a failed (attempt #4) with result Err(Rpc(Error { code: ServerError(-32000), message: "getReceipts error: fee cap less than block base fee: address 0xC7b90a1b52d5E28EB3e6ea35E952975467bf9930, gasFeeCap: 0 baseFee: 205929208", data: None })), sgd: 997, subgraph_id: QmWHxCtPAktZnfqnCEgPcuSesv4oDQssjgyzMvCAE4RQqr, component: BlockStream

Jul 01 18:50:54.214 DEBG Trying again after eth_getTransactionReceipt RPC call for transaction 0x4b50e89b65281735a845eb7455898f41b001de761357fa55940ef0768ec42103 failed (attempt #4) with result Err(Rpc(Error { code: ServerError(-32000), message: "getReceipts error: fee cap less than block base fee: address 0x67342Ec1440bFB760Ba9805a7735e0D12Bb9fEc7, gasFeeCap: 0 baseFee: 141015840", data: None })), sgd: 750, subgraph_id: QmUbSegbKWhJ4N98iRjfEpMiogGLF2cN77Uqqbd5NKm6g7, component: BlockStream

Jul 01 18:51:33.669 WARN Unexpected RPC error, error: Rpc(Error { code: ServerError(-32000), message: "fee cap less than block base fee: address 0x19aC7c69e5F1AC95b8d49b30Cbb79e81f1ab0dba, gasFeeCap: 0 baseFee: 9: blockNum=19067637, txNum=108853209, %!s(<nil>)", data: None }), sgd: 967, subgraph_id: QmS2S1qgZbM5Cr3Q6DZNcLuHnCavzqbvBFcPksNMCeoDEN, component: BlockStream

Jul 01 18:52:46.646 WARN Unexpected RPC error, error: Rpc(Error { code: ServerError(-32000), message: "fee cap less than block base fee: address 0xdac288df7A6e253578711fdD1BF3cCB877f0F7F9, gasFeeCap: 0 baseFee: 7: blockNum=19606562, txNum=116983371, %!s(<nil>)", data: None }), sgd: 1077, subgraph_id: QmNafaHCm1PHNhR5LD9raptJiB6KDxW3Yc7pYpj4ZqjHup, component: BlockStream

If we look at the logs, there seem to be two separate issues here, namely:

• OP Erigon stack - insufficient funds for gas * price + value
• Canonical Erigon V3 - fee cap less than block base fee

Relevant log output

No response

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

[alex-pakalniskis]

Deeper investigation still required but @mangas suggested this could be a configuration issue as he experienced a similar problem with Reth.

Relevant configurations:

- name: GRAPH_GETH_ETH_CALL_ERRORS
  value: "invalidjump;invalidfeopcode"
- name: GRAPH_ETH_CALL_GAS
  value: "50000000"
- name: GRAPH_ETH_CALL_NO_GAS
  value: "421613,42161"

[trader-payne]

@alex-pakalniskis just seeing your comment now.
I have set the following:

GRAPH_ETH_CALL_GAS: 50000000
GRAPH_ETH_CALL_NO_GAS: "421613,42161" # I had optimism&gnosis chain IDs here too

I didn't have the GRAPH_GETH_ETH_CALL_ERRORS variable as I don't know what it's doing.
I'll try to replicate the issue again, this time having the 50000000 value as a string "50000000".

I can give you guys access to Gnosis V2 and V3 endpoints if you want to try it out. I'll also spin up a fresh test node to triple check.

[suntzu93]

I can confirm that op-erigon (optimism) and op-erigon (base) on version v2.60.4-0.6.6 fixed the issue of insufficient funds for gas * price + value.

[cryptovestor21]

I can confirm that op-erigon (optimism) and op-erigon (base) on version v2.60.4-0.6.6 fixed the issue of insufficient funds for gas * price + value.

+1 for base.

erigon version 0.07.0-unstable (v2.60.5-0.7.0)
op-node version v1.9.0-ec45f663-1723023640