Emitting Ethereum related events in PostTxProcessing EVM hooks make Ethereum events inconsistent

[lumtis]

The PR #991 introduces payment of gas when sending tokens out of ZetaChain, the method is called in the PostTxProcessing hook, the method call CallEVM that will emit Ethereum events.

It appears that emitting Ethereum events while in PostTxProcessing makes the overall event logs emitted inconsistent.

The tx can be processed but then when querying the receipt from the tx hash, inconsistencies happen.

If the all the events in CallEVMWIthData are emitted in a PostTxProcessing call:

	ctx.EventManager().EmitEvents(sdk.Events{
		sdk.NewEvent(
			evmtypes.EventTypeEthereumTx,
			attrs...,
		),
		sdk.NewEvent(
			evmtypes.EventTypeTxLog,
			txLogAttrs...,
		),
		sdk.NewEvent(
			sdk.EventTypeMessage,
			sdk.NewAttribute(sdk.AttributeKeyModule, types.ModuleName),
			sdk.NewAttribute(sdk.AttributeKeySender, from.Hex()),
			sdk.NewAttribute(evmtypes.AttributeKeyTxType, "88"), // type 88: synthetic Eth tx
		),
	})

Then, the following rpc error happens when trying to get the receipt:

acore0  | panic({0x27c80c0, 0x4004074690})
zetacore0  | 	/usr/local/go/src/runtime/panic.go:884 +0x20c
zetacore0  | github.com/zeta-chain/zetacore/rpc/types.(*ParsedTxs).updateTx(0x4003820880, 0x1, {0x400399ddc0?, 0x400307f500?, 0x40039a9f40?})
zetacore0  | 	/go/delivery/zeta-node/rpc/types/events.go:283 +0x164
zetacore0  | github.com/zeta-chain/zetacore/rpc/types.ParseTxResult(0x4003d9e168, {0x0, 0x0})

After investigation, the logic parsing tx events is somehow expecting the events to be emitted in "format 2", introduced in evmos/ethermint#1121
Possibly because the first Ethereum events emitted in the TX are in "format 2"

If we emit the events in the format 2:

	ctx.EventManager().EmitEvents(sdk.Events{
		sdk.NewEvent(
			evmtypes.EventTypeEthereumTx,
			sdk.NewAttribute(evmtypes.AttributeKeyEthereumTxHash, ethTxHash.String()),
			sdk.NewAttribute(evmtypes.AttributeKeyTxIndex, strconv.FormatUint(8888, 10)),
		),
		sdk.NewEvent(
			evmtypes.EventTypeEthereumTx,
			attrs...,
		),
		sdk.NewEvent(
			evmtypes.EventTypeTxLog,
			txLogAttrs...,
		),
		sdk.NewEvent(
			sdk.EventTypeMessage,
			sdk.NewAttribute(sdk.AttributeKeyModule, types.ModuleName),
			sdk.NewAttribute(sdk.AttributeKeySender, from.Hex()),
			sdk.NewAttribute(evmtypes.AttributeKeyTxType, "88"), // type 88: synthetic Eth tx
		),
	})

Then the following RPC error happens when getting the receipt:

zetacore0  | 2:08PM ERR RPC method eth_getTransactionReceipt crashed: runtime error: index out of range [4] with length 1
zetacore0  | goroutine 5553 [running]:
zetacore0  | github.com/ethereum/go-ethereum/rpc.(*callback).call.func1()
zetacore0  | 	/go/pkg/mod/github.com/ethereum/go-ethereum@v1.10.26/rpc/service.go:200 +0x74
zetacore0  | panic({0x27c8460, 0x4003e248a0})
zetacore0  | 	/usr/local/go/src/runtime/panic.go:884 +0x20c
zetacore0  | github.com/zeta-chain/zetacore/rpc/backend.(*Backend).GetTransactionReceipt(0x4000d1a500, {0xb9, 0x22, 0x98, 0x22, 0xa6, 0xaa, 0xdf, 0xd3, 0x49, ...})
zetacore0  | 	/go/delivery/zeta-node/rpc/backend/tx_info.go:171 +0x1830
zetacore0  | github.com/zeta-chain/zetacore/rpc/namespaces/ethereum/eth.(*PublicAPI).GetTransactionReceipt(0x4002f12f60, {0xb9, 0x22, 0x98, 0x22, 0xa6, 0xaa, 0xdf, 0xd3, 0x49, ...})
zetacore0  | 	/go/delivery/zeta-node/rpc/namespaces/ethereum/eth/api.go:196 +0x1a4
zetacore0  | reflect.Value.call({0x4001687aa0?, 0x4003280f20?, 0xffff80ed9108?}, {0x2972ca1, 0x4}, {0x4003f7e0a0, 0x2, 0x1b11438?})
zetacore0  | 	/usr/local/go/src/reflect/value.go:584 +0x688
zetacore0  | reflect.Value.Call({0x4001687aa0?, 0x4003280f20?, 0x40048772e0?}, {0x4003f7e0a0?, 0x1?, 0x16?})
zetacore0  | 	/usr/local/go/src/reflect/value.go:368 +0x90

It is possibly because we don't use the right AttributeKeyTxIndex in the event we emit.

Finally, when we skip the Ethereum tx event emit but keep the other events:

	ctx.EventManager().EmitEvents(sdk.Events{
		sdk.NewEvent(
			evmtypes.EventTypeEthereumTx,
			attrs...,
		),
		sdk.NewEvent(
			evmtypes.EventTypeTxLog,
			txLogAttrs...,
		),
		sdk.NewEvent(
			sdk.EventTypeMessage,
			sdk.NewAttribute(sdk.AttributeKeyModule, types.ModuleName),
			sdk.NewAttribute(sdk.AttributeKeySender, from.Hex()),
			sdk.NewAttribute(evmtypes.AttributeKeyTxType, "88"), // type 88: synthetic Eth tx
		),
	})

We can get the receipt but in the smoke test we will not get the event logs:

orchestrator  | =======================================
orchestrator  | Step 4: Sending ZETA from ZEVM to Ethereum
orchestrator  | =======================================
orchestrator  | zevm chainid: 101
orchestrator  | Deposit tx hash: 0xc00861d0c932d89f645fad9d1feff9b1bb5d90da97e494640d96be6f6fe0276c
orchestrator  | Deposit tx receipt: status 1
orchestrator  | wzeta.approve tx hash: 0x6d34d516b544c7d4d314ca41bbd9d64d7c6ce1f85c2eff2a4e56d0c34305cde4
orchestrator  | approve tx receipt: status 1
orchestrator  | send tx hash: 0xf052d915c3bc447c53bc4198905a567a9b254c94f0a4dbd72df64703c0ed52fc
orchestrator  | send tx receipt: status 1
orchestrator  |   Logs:
orchestrator  | waiting for cctx status to change to final...

Instead of

orchestrator  | =======================================
orchestrator  | Step 4: Sending ZETA from ZEVM to Ethereum
orchestrator  | =======================================
orchestrator  | zevm chainid: 101
orchestrator  | Deposit tx hash: 0xc00861d0c932d89f645fad9d1feff9b1bb5d90da97e494640d96be6f6fe0276c
orchestrator  | Deposit tx receipt: status 1
orchestrator  | wzeta.approve tx hash: 0x6d34d516b544c7d4d314ca41bbd9d64d7c6ce1f85c2eff2a4e56d0c34305cde4
orchestrator  | approve tx receipt: status 1
orchestrator  | send tx hash: 0xf052d915c3bc447c53bc4198905a567a9b254c94f0a4dbd72df64703c0ed52fc
orchestrator  | send tx receipt: status 1
orchestrator  |   Logs:
orchestrator  |     Dest Addr: 0xE5C5367B8224807Ac2207d350E60e1b6F27a7ecC
orchestrator  |     Dest Chain: 1337
orchestrator  |     Dest Gas: 250000
orchestrator  |     Zeta Value: 10000000000000000000
orchestrator  | waiting for cctx status to change to final...

We should investigate if there is a correct way to emit the events when using the call EVM method in PostTxProcessing