Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Near watcher test is flaky #2251

Closed
jynnantonix opened this issue Jan 16, 2023 · 2 comments
Closed

Near watcher test is flaky #2251

jynnantonix opened this issue Jan 16, 2023 · 2 comments
Milestone
Maintenance - Nod...

Comments

@jynnantonix
Copy link
Contributor

Cleaned up excerpt from https://github.com/wormhole-foundation/wormhole/actions/runs/3927088030/jobs/6714400443#logs :

2023-01-16T05:56:56.9357322Z 2023-01-16T05:56:56.792Z	ERROR	root.test.nearwatch	near/tx_processing.go:167	Wormhole publish event.block does not equal receipt_outcome[x].block_height	{"error_type": "malformed_wormhole_event", "log_msg_type": "tx_processing_error", "event.block": 76538234, "receipt_outcome[x].block_height": 76538229}
2023-01-16T05:56:56.9358035Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processWormholeLog
2023-01-16T05:56:56.9358643Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:167
2023-01-16T05:56:56.9359102Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processOutcome
2023-01-16T05:56:56.9359696Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:109
2023-01-16T05:56:56.9360144Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processTx
2023-01-16T05:56:56.9360932Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:53
2023-01-16T05:56:56.9361468Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).runTxProcessor
2023-01-16T05:56:56.9362004Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/watcher.go:216
2023-01-16T05:56:56.9362397Z github.com/certusone/wormhole/node/pkg/common.RunWithScissors.func1
2023-01-16T05:56:56.9362913Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/common/scissors.go:35
2023-01-16T05:56:56.9377533Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processWormholeLog
2023-01-16T05:56:56.9378067Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:154
2023-01-16T05:56:56.9378466Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processOutcome
2023-01-16T05:56:56.9378962Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:109
2023-01-16T05:56:56.9379345Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).processTx
2023-01-16T05:56:56.9379841Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/tx_processing.go:53
2023-01-16T05:56:56.9380219Z github.com/certusone/wormhole/node/pkg/watchers/near.(*Watcher).runTxProcessor
2023-01-16T05:56:56.9380709Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/watchers/near/watcher.go:216
2023-01-16T05:56:56.9381094Z github.com/certusone/wormhole/node/pkg/common.RunWithScissors.func1
2023-01-16T05:56:56.9381569Z 	/app/actions-runner/_work/wormhole/wormhole/node/pkg/common/scissors.go:35
2023-01-16T05:56:56.9414509Z     assertions.go:237: 
2023-01-16T05:56:56.9414804Z                         
2023-01-16T05:56:56.9415145Z 	Error Trace:	watcher_test.go:114
2023-01-16T05:56:56.9415353Z         
2023-01-16T05:56:56.9415597Z 			supervisor_processor.go:147
2023-01-16T05:56:56.9415823Z         
2023-01-16T05:56:56.9416013Z 			asm_amd64.s:1594
2023-01-16T05:56:56.9416210Z         
2023-01-16T05:56:56.9417719Z 	Error:		Not equal: &common.MessagePublication{TxHash:0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545833, Timestamp:time.Date(2022, time.October, 19, 1, 28, 9, 0, time.Local), Nonce:0x48fe179, Sequence:0x107, ConsistencyLevel:0x0, EmitterChain:0xf, EmitterAddress:vaa.Address{0x14, 0x84, 0x10, 0x49, 0x9d, 0x3f, 0xcd, 0xa4, 0xdc, 0xfd, 0x68, 0xa1, 0xeb, 0xfc, 0xdd, 0xdd, 0xa1, 0x6a, 0xb2, 0x83, 0x26, 0x44, 0x8d, 0x4a, 0xae, 0x4d, 0x2f, 0x4, 0x65, 0xcd, 0xfc, 0xb7}, Payload:[]uint8{0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x42, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x1, 0x8, 0xbc, 0x32, 0xf7, 0xde, 0x18, 0xa5, 0xf6, 0xe1, 0xe7, 0xd6, 0xee, 0x7a, 0xff, 0x9f, 0x5f, 0xc8, 0x58, 0xd0, 0xd8, 0x7a, 0xc0, 0xda, 0x94, 0xdd, 0x8d, 0x2a, 0x5d, 0x26, 0x7d, 0x6b, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Unreliable:false} (expected)
2023-01-16T05:56:56.9426956Z         	        != &common.MessagePublication{TxHash:0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545832, Timestamp:time.Date(2022, time.October, 19, 1, 28, 3, 0, time.Local), Nonce:0x48fe179, Sequence:0x106, ConsistencyLevel:0x0, EmitterChain:0xf, EmitterAddress:vaa.Address{0x14, 0x84, 0x10, 0x49, 0x9d, 0x3f, 0xcd, 0xa4, 0xdc, 0xfd, 0x68, 0xa1, 0xeb, 0xfc, 0xdd, 0xdd, 0xa1, 0x6a, 0xb2, 0x83, 0x26, 0x44, 0x8d, 0x4a, 0xae, 0x4d, 0x2f, 0x4, 0x65, 0xcd, 0xfc, 0xb7}, Payload:[]uint8{0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x42, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x1, 0x8, 0xbc, 0x32, 0xf7, 0xde, 0x18, 0xa5, 0xf6, 0xe1, 0xe7, 0xd6, 0xee, 0x7a, 0xff, 0x9f, 0x5f, 0xc8, 0x58, 0xd0, 0xd8, 0x7a, 0xc0, 0xda, 0x94, 0xdd, 0x8d, 0x2a, 0x5d, 0x26, 0x7d, 0x6b, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Unreliable:false} (actual)
2023-01-16T05:56:56.9428213Z         	
2023-01-16T05:56:56.9428436Z         	Diff:
2023-01-16T05:56:56.9428803Z         	--- Expected
2023-01-16T05:56:56.9429070Z         	+++ Actual
2023-01-16T05:56:56.9429520Z         	@@ -1,6 +1,6 @@
2023-01-16T05:56:56.9430025Z         	-(*common.MessagePublication)(0xc001bb2480)({
2023-01-16T05:56:56.9430872Z         	- TxHash: (common.Hash) (len=32 cap=32) 0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545833,
2023-01-16T05:56:56.9431524Z         	- Timestamp: (time.Time) 2022-10-19 01:28:09 +0000 UTC,
2023-01-16T05:56:56.9431977Z         	+(*common.MessagePublication)(0xc00278a900)({
2023-01-16T05:56:56.9432688Z         	+ TxHash: (common.Hash) (len=32 cap=32) 0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545832,
2023-01-16T05:56:56.9433333Z         	+ Timestamp: (time.Time) 2022-10-19 01:28:03 +0000 UTC,
2023-01-16T05:56:56.9433697Z         	  Nonce: (uint32) 76538233,
2023-01-16T05:56:56.9434089Z         	- Sequence: (uint64) 263,
2023-01-16T05:56:56.9434414Z         	+ Sequence: (uint64) 262,
2023-01-16T05:56:56.9434770Z         	  ConsistencyLevel: (uint8) 0,
2023-01-16T05:56:56.9435302Z         
2023-01-16T05:56:56.9445410Z     assertions.go:237: 
2023-01-16T05:56:56.9445739Z                         
2023-01-16T05:56:56.9445996Z 	Error Trace:	watcher_test.go:114
2023-01-16T05:56:56.9446245Z         
2023-01-16T05:56:56.9446521Z 			supervisor_processor.go:147
2023-01-16T05:56:56.9446757Z         
2023-01-16T05:56:56.9446980Z 			asm_amd64.s:1594
2023-01-16T05:56:56.9447201Z         
2023-01-16T05:56:56.9449246Z 	Error:		Not equal: &common.MessagePublication{TxHash:0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545832, Timestamp:time.Date(2022, time.October, 19, 1, 28, 3, 0, time.Local), Nonce:0x48fe179, Sequence:0x106, ConsistencyLevel:0x0, EmitterChain:0xf, EmitterAddress:vaa.Address{0x14, 0x84, 0x10, 0x49, 0x9d, 0x3f, 0xcd, 0xa4, 0xdc, 0xfd, 0x68, 0xa1, 0xeb, 0xfc, 0xdd, 0xdd, 0xa1, 0x6a, 0xb2, 0x83, 0x26, 0x44, 0x8d, 0x4a, 0xae, 0x4d, 0x2f, 0x4, 0x65, 0xcd, 0xfc, 0xb7}, Payload:[]uint8{0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x42, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x1, 0x8, 0xbc, 0x32, 0xf7, 0xde, 0x18, 0xa5, 0xf6, 0xe1, 0xe7, 0xd6, 0xee, 0x7a, 0xff, 0x9f, 0x5f, 0xc8, 0x58, 0xd0, 0xd8, 0x7a, 0xc0, 0xda, 0x94, 0xdd, 0x8d, 0x2a, 0x5d, 0x26, 0x7d, 0x6b, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Unreliable:false} (expected)
2023-01-16T05:56:56.9457112Z         	        != &common.MessagePublication{TxHash:0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545833, Timestamp:time.Date(2022, time.October, 19, 1, 28, 9, 0, time.Local), Nonce:0x48fe179, Sequence:0x107, ConsistencyLevel:0x0, EmitterChain:0xf, EmitterAddress:vaa.Address{0x14, 0x84, 0x10, 0x49, 0x9d, 0x3f, 0xcd, 0xa4, 0xdc, 0xfd, 0x68, 0xa1, 0xeb, 0xfc, 0xdd, 0xdd, 0xa1, 0x6a, 0xb2, 0x83, 0x26, 0x44, 0x8d, 0x4a, 0xae, 0x4d, 0x2f, 0x4, 0x65, 0xcd, 0xfc, 0xb7}, Payload:[]uint8{0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x42, 0x40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x1, 0x8, 0xbc, 0x32, 0xf7, 0xde, 0x18, 0xa5, 0xf6, 0xe1, 0xe7, 0xd6, 0xee, 0x7a, 0xff, 0x9f, 0x5f, 0xc8, 0x58, 0xd0, 0xd8, 0x7a, 0xc0, 0xda, 0x94, 0xdd, 0x8d, 0x2a, 0x5d, 0x26, 0x7d, 0x6b, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, Unreliable:false} (actual)
2023-01-16T05:56:56.9458390Z         	
2023-01-16T05:56:56.9458627Z         	Diff:
2023-01-16T05:56:56.9458960Z         	--- Expected
2023-01-16T05:56:56.9459227Z         	+++ Actual
2023-01-16T05:56:56.9459664Z         	@@ -1,6 +1,6 @@
2023-01-16T05:56:56.9460156Z         	-(*common.MessagePublication)(0xc000b1cf30)({
2023-01-16T05:56:56.9461165Z         	- TxHash: (common.Hash) (len=32 cap=32) 0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545832,
2023-01-16T05:56:56.9461831Z         	- Timestamp: (time.Time) 2022-10-19 01:28:03 +0000 UTC,
2023-01-16T05:56:56.9462277Z         	+(*common.MessagePublication)(0xc00278a990)({
2023-01-16T05:56:56.9462968Z         	+ TxHash: (common.Hash) (len=32 cap=32) 0x5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f5f545833,
2023-01-16T05:56:56.9463616Z         	+ Timestamp: (time.Time) 2022-10-19 01:28:09 +0000 UTC,
2023-01-16T05:56:56.9463994Z         	  Nonce: (uint32) 76538233,
2023-01-16T05:56:56.9464373Z         	- Sequence: (uint64) 262,
2023-01-16T05:56:56.9464696Z         	+ Sequence: (uint64) 263,
2023-01-16T05:56:56.9465299Z         	  ConsistencyLevel: (uint8) 0,
2023-01-16T05:56:56.9465536Z         
2023-01-16T05:56:56.9465867Z --- FAIL: TestWatcherSynthetic (0.03s)

The affected PR is #2221, which is a rust SDK change that shouldn't affect anything (because the new type isn't used anywhere).

FYI @jumpsiegel
@jynnantonix
Copy link
Contributor Author

Here's a successful run of the lint-and-tests job on the same PR with no changes: https://github.com/wormhole-foundation/wormhole/actions/runs/3927088030/jobs/6714592138

@evan-gray
Copy link
Contributor

Fixed in #4044

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Maintenance - Node/Watcher/Guardian
Development

No branches or pull requests
3 participants
@jynnantonix @aadam-10 @evan-gray