algod: Migrate internal uses of v1 algod API to v2 #4684

algochoi · 2022-10-21T18:00:49Z

Summary

As part of sunsetting v1 algod APIs, we want to only use v2 algod APIs in the internal REST client. Relates to #2976.

This PR changes the client code and internal tests/tooling to use v2 models and endpoints.

There are two v1 endpoints that do not have an equivalent endpoint in v2:

/v1/account/%s/transactions
- There is a direct test in restClient_test.go that uses this endpoint - resolve/delete test when v1 API is sunsetted.
/v1/account/%s/transactions/%s
- Replaced with v2 API: /v2/account/%s/transactions/pending

Major Changes:

restClient.go, libgoal: Add v2 endpoints and use v2 models/responses. Add helper function in libgoal to parse pending transaction responses into a struct by decoding from msgpack.
algoh: Use v2 Block instead of v1 Block. Since the v2 Block API does not return the Proposer information, we get the raw block and parse it for the block + certificate.
loadgenerator (along with tests) now uses the v2 endpoint to send transactions
pingpong now uses the v2 models and endpoints
The block and transaction endpoints return different models in v1 versus v2 - related changes in response schema.
Change asset information with v2 responses - they now have pointer fields to denote whether a field is empty or not

Testing:

Modified and ran existing tests on CI
Pingpong tests show that the delta in performance with respect to master branch is <5%
Thanks to @algolucky for testing the changes on loadgenerator !

Future work

For the sake of keeping diffs small in a single PR, we should do the following in a future ticket:

Delete v1 endpoint usage in REST client and libgoal client
Delete any legacy or deprecated goal commands that use v1 functionality (legacyListParticipationKeysCommand)
Remove redundant tests that test both V1 and V2 functionality

codecov · 2022-10-27T19:37:24Z

Codecov Report

Merging #4684 (52f3b85) into master (ab87a8a) will decrease coverage by 0.06%.
The diff coverage is 7.88%.

@@            Coverage Diff             @@
##           master    #4684      +/-   ##
==========================================
- Coverage   54.49%   54.43%   -0.07%     
==========================================
  Files         407      407              
  Lines       52425    52499      +74     
==========================================
+ Hits        28569    28576       +7     
- Misses      21472    21540      +68     
+ Partials     2384     2383       -1

Impacted Files	Coverage Δ
cmd/algoh/deadman.go	`0.00% <0.00%> (ø)`
cmd/goal/account.go	`12.62% <0.00%> (ø)`
cmd/goal/application.go	`12.64% <0.00%> (ø)`
cmd/goal/asset.go	`15.58% <0.00%> (-0.67%)`	⬇️
cmd/goal/clerk.go	`8.75% <0.00%> (ø)`
cmd/goal/interact.go	`3.62% <0.00%> (ø)`
cmd/goal/ledger.go	`16.21% <0.00%> (ø)`
cmd/goal/node.go	`12.71% <0.00%> (ø)`
daemon/algod/api/server/v2/handlers.go	`0.00% <0.00%> (ø)`
daemon/algod/api/server/v2/utils.go	`13.19% <0.00%> (ø)`
... and 14 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

daemon/algod/api/server/v2/handlers.go

…ges line up with v2 response

test/scripts/e2e_subs/e2e-app-simple.sh

test/e2e-go/cli/goal/expect/goalAppAccountAddressTest.exp

libgoal/libgoal.go

* Delete extraneous function makeTestBlock inline * Add test for blockstats event details * Delete another comment

…gorand into v2-goal-migration

cmd/algoh/blockstats.go

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

cmd/goal/asset.go

daemon/algod/api/client/restClient.go

brianolson · 2022-11-02T21:54:58Z

I looked at shared/pingpong/*.go and libgoal/*.go and things look good, but a couple deprecated functions in libgoal/*.go -- could we just delete them? Assume that nothing outside this repository uses them and other things use go-algorand-sdk ?

algochoi · 2022-11-03T13:47:07Z

@brianolson Thanks a lot for confirming the pingpong files 👍

I looked at shared/pingpong/.go and libgoal/.go and things look good, but a couple deprecated functions in libgoal/*.go -- could we just delete them? Assume that nothing outside this repository uses them and other things use go-algorand-sdk ?

Yes, we can do that - I noted down things we can delete in a near future PR (deleting all deprecated libgoal/client code, tests, deprecated goal commands) to keep diffs smaller for reviewers and narrow the scope of work here.

…gorand into v2-goal-migration

test/scripts/e2e_subs/e2e-app-simple.sh

michaeldiamant

@algochoi Thanks for covering a lot of ground here - Looks ready to go from my side.

jasonpaulos · 2022-11-03T17:55:42Z

libgoal/libgoal.go

 	algod, err := c.ensureAlgodClient()
 	if err == nil {
 		var resp []byte
 		resp, err = algod.RawBlock(round)
 		if err == nil {
-			var b rpcs.EncodedBlockCert
-			err = protocol.DecodeReflect(resp, &b)
+			err = protocol.DecodeReflect(resp, &blockCert)


nit: I know you didn't write the original code, but rpcs.EncodedBlockCert has generated msgp methods, so it would benefit slightly from using protocol.Decode instead of protocol.DecodeReflect

jasonpaulos · 2022-11-03T19:00:01Z

cmd/goal/asset.go

 		}

-		reserve, err := client.AccountInformation(params.ReserveAddr)
+		reserve, err := client.AccountInformationV2(*asset.Params.Reserve, true)


I think client.AccountAssetInformation(*asset.Params.Reserve, assetID) could be used instead of client.AccountInformationV2

Followed up offline: It looks like we'd want to change the behavior so that goal asset info fails if the reserve doesn’t hold the asset - this will likely involve changing a test (test/scripts/e2e_subs/asset-misc.sh) to address this in a follow-up PR.

jasonpaulos · 2022-11-03T19:03:40Z

libgoal/transactions.go

 	} else {
 		// Fetch the current state, to fill in as a template
-		current, err := c.AccountInformation(creator)
+		current, err := c.AccountInformationV2(creator, true)


I was going to suggest this could be improved with client.AccountAssetInformation, but I don't think this side of the if statement needs to exist at all.

I think I agree - I'll follow up on this in another PR

michaeldiamant · 2022-11-03T21:16:01Z

Merging - Have agreed with @algochoi that the remaining open threads will be addressed in a prior PR before closing the story. Merging the PR to minimize merge conflicts.

algochoi added 14 commits October 12, 2022 14:30

Delete kmd API affinity

86fd07e

Delete affinity in algod

c7f6cea

Migrate GetPendingTransactions

d3277ed

Fix response field

9dca911

Migrate Assets and Accounts to V2

5c846ba

Fix linting and change pending transactions by addr

ff0fb61

Change pending transaction and params/fee for v2 api

ad0606a

Change v1 block to v2/bookkeeping block response

0e6c5ff

Merge branch 'master' into v2-goal-migration

60c2cba

Revise transaction parsing in tests

55f1662

Fix asset call to include creatables

af4ae19

Fix asset field derefs

2ec5fc2

Try reverting goal waitfortxn

292b4f6

Fix transaction encoding again

ca0eda3

algochoi added the Enhancement label Oct 25, 2022

algochoi added 6 commits October 25, 2022 12:51

Check for empty asset unit names to pass expect tests

96bf392

Convert v1 endpoints to similar v2 endpoints and change tests

08f14f1

Fix note test that was changed for debugging

f69c7d3

Change pingpong tests to V2 api

5b0e7a7

Merge branch 'master' into v2-goal-migration

1acdf98

Fix algoh mockClient tests

478dbec

algochoi added 2 commits October 27, 2022 21:23

Change send transaction API from v1 to v2

3125af5

Fix some tests regarding send transaction

5c36058

algochoi commented Oct 28, 2022

View reviewed changes

daemon/algod/api/server/v2/handlers.go Show resolved Hide resolved

Minor changes on comments and change expect test cases so error messa…

8b212cb

…ges line up with v2 response

algochoi commented Oct 31, 2022

View reviewed changes

test/scripts/e2e_subs/e2e-app-simple.sh Outdated Show resolved Hide resolved

algochoi commented Oct 31, 2022

View reviewed changes

test/e2e-go/cli/goal/expect/goalAppAccountAddressTest.exp Outdated Show resolved Hide resolved

algochoi commented Oct 31, 2022

View reviewed changes

libgoal/libgoal.go Outdated Show resolved Hide resolved

algochoi commented Oct 31, 2022

View reviewed changes

libgoal/libgoal.go Outdated Show resolved Hide resolved

michaeldiamant mentioned this pull request Nov 2, 2022

test: Use /v2/stateproofs API #4717

Closed

algochoi added 3 commits November 2, 2022 09:53

Add blockstats test to check for event details (#26)

37a0812

* Delete extraneous function makeTestBlock inline * Add test for blockstats event details * Delete another comment

Merge branch 'v2-goal-migration' of https://github.com/algochoi/go-al…

6af0e28

…gorand into v2-goal-migration

Change mockclient in algoh to make block with round number

99f2d2c

winder reviewed Nov 2, 2022

View reviewed changes

cmd/algoh/blockstats.go Outdated Show resolved Hide resolved

Update cmd/algoh/blockstats.go

bfe1a2f

Co-authored-by: Will Winder <wwinder.unh@gmail.com>

winder reviewed Nov 2, 2022

View reviewed changes

cmd/goal/asset.go Show resolved Hide resolved

winder reviewed Nov 2, 2022

View reviewed changes

daemon/algod/api/client/restClient.go Outdated Show resolved Hide resolved

algochoi mentioned this pull request Nov 2, 2022

Goal remote node config #2904

Closed

jasonpaulos and others added 4 commits November 3, 2022 10:01

Add more error formatting for REST responses (#27)

e1e2a4b

Change error messages on tests after client side parsing

bb260ab

Merge branch 'v2-goal-migration' of https://github.com/algochoi/go-al…

bb5170f

…gorand into v2-goal-migration

Change http response message in SPFullPool test

d21a9d6

michaeldiamant reviewed Nov 3, 2022

View reviewed changes

test/scripts/e2e_subs/e2e-app-simple.sh Outdated Show resolved Hide resolved

algochoi added 2 commits November 3, 2022 14:14

Revise error message check

5b680e3

Delete old error message

52f3b85

michaeldiamant approved these changes Nov 3, 2022

View reviewed changes

winder approved these changes Nov 3, 2022

View reviewed changes

jasonpaulos reviewed Nov 3, 2022

View reviewed changes

michaeldiamant merged commit 06d146b into algorand:master Nov 3, 2022

algochoi deleted the v2-goal-migration branch November 7, 2022 14:08

algochoi mentioned this pull request Nov 7, 2022

goal: Change asset commands to use AccountAssetInformation #4758

Merged

This was referenced Nov 21, 2022

go-algorand 3.12.1-beta Release PR #4820

Closed

go-algorand 3.12.1-beta Release PR #4823

Merged

onetechnical mentioned this pull request Nov 22, 2022

go-algorand v3.12.2-beta #4828

Merged

Algo-devops-service mentioned this pull request Nov 28, 2022

go-algorand 3.12.2-stable Release PR #4837

Merged

cce mentioned this pull request Apr 23, 2024

Incentives: Suspend "absentee" accounts that don't propose. #5757

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

algod: Migrate internal uses of v1 algod API to v2 #4684

algod: Migrate internal uses of v1 algod API to v2 #4684

algochoi commented Oct 21, 2022 •

edited

Loading

codecov bot commented Oct 27, 2022 •

edited

Loading

brianolson commented Nov 2, 2022 •

edited

Loading

algochoi commented Nov 3, 2022

michaeldiamant left a comment

jasonpaulos Nov 3, 2022

jasonpaulos Nov 3, 2022

algochoi Nov 3, 2022

jasonpaulos Nov 3, 2022

algochoi Nov 3, 2022

michaeldiamant commented Nov 3, 2022

algod: Migrate internal uses of v1 algod API to v2 #4684

algod: Migrate internal uses of v1 algod API to v2 #4684

Conversation

algochoi commented Oct 21, 2022 • edited Loading

Summary

Future work

codecov bot commented Oct 27, 2022 • edited Loading

Codecov Report

brianolson commented Nov 2, 2022 • edited Loading

algochoi commented Nov 3, 2022

michaeldiamant left a comment

Choose a reason for hiding this comment

jasonpaulos Nov 3, 2022

Choose a reason for hiding this comment

jasonpaulos Nov 3, 2022

Choose a reason for hiding this comment

algochoi Nov 3, 2022

Choose a reason for hiding this comment

jasonpaulos Nov 3, 2022

Choose a reason for hiding this comment

algochoi Nov 3, 2022

Choose a reason for hiding this comment

michaeldiamant commented Nov 3, 2022

algochoi commented Oct 21, 2022 •

edited

Loading

codecov bot commented Oct 27, 2022 •

edited

Loading

brianolson commented Nov 2, 2022 •

edited

Loading