Use the latest version of the Stellar ingestion library #122
Conversation
internal/input/ledger_range.go
Outdated
| @@ -2,9 +2,9 @@ package input | |||
|
|
|||
| import ( | |||
| "fmt" | |||
| "github.com/stellar/go/historyarchive" | |||
internal/input/operations.go
Outdated
| "github.com/stellar/go/ingest" | ||
| "github.com/stellar/go/xdr" | ||
| "github.com/stellar/stellar-etl/internal/utils" | ||
| "io" |
There was a problem hiding this comment.
| "github.com/stellar/go/ingest" | |
| "github.com/stellar/go/xdr" | |
| "github.com/stellar/stellar-etl/internal/utils" | |
| "io" | |
| "io" | |
| "github.com/stellar/go/ingest" | |
| "github.com/stellar/go/xdr" | |
| "github.com/stellar/stellar-etl/internal/utils" |
internal/utils/main.go
Outdated
| "github.com/stellar/go/historyarchive" | ||
| "github.com/stellar/go/ingest/ledgerbackend" |
internal/utils/main.go
Outdated
| return historyArchiveBackend{client: client, ledgers: ledgers}, nil | ||
| } | ||
|
|
||
| func CreateHistoryArchiveClient() (historyarchive.ArchiveInterface, error) { | ||
| archiveStellarURL := "http://history.stellar.org/prd/core-live/core_live_001" |
There was a problem hiding this comment.
Not in a scope of this PR but this should be https because tx data is not signed.
| archiveStellarURL := "http://history.stellar.org/prd/core-live/core_live_001" | |
| archiveStellarURL := "https://history.stellar.org/prd/core-live/core_live_001" |
There was a problem hiding this comment.
Likewise not in scope, but using the new historyarchive.NewArchivePool (stellar/go#3402) with multiple URLs here could be useful here for resilience.
tamirms
force-pushed
the
ingestion-library
branch
from
fd84ca1 to
477ac88
Compare
| } | ||
|
|
||
| err = validateLedgerRange(start, end, latestNum) | ||
| backend, err := utils.CreateBackend(start, end) |
There was a problem hiding this comment.
does a lot of the validation happen automatically now? @tamirms
There was a problem hiding this comment.
I noticed that a lot of the checks where repeated every time a backend was created so I thought it would be better to bundle the checks in utils.CreateBackend() to avoid repeated code
There was a problem hiding this comment.
Thank you for making this change!
From what I understand, this is mostly a refactor of patterns into the utility function and not much in the API usage for history archives has changed other than using the interface rather than the concrete backend object in these structs. Is that right?
Am I missing anything? It's possible that I missed the main area that contained the functional change.
Also, were you able to test it by running one of the commands? If so, did that give expected results?
That's right, we've named some package names and did some structural refactoring. Otherwise, the functionality should be identical to what was provided before.
I have not but I will try to see if it's possible to run the commands locally |
|
Great, if we can run locally to test it would be great. Otherwise good to merge on my end 🎉 |
|
I have run the History Archive Commands on some sample inputs and compared the output with running the same commands using the stellar-etl code in the master branch. The outputs match! So, I will merge the PR |
I have updated the stellar-etl repo to use the non experimental ingestion library which we have just published:
https://github.com/stellar/go/releases/tag/stellar-ingest-v1.0.0
The API for the ingestion library should be somewhat stable now. So do not expect many breaking changes in the future.