feat: add pkg extractor #5

leohhhn · 2023-10-04T16:26:14Z

Description

This PR adds a Go program that extracts Gno packages from log files, later to be uploaded to a separate archive repo.

leohhhn · 2023-10-04T16:56:25Z

Current state of the extractor is the following:

Discovers .log files with root at LOGS_DIR, set in extractor/.env
Unmarshals log entries and extracts packages into EXTRACTION_DIR, keeping the path structure same as GnoLand
Adds metadata file to each package folder, which includes creator of package, the deposit upon package creation, and is a place for useful info in the future (i.e. timestamp, block height, chain-id, etc.)

Blockers:

How and where do we upload the archived packages? Should we make a gno-bot GH account for this?

TODO:

Write a script to upload the extracted packages to an archive repo

Down the line:

Modify extractor script to match Milos' txsync & indexer efforts

extractor/extractor.go

extractor/helpers.go

extractor/.gitignore

extractor/README.md

extractor/main.go

thehowl

yeah mostly fine. just trying to run it myself but I think it's good to merge aside from some minor changes.

extractor/main.go

zivkovicmilos

Looks good, but I've left some comments that need to be addressed 💯

extractor/main_test.go

zivkovicmilos

I've left a second round of reviews, please see the comments and we can go from there 🙏

extractor/main.go

extractor/main_test.go

ajnavarro

Just a bunch of nits, but I found a swallowed error that we have to fix!

extractor/main.go

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

extractor/main.go

This reverts commit 872fd12.

zivkovicmilos

Looks great now 💯

Thank you for looking into the conversations

leohhhn added 3 commits October 4, 2023 18:19

add extractor

da5f8d6

file refactor

596f696

added walker log file discovery

af29063

leohhhn requested a review from zivkovicmilos October 4, 2023 16:56

add whitespace

2fbdfbd

zivkovicmilos requested a review from ajnavarro October 4, 2023 17:28

thehowl reviewed Oct 5, 2023

View reviewed changes

extractor/extractor.go Outdated Show resolved Hide resolved

leohhhn self-assigned this Oct 5, 2023

leohhhn mentioned this pull request Oct 6, 2023

Indexer for Extracting Source Code from Testnet Blocks to GitHub gnolang/hackerspace#39

Open

zivkovicmilos reviewed Oct 9, 2023

View reviewed changes

extractor/helpers.go Outdated Show resolved Hide resolved

zivkovicmilos reviewed Oct 9, 2023

View reviewed changes

extractor/.gitignore Outdated Show resolved Hide resolved

zivkovicmilos reviewed Oct 9, 2023

View reviewed changes

extractor/README.md Outdated Show resolved Hide resolved

leohhhn added 3 commits October 12, 2023 00:05

WIP full refactor in progress

48ab9ff

fixed decoding

ba70aac

add tests, updated README

485dfb4

leohhhn marked this pull request as ready for review October 20, 2023 15:06

leohhhn requested review from thehowl and zivkovicmilos October 20, 2023 15:06

leohhhn mentioned this pull request Oct 20, 2023

Consider making the source extraction straight in this repo. #2

Closed

thehowl reviewed Oct 26, 2023

View reviewed changes

extractor/main.go Outdated Show resolved Hide resolved

thehowl reviewed Oct 26, 2023

View reviewed changes

extractor/main.go Outdated Show resolved Hide resolved

removed unused errors, removed args

1de5496

thehowl approved these changes Oct 27, 2023

View reviewed changes

zivkovicmilos requested changes Oct 27, 2023

View reviewed changes

extractor/main_test.go Outdated Show resolved Hide resolved

extractor/main_test.go Show resolved Hide resolved

extractor/main_test.go Outdated Show resolved Hide resolved

extractor/main_test.go Show resolved Hide resolved

extractor/main_test.go Outdated Show resolved Hide resolved

thehowl reviewed Oct 27, 2023

View reviewed changes

extractor/main_test.go Outdated Show resolved Hide resolved

test fixes

7bedfa3

leohhhn requested a review from zivkovicmilos October 27, 2023 21:39

moul approved these changes Nov 8, 2023

View reviewed changes

zivkovicmilos requested changes Nov 30, 2023

View reviewed changes

leohhhn added 2 commits December 13, 2023 16:52

replace t.Fatalf with require

25589bd

move global variables to smaller scopes

20dd043

leohhhn requested a review from zivkovicmilos December 13, 2023 16:20

ajnavarro requested changes Dec 18, 2023

View reviewed changes

zivkovicmilos mentioned this pull request Dec 20, 2023

Minutes: Core Staff Weekly Syncs [every Monday] gnolang/meetings#36

Open

leohhhn and others added 9 commits December 21, 2023 08:51

rename error

36253a3

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

remove unused anon vars

b56850a

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

edit comment

0932074

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

rename error

55c4fa0

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

update map storage for paths

afc7487

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

update map storage for paths gnolang#2

3a86407

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

add conversion check

01ca7fb

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

add error check

6b6349c

Co-authored-by: Antonio Navarro Perez <antnavper@gmail.com>

format & handle errors

928801c

leohhhn requested a review from ajnavarro December 21, 2023 08:04

move cleanup above err check

872fd12

ajnavarro requested changes Dec 21, 2023

View reviewed changes

extractor/main.go Outdated Show resolved Hide resolved

ajnavarro self-requested a review December 21, 2023 09:14

ajnavarro approved these changes Dec 21, 2023

View reviewed changes

leohhhn added 7 commits December 21, 2023 11:57

Revert "move cleanup above err check"

f20d8f3

This reverts commit 872fd12.

swap arg order

65bb9c9

wip tests

b24b21a

added more tests

03fa2de

modify source path

7638489

parallel test failing, save

1ed77fc

fix broken tests

3965302

zivkovicmilos approved these changes Dec 29, 2023

View reviewed changes

leohhhn merged commit ca7cf22 into gnolang:main Dec 29, 2023

leohhhn deleted the feat/extractor branch December 29, 2023 15:29

leohhhn mentioned this pull request Jan 9, 2024

Make the fetching script and this repo more reliable #1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add pkg extractor #5

feat: add pkg extractor #5

leohhhn commented Oct 4, 2023

leohhhn commented Oct 4, 2023 •

edited

Loading

thehowl left a comment

zivkovicmilos left a comment

zivkovicmilos left a comment

ajnavarro left a comment

zivkovicmilos left a comment

feat: add pkg extractor #5

feat: add pkg extractor #5

Conversation

leohhhn commented Oct 4, 2023

Description

leohhhn commented Oct 4, 2023 • edited Loading

thehowl left a comment

Choose a reason for hiding this comment

zivkovicmilos left a comment

Choose a reason for hiding this comment

zivkovicmilos left a comment

Choose a reason for hiding this comment

ajnavarro left a comment

Choose a reason for hiding this comment

zivkovicmilos left a comment

Choose a reason for hiding this comment

leohhhn commented Oct 4, 2023 •

edited

Loading