Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use shared piece reader for blockstore get #1567

Closed
wants to merge 81 commits into from
Closed
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
7b745c3
feat: add local index directory
jacobheun Mar 17, 2023
2172161
refactor: merge from main into lid branch (#1339)
dirkmc Mar 29, 2023
d9923e6
merge: main to lid (#1370)
jacobheun Apr 13, 2023
2606bb0
fix: make devnet work for lid (#1375)
jacobheun Apr 15, 2023
caf39ac
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 16, 2023
55a719c
feat: fail deal if start epoch passed (#1319)
LexLuthr Apr 13, 2023
c451816
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 17, 2023
624f3ea
fix: makefile
jacobheun Apr 17, 2023
1e4fa15
fix: db migration ordering
jacobheun Apr 17, 2023
c48a8fa
fix: correct rootcid formatting
jacobheun Apr 17, 2023
80dcdc9
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 24, 2023
0a72cc7
fix: prevent accidental removal of valid sector index announcements
jacobheun Apr 26, 2023
6d50bcf
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 26, 2023
2bb534a
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 27, 2023
3d54850
chore: go mod tidy
jacobheun Apr 28, 2023
a4fdb4d
merge: remote-tracking branch 'origin/main' into nonsense/piece-store
jacobheun Apr 28, 2023
dac04ee
fix: download block (#1440)
dirkmc May 9, 2023
f5de340
LID yugabyte db impl (#1391)
dirkmc May 10, 2023
aeb34d3
script to migrate from couchbase to yugabyte (#1445)
dirkmc May 12, 2023
1316ee5
Merge branch 'main' of github.com:filecoin-project/boost into nonsens…
dirkmc May 12, 2023
2b4de7c
Change service GetIndex / AddIndex to return channel instead of array…
dirkmc May 17, 2023
3dc7e5a
local index directory: recover tool (#1410)
nonsense May 19, 2023
4ef9a56
LID benchmarking tool (#1276)
dirkmc May 19, 2023
a80b6e3
fix: failing tests due to bad merge
dirkmc May 19, 2023
e61fd62
fix: flaky TestMultipleDealsConcurrent
dirkmc May 19, 2023
93e3346
more logs
nonsense May 22, 2023
5fe0bfd
piece doctor and sector state manager refactor (#1463)
nonsense Jun 1, 2023
5e511a8
Merge from main to lid branch (#1483)
dirkmc Jun 1, 2023
a614b3d
update local index directory ui (#1477)
dirkmc Jun 5, 2023
a071a03
feat: surface indexing errors (#1490)
dirkmc Jun 5, 2023
a9953ff
feat: log panic (instead of just printing to stdout) (#1491)
dirkmc Jun 5, 2023
a5340e9
split flagged pieces into unsealed/sealed tables (#1493)
nonsense Jun 6, 2023
815ddea
refactor: remove couchbase tests (#1496)
dirkmc Jun 7, 2023
3f79cda
refactor: remove piece directory couchbase tests (#1497)
dirkmc Jun 7, 2023
4e14ce0
GraphQL resolvers for LID (#1494)
nonsense Jun 7, 2023
b8626da
fix: piece directory tests (#1498)
dirkmc Jun 7, 2023
f71b0ff
log line for only sealed sectors
nonsense Jun 8, 2023
3d64fb6
more logs
nonsense Jun 8, 2023
5d2b6ff
feat: flagged pieces (#1501)
dirkmc Jun 8, 2023
c4e0b01
check that sector has deals for unsealed sectors (#1502)
nonsense Jun 8, 2023
54edf11
rename heading
nonsense Jun 8, 2023
e9e236d
piece doctor to ignore expired/slashed deals (#1503)
nonsense Jun 9, 2023
4000480
fix pagination
nonsense Jun 9, 2023
969f934
LID landing page: add stats around Flagged and non-Flagged pieces (#1…
nonsense Jun 9, 2023
a5d29f0
feat: split flagged pieces page into flagged / flagged because unseal…
dirkmc Jun 9, 2023
4ee35f1
fix: display of no flagged pieces (#1511)
dirkmc Jun 9, 2023
2ea1ed9
disable dummy panels - block stats; deal data (#1510)
nonsense Jun 12, 2023
aaf67ec
fix unsealed field in flagged piece (#1515)
nonsense Jun 12, 2023
b2bf8ce
Merge branch 'main' into nonsense/piece-store
Jun 12, 2023
f73cf3d
update ffi
Jun 12, 2023
5caa93b
fix main merge issue
Jun 12, 2023
cdc40ec
fix go mod
Jun 12, 2023
1edb02e
Add info boxes on LID UI page (#1516)
dirkmc Jun 12, 2023
b256c2e
feat: replace migrate couchbase command with migrate yugavbyte (#1518)
dirkmc Jun 13, 2023
966a385
remove redundant makefile (#1519)
nonsense Jun 13, 2023
2a7ed2c
merge nonsense/piece-store to release/v2
nonsense Jun 14, 2023
0859578
Merge branch 'main' into release/v2
nonsense Jun 14, 2023
7ef2aa8
fix: inspect page - dont try to fetch root cid (#1525)
dirkmc Jun 14, 2023
33ba7de
feat: add send epoch, time, elapsed epoch and elapsed time for each m…
LexLuthr Jun 15, 2023
8a34ce9
add logs
nonsense Jun 15, 2023
55c086b
fix devnet: use ws instead of http to connect to boostd-data
nonsense Jun 15, 2023
6b9ffed
feat: make legacy deals optional (#1524)
LexLuthr Jun 20, 2023
248dae4
refactor: remove couchbase implementation (#1535)
dirkmc Jun 23, 2023
f1cec09
Update lotus and boxo versions (#1466) (#1537)
dirkmc Jun 26, 2023
d2a7df9
feat: update boost-gfm to v1.26.7 (#1538)
dirkmc Jun 26, 2023
6d812ee
fix: piece doctor tests (#1540)
dirkmc Jun 27, 2023
ecbc081
refactor: build indexes for legacy deals (#1539)
dirkmc Jun 28, 2023
01e731c
feat: http index announcements (#1418)
jacobheun Jun 28, 2023
7263642
feat: check unseal status of piece through both apis (#1548)
dirkmc Jun 30, 2023
14a454d
fix: metrics and Grafana (#1546)
LexLuthr Jun 30, 2023
7409f90
fix: add missing PieceDeal (PieceCid) index (#1551)
dirkmc Jul 3, 2023
16c6310
fix: iterate all deals to index piece (#1549)
LexLuthr Jul 3, 2023
497472d
refactor: separate yugabyte / leveldb tests for easier local testing …
dirkmc Jul 3, 2023
5d903fa
feat: refactor mpool page in UI (#1530)
LexLuthr Jul 7, 2023
540b41b
migrate config to v5 (#1560)
LexLuthr Jul 11, 2023
f69945e
chore: release v2.0.0-rc1 (#1561)
LexLuthr Jul 11, 2023
01a5a35
Upgrade to index-provider v0.13.4 (#1559)
masih Jul 11, 2023
d715f70
feat: add IPNI itest (#1563)
LexLuthr Jul 12, 2023
6e858b9
IPNI UX (#1562)
dirkmc Jul 13, 2023
8c9f8dc
feat: use shared piece reader for blockstore get
dirkmc Jul 13, 2023
0731f5b
fix: booster-bitswap client - extract blocks from identity cids
dirkmc Jul 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
version: 2.1
orbs:
go: gotest/tools@0.0.13
go: gotest/tools@0.0.14

executors:
golang:
@@ -83,6 +83,19 @@ jobs:
paths:
- linux

lid-docker-compose:
description: 'Run LID integration tests'
machine:
image: ubuntu-2004:202104-01
resource_class: xlarge
steps:
- checkout
- run:
name: local index directory docker compose tests
command: |
set -x
make test-lid

test:
description: |
Run go tests
@@ -98,6 +111,10 @@ jobs:
type: string
default: "./..."
description: Import paths of packages to be tested.
cwd:
type: string
default: "."
description: Directory in which to run the test command
suite:
type: string
default: unit
@@ -110,9 +127,16 @@ jobs:
command: make boostci
no_output_timeout: 30m
- download-params
# Note: setup_remote_docker takes about 10s and it's only needed for
# local index directory tests so it may be worth refactoring so that it's
# only included for local index directory
- setup_remote_docker:
version: 20.10.18
docker_layer_caching: true
- run:
name: go test
command: |
cd << parameters.cwd >>
go test \
<< parameters.go-test-flags >> \
<< parameters.target >>
@@ -306,7 +330,20 @@ workflows:
suite: itest-markets_v1_offline_deal
target: "./itests/markets_v1_offline_deal_test.go"

- test:
name: test-itest-disabled_markets_v1_deal
suite: itest-disabled_markets_v1_deal
target: "./itests/disabled_markets_v1_deal_test.go"

- test:
name: test-all
suite: all
target: "`go list ./... | grep -v boost/itests`"

- test:
name: test-itest-ipni
suite: itest-ipni
target: "./itests/ipni_publish_test.go"

- lid-docker-compose

3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
/boost
/boostx
/boostd
/boostd-data
/migrate-lid
/devnet
/booster-http
/booster-bitswap
/docgen-md
/docgen-openrpc
extern/filecoin-ffi/rust/target
extern/boostd-data/boostd-data
**/*.a
**/*.pc
/**/*/.DS_STORE
34 changes: 29 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
@@ -7,6 +7,7 @@ unexport GOFLAGS

GOCC?=go

ARCH?=$(shell arch)
GOVERSION:=$(shell $(GOCC) version | tr ' ' '\n' | grep go1 | sed 's/^go//' | awk -F. '{printf "%d%03d%03d", $$1, $$2, $$3}')
ifeq ($(shell expr $(GOVERSION) \< 1016000), 1)
$(warning Your Golang version is go$(shell expr $(GOVERSION) / 1000000).$(shell expr $(GOVERSION) % 1000000 / 1000).$(shell expr $(GOVERSION) % 1000))
@@ -79,19 +80,35 @@ calibnet-go: build-go
deps: $(BUILD_DEPS)
.PHONY: deps

migrate-lid: $(BUILD_DEPS)
rm -f migrate-lid
$(GOCC) build $(GOFLAGS) -o migrate-lid ./cmd/migrate-lid
.PHONY: migrate-lid
BINS+=migrate-lid

boostx: $(BUILD_DEPS)
rm -f boostx
$(GOCC) build $(GOFLAGS) -o boostx ./cmd/boostx
.PHONY: boostx
BINS+=boostx

boost: $(BUILD_DEPS)
rm -f boost boostd boostx
rm -f boost
$(GOCC) build $(GOFLAGS) -o boost ./cmd/boost
$(GOCC) build $(GOFLAGS) -o boostx ./cmd/boostx
$(GOCC) build $(GOFLAGS) -o boostd ./cmd/boostd
.PHONY: boost
BINS+=boost boostx boostd
BINS+=boost

boostd: $(BUILD_DEPS)
rm -f boostd
$(GOCC) build $(GOFLAGS) -o boostd ./cmd/boostd
.PHONY: boostd
BINS+=boostd

boostd-data:
$(MAKE) -C ./extern/boostd-data
install -C ./extern/boostd-data/boostd-data ./boostd-data
.PHONY: boostd-data
BINS+=boostd-data

booster-http: $(BUILD_DEPS)
rm -f booster-http
@@ -126,7 +143,7 @@ update-react: validate-node-version
npm run --prefix react build
.PHONY: react

build-go: boost devnet
build-go: boost boostd boostx boostd-data booster-http booster-bitswap devnet migrate-lid
.PHONY: build-go

build: react build-go
@@ -141,6 +158,10 @@ install-boost:
install -C ./boost /usr/local/bin/boost
install -C ./boostd /usr/local/bin/boostd
install -C ./boostx /usr/local/bin/boostx
install -C ./boostd-data /usr/local/bin/boostd-data
install -C ./booster-http /usr/local/bin/booster-http
install -C ./booster-bitswap /usr/local/bin/booster-bitswap
install -C ./migrate-lid /usr/local/bin/migrate-lid

install-devnet:
install -C ./devnet /usr/local/bin/devnet
@@ -252,6 +273,9 @@ docker/all: $(lotus_build_cmd) docker/boost docker/booster-http docker/booster-b
docker/lotus docker/lotus-miner
.PHONY: docker/all

test-lid:
cd ./extern/boostd-data && ARCH=$(ARCH) docker-compose up --build --exit-code-from go-tests

devnet/up:
rm -rf ./docker/devnet/data && docker compose -f ./docker/devnet/docker-compose.yaml up -d

11 changes: 4 additions & 7 deletions api/api.go
Original file line number Diff line number Diff line change
@@ -3,7 +3,6 @@ package api
import (
"context"

"github.com/filecoin-project/boost-gfm/piecestore"
"github.com/filecoin-project/boost-gfm/retrievalmarket"
"github.com/filecoin-project/boost-gfm/storagemarket"
smtypes "github.com/filecoin-project/boost/storagemarket/types"
@@ -35,6 +34,7 @@ type Boost interface {

// MethodGroup: Boost
BoostIndexerAnnounceAllDeals(ctx context.Context) error //perm:admin
BoostIndexerListMultihashes(ctx context.Context, proposalCid cid.Cid) ([]multihash.Multihash, error) //perm:admin
BoostIndexerAnnounceLatest(ctx context.Context) (cid.Cid, error) //perm:admin
BoostIndexerAnnounceLatestHttp(ctx context.Context, urls []string) (cid.Cid, error) //perm:admin
BoostOfflineDealWithData(ctx context.Context, dealUuid uuid.UUID, filePath string, delAfterImport bool) (*ProviderDealRejectionInfo, error) //perm:admin
@@ -56,6 +56,9 @@ type Boost interface {
BlockstoreHas(ctx context.Context, c cid.Cid) (bool, error) //perm:read
BlockstoreGetSize(ctx context.Context, c cid.Cid) (int, error) //perm:read

// MethodGroup: PieceDirectory
PdBuildIndexForPieceCid(ctx context.Context, piececid cid.Cid) error //perm:admin

// RuntimeSubsystems returns the subsystems that are enabled
// in this instance.
RuntimeSubsystems(ctx context.Context) (lapi.MinerSubsystems, error) //perm:read
@@ -75,12 +78,6 @@ type Boost interface {
MarketPendingDeals(ctx context.Context) (lapi.PendingDealInfo, error) //perm:write
SectorsRefs(context.Context) (map[string][]lapi.SealedRef, error) //perm:read

PiecesListPieces(ctx context.Context) ([]cid.Cid, error) //perm:read
PiecesListCidInfos(ctx context.Context) ([]cid.Cid, error) //perm:read
PiecesGetPieceInfo(ctx context.Context, pieceCid cid.Cid) (*piecestore.PieceInfo, error) //perm:read
PiecesGetCIDInfo(ctx context.Context, payloadCid cid.Cid) (*piecestore.CIDInfo, error) //perm:read
PiecesGetMaxOffset(ctx context.Context, pieceCid cid.Cid) (uint64, error) //perm:read

// MethodGroup: Actor
ActorSectorSize(context.Context, address.Address) (abi.SectorSize, error) //perm:read

80 changes: 20 additions & 60 deletions api/proxy_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file modified build/openrpc/boost.json.gz
Binary file not shown.
2 changes: 1 addition & 1 deletion build/version.go
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ package build

var CurrentCommit string

const BuildVersion = "1.7.3"
const BuildVersion = "2.0.0-rc1"

func UserVersion() string {
return BuildVersion + CurrentCommit
8 changes: 4 additions & 4 deletions car/car_offset_writer.go
Original file line number Diff line number Diff line change
@@ -6,12 +6,12 @@ import (
"fmt"
"io"

"github.com/ipfs/go-blockservice"
"github.com/ipfs/boxo/blockservice"
blockstore "github.com/ipfs/boxo/blockstore"
offline "github.com/ipfs/boxo/exchange/offline"
"github.com/ipfs/boxo/ipld/merkledag"
"github.com/ipfs/go-cid"
blockstore "github.com/ipfs/go-ipfs-blockstore"
offline "github.com/ipfs/go-ipfs-exchange-offline"
format "github.com/ipfs/go-ipld-format"
"github.com/ipfs/go-merkledag"
"github.com/ipld/go-car"
"github.com/ipld/go-car/util"
)
16 changes: 8 additions & 8 deletions car/car_offset_writer_test.go
Original file line number Diff line number Diff line change
@@ -7,18 +7,18 @@ import (
"math/rand"
"testing"

"github.com/ipfs/go-blockservice"
"github.com/ipfs/boxo/blockservice"
bstore "github.com/ipfs/boxo/blockstore"
chunk "github.com/ipfs/boxo/chunker"
"github.com/ipfs/boxo/ipld/merkledag"
merkledagpb "github.com/ipfs/boxo/ipld/merkledag/pb"
"github.com/ipfs/boxo/ipld/unixfs/importer/balanced"
"github.com/ipfs/boxo/ipld/unixfs/importer/helpers"
blocks "github.com/ipfs/go-block-format"
"github.com/ipfs/go-cidutil"
"github.com/ipfs/go-datastore"
dss "github.com/ipfs/go-datastore/sync"
bstore "github.com/ipfs/go-ipfs-blockstore"
chunk "github.com/ipfs/go-ipfs-chunker"
format "github.com/ipfs/go-ipld-format"
"github.com/ipfs/go-libipfs/blocks"
"github.com/ipfs/go-merkledag"
merkledagpb "github.com/ipfs/go-merkledag/pb"
"github.com/ipfs/go-unixfs/importer/balanced"
"github.com/ipfs/go-unixfs/importer/helpers"
"github.com/ipld/go-car"
mh "github.com/multiformats/go-multihash"
"github.com/stretchr/testify/require"
6 changes: 3 additions & 3 deletions car/car_reader_seeker_test.go
Original file line number Diff line number Diff line change
@@ -8,11 +8,11 @@ import (
"math/rand"
"testing"

"github.com/ipfs/go-blockservice"
"github.com/ipfs/boxo/blockservice"
bstore "github.com/ipfs/boxo/blockstore"
"github.com/ipfs/boxo/ipld/merkledag"
"github.com/ipfs/go-datastore"
dss "github.com/ipfs/go-datastore/sync"
bstore "github.com/ipfs/go-ipfs-blockstore"
"github.com/ipfs/go-merkledag"
"github.com/stretchr/testify/require"
)

Loading