I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

ChristianOConnor · 2021-08-21T20:50:17Z

I made a hot storage deal and I still can't access the content on IPFS. The content that I put into this deal is the directory of a static website. When I double click the index.html file in Windows Explorer, it opens perfectly in Firefox so I know the website works. I tried to access it with https://ipfs.io/ipfs/<CID> and https://<CID>.ipfs.dweb.link but they both failed. I did the deal perfectly. I did it exactly to spec for a hot storage deal which I believe is the way to get your content on IPFS. Here's what I did to make my storage deal.

My config file aka config-new.json looks like this.

{
  "hot":  {
    "enabled":  true,
    "allowUnfreeze":  true,
    "unfreezeMaxPrice":  "100000000000000000",
    "ipfs":  {
      "addTimeout":  "900"
    }
  },
  "cold":  {
    "enabled":  true,
    "filecoin":  {
      "replicationFactor":  "2",
      "dealMinDuration":  "518400",
      "excludedMiners":  [],
      "trustedMiners":  ["<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>", "<Miner Id>",],
      "countryCodes":  [],
      "renew":  {
        "enabled":  false,
        "threshold":  "0"
      },
      "address":  "<my filecoin address>",
      "maxPrice":  "100000000000000000",
      "fastRetrieval":  true,
      "dealStartOffset":  "8640",
      "verifiedDeal":  false
    }
  },
  "repairable":  false
}

git clone https://github.com/textileio/powergate.git
cd powergate/
cd docker/
nano docker-compose.yaml where I added "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" between lines 32 and 33.
make up
Waited for the node to finish importing and then syncing.
^C then make down then deleted the line "["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"]" from docker-compose.yaml
make up
Now that the node was running I typed cd .. so I was in the repo's root directory, then make install-pow
with the pow command in my GOPATH I typed pow to make sure pow was linked fine to powd. It was.
pow admin users create
copied the token and ran export POW_TOKEN=<token copied to here>
Then pow wallet addrs and funded the address
I went to the directory behind the folder of my static website which is about 5GB in size.
I typed pow data stage <my-static-site-folder>
After it was finished staging and printed out the CID I typed pow config apply --watch <CID> -c custom-080121-hot-config.json waited a long time and it worked! I had a Filecoin deal that was duplicated across 2 miners.

So I now have a storage deal but the directory still isn't on IPFS. Why won't it work?

The text was updated successfully, but these errors were encountered:

ChristianOConnor · 2021-08-23T20:42:42Z

@jsign

ChristianOConnor · 2021-08-25T11:08:04Z

@jsign @andrew This is still a pressing issue, and at this point, I think it's a bug in the code and not something that I did. My files still cannot be found on IPFS. Please someone explain to me what the problem is. What is going on?

andrewxhill · 2021-08-25T15:17:53Z

it sounds like you are running this in a way where your IPFS node isn't accessible to the IPFS network.

here is a good discussion w/ different checks & solutions https://discuss.ipfs.io/t/added-a-file-but-its-not-accessible-on-gateway/4667/5

ChristianOConnor · 2021-08-30T09:44:54Z

it sounds like you are running this in a way where your IPFS node isn't accessible to the IPFS network.

here is a good discussion w/ different checks & solutions https://discuss.ipfs.io/t/added-a-file-but-its-not-accessible-on-gateway/4667/5

Awesome thanks for the info I'm spinning up my Powergate node now to troubleshoot.
Still though, as I'm waiting for my Powergate node to sync, doesn't it not really matter if IPFS can't connect? Doesn't it deploy to IPFS from the Filecoin miner? I though hot storage meant that Filecoin stores the file and deploys it to IPFS.

ChristianOConnor · 2021-08-30T10:35:28Z

Now I can't access my install anymore. I had to move my install to another computer and now the lotus node just says this:

lotus_1       | 2021-08-30T08:55:51.357Z        INFO    backupds        backupds/log.go:125         opening log     {"file": "/data/.lotus/kvlog/metadata/1630310485.log.cbor"}
lotus_1       | ERROR: initializing node: starting node: could not build arguments for function     "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/    lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".    makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.RawHost: could not     build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12):     could not build value group []config.Option[group="libp2p"]: could not build arguments for     function "github.com/filecoin-project/lotus/node/modules/lp2p".ConnGaterOption (/lotus/node/    modules/lp2p/conngater.go:14): failed to build *conngater.BasicConnectionGater: could not build     arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed     to build dtypes.MetadataDS: received non-nil error from function "reflect".makeFuncStub (/usr/    local/go/src/reflect/asm_amd64.s:12): opening backupds: opening log: loghead log file doesn't     match, opening /data/.lotus/kvlog/metadata/1630310485.log.cbor, expected 1630206194.log.cbor
lotus_1       | 
mainnet_lotus_1 exited with code 1

Every time I try to fix a problem with this I just get more problems lol. Does anyone know what this error means?

jsign · 2021-08-30T13:25:29Z

The Lotus node can be somewhat complex to operate and I've deal (and still do) with some errors like that one. To be clear, that's not particularly related to Powergate but is more like a Lotus question.

For that error you can:

Stop lotus
Remove /data/.lotus/kvlog/metadata/1630310485.log.cbor, this file should be safe to delete. (you'll have to believe me on this; might worth making a backup)
Start again lotus.

ChristianOConnor · 2021-08-30T20:37:49Z

The Lotus node can be somewhat complex to operate and I've deal (and still do) with some errors like that one. To be clear, that's not particularly related to Powergate but is more like a Lotus question.

For that error you can:
1. Stop lotus

2. Remove `/data/.lotus/kvlog/metadata/1630310485.log.cbor`, this file should be safe to delete. (you'll have to believe me on this; might worth making a backup)

3. Start again lotus.

Okay I did what you said. I went to the docker directory of the repo then make up. I opened a new terminal and went into the lotus node with docker exec -it mainnet_lotus_1 /bin/sh then real quick ran the command rm /data/.lotus/kvlog/metadata/1630310485.log.cbor. Then I exited out of the docker container.

Now I get the following error:

lotus_1       | 2021-08-30T20:33:16.287Z        ERROR   statetree       state/statetree.go:266  failed to load state tree: failed to load hamt node: blockstore: block not found

I restarted the node and now I got this more complete error

lotus_1       | 2021-08-30T20:40:02.531Z        WARN    modules modules/services.go:72  failed to     say hello     {"error": "load state tree: failed to load state tree: failed to load hamt node:     blockstore: block not found", "errorVerbose": "load state tree:\n    github.com/filecoin-project/    lotus/chain/store.(*ChainStore).Weight\n        /lotus/chain/store/weight.go:34\n  - failed to     load state tree:\n    github.com/filecoin-project/lotus/chain/state.LoadStateTree\n        /lotus/    chain/state/statetree.go:267\n  - failed to load hamt node:\n    github.com/filecoin-project/    specs-actors/actors/util/adt.AsMap\n        /go/pkg/mod/github.com/filecoin-project/    specs-actors@v0.9.13/actors/util/adt/map.go:41\n  - blockstore: block not found", "peer":     "12D3KooWNtDE3J7pdtAzq6Rp3d8eHnJ663H6EpUYxr3xBaMTJRhJ", "supported": ["/ipfs/id/1.0.0", "/meshsub/    1.0.0", "/fil/chain/xchg/0.0.1", "/chain/ipfs/bitswap/1.0.0", "/p2p/id/delta/1.0.0", "/floodsub/1.    0.0", "/chain/ipfs/bitswap/1.1.0", "/fil/datatransfer/1.1.0", "/fil/datatransfer/1.0.0", "/ipfs/    ping/1.0.0", "/libp2p/autonat/1.0.0", "/fil/hello/1.0.0", "/fil/sync/blk/0.0.1", "/chain/ipfs/    bitswap", "/ipfs/id/push/1.0.0", "/meshsub/1.1.0", "/chain/ipfs/bitswap/1.2.0", "/ipfs/graphsync/1.    0.0", "/fil/kad/testnetnet/kad/1.0.0"], "agent": "lotus-1.10.1+mainnet+git.d6f3d3c36"}

jsign · 2021-08-30T20:42:57Z

That's just a warning which is pretty common.
How are you evaluating if things are working properly or not?

Like, does: docker exec -it testnet_lotus_1 lotus sync wait say that you're syncing and all is OK?

ChristianOConnor · 2021-08-30T22:11:49Z

Oh nice, yeah It looks like it's working. This is the result.

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 10; Base: 1064014; Target: 1068500 (diff: 4486)
State: fetching messages; Current Epoch: 1064013; Todo: 4487
Validated 0 messages (0 per second)

ChristianOConnor · 2021-08-30T22:23:27Z

Wtf... now the messages are making no sense

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 15; Base: 1064014; Target: 1068523 (diff: 4509)
State: header sync; Current Epoch: 1068523; Todo: 0
Validated 0 messages (0 per second)

the diff is higher but Todo is 0????

ChristianOConnor · 2021-08-30T22:24:28Z

Now the number is going up

docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 15; Base: 1064014; Target: 1068523 (diff: 4509)
State: fetching messages; Current Epoch: 1064013; Todo: 4510
Validated 0 messages (0 per second)

jsign · 2021-08-30T23:19:21Z

Diff should be going down. Considering your 4k TODO you definitely want to sync from a snapshot again.
https://docs.textile.io/powergate/mainnet/#recover-or-prune-your-existing-lotus-node-with-a-snapshot

After the import you'll be ~300 blocks await from the tip.

ChristianOConnor · 2021-08-31T02:58:21Z

Nope, still not working. I re-sunk the lotus node following your link. It still won't sync. The diff just keeps growing and not syncing a single block.

Here it is at 10:47PM EST
docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 3; Base: 1064014; Target: 1069049 (diff: 5035)
State: fetching messages; Current Epoch: 1064013; Todo: 5036
Validated 0 messages (0 per second)

And here it is at 10:57PM EST
docker exec -it mainnet_lotus_1 lotus sync wait
Worker: 4; Base: 1064014; Target: 1069063 (diff: 5049)
State: error; Current Epoch: 1064013; Todo: 5050
Validated 0 messages (0 per second)

I did the re-sync exactly as your link said.

make down
docker run --rm -v mainnet_powergate-lotus:/data ubuntu rm -rf /data/.lotus/datastore/chain
add command: ["lotus", "daemon", "--import-snapshot", "https://fil-chain-snapshots-fallback.s3.amazonaws.com/mainnet/minimal_finality_stateroots_latest.car"] between line 32 and 33 of docker-compose.yaml.
make up
wait for the sync to finish (and I waited an extra 10 - 20 minutes after the blockchain was downloaded to make sure the sync completed).
^C the docker stack
make down
docker run --rm -v mainnet_powergate-lotus:/data ubuntu rm -rf /data/.lotus/datastore/chain
deleted line 33 added in step 3 to the docker-compose.yaml file
make up

And it's still not syncing. the Diff just keeps getting bigger... wtf... ????!!!!

ChristianOConnor · 2021-08-31T04:23:48Z

I tried the 10 steps again and now I get this error at step 4:
ERROR: initializing node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.NetworkName: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.AfterGenesisSet: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): getting genesis block failed: blockstore: block not found

jsign · 2021-08-31T12:14:20Z

Most probably some step above wasn't executed correctly. (e.g: yesterday I pruned some nodes doing exactly that).

Did step 4 download a 40GiB file? Time between 4 and 5 should be at least 20 or 30 min today if you have very good bandwidth.
After doing step 2, you can docker run inside the container after and check that /data/.lotus/datastore/chain folder doesn't exist (or is very small if was boostraped again when you started the container). If this isn't the case, your Lotus volume might not be mainnet_powergate-lotus.

ChristianOConnor · 2021-08-31T21:32:17Z

Most probably some step above wasn't executed correctly. (e.g: yesterday I pruned some nodes doing exactly that).

* Did step 4 download a 40GiB file? Time between 4 and 5 should be at least 20 or 30 min today if you have very good bandwidth.

* After doing step 2, you can `docker run` inside the container after and check that `/data/.lotus/datastore/chain` folder doesn't exist (or is very small if was boostraped again when you started the container). If this isn't the case, your Lotus volume might not be `mainnet_powergate-lotus`.

Now it liturally won't even let me start the node. I get this now...

"lotus_1 | ERROR: initializing node: starting node: could not build arguments for function "github.com/filecoin-project/lotus/node/modules/lp2p".StartListening.func1 (/lotus/node/modules/lp2p/addrs.go:98): failed to build host.Host: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build lp2p.BaseIpfsRouting: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.NetworkName: could not build arguments for function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): failed to build dtypes.AfterGenesisSet: received non-nil error from function "reflect".makeFuncStub (/usr/local/go/src/reflect/asm_amd64.s:12): getting genesis block failed: blockstore: block not found"

This is beyond frustrating...

jsign · 2021-08-31T21:46:51Z

I'm sorry to hear that. I hope solving this Lotus problem can be easier, but usually deleting the state and spinning again from a snapshot usually solves all the problems.

ChristianOConnor · 2021-08-31T21:50:39Z

I'm sorry to hear that. I hope solving this Lotus problem can be easier, but usually deleting the state and spinning again from a snapshot usually solves all the problems.

It's not your fault and I'm really sorry to be such a pain with this. I just really want to incorporate Filecoin as the backend of my storage app and for some reason that's proving to be impossible. I will file a bug request with Lotus to see what they say.

Again I really appreciate your help and I love all that you guys do. Please keep following this issue thread as I continue to post updates and ask for help regarding this situation. I really appreciate that you guys have been helping me along the way.

jsign · 2021-08-31T21:52:43Z

We're working on a new and easier solution to allow people like you use Filecoin for your products.
I'll keep you posted around that.

ChristianOConnor · 2021-08-31T23:01:43Z

We're working on a new and easier solution to allow people like you use Filecoin for your products.
I'll keep you posted around that.

Oh wow that would be amazing! Please keep me posted on that. You can reach me through Github or contact me on Reddit as "u/warpanomaly" or Twitter as "OConnorROCKS" Thanks again!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

ChristianOConnor commented Aug 21, 2021 •

edited

Loading

ChristianOConnor commented Aug 23, 2021

ChristianOConnor commented Aug 25, 2021

andrewxhill commented Aug 25, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021 •

edited

Loading

jsign commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021 •

edited

Loading

jsign commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

jsign commented Aug 30, 2021 •

edited

Loading

ChristianOConnor commented Aug 31, 2021 •

edited

Loading

ChristianOConnor commented Aug 31, 2021 •

edited

Loading

jsign commented Aug 31, 2021

ChristianOConnor commented Aug 31, 2021

jsign commented Aug 31, 2021

ChristianOConnor commented Aug 31, 2021

jsign commented Aug 31, 2021 •

edited

Loading

ChristianOConnor commented Aug 31, 2021

I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

I Made A Hot Storage Deal And I Still Can't Access The Content on IPFS #858

Comments

ChristianOConnor commented Aug 21, 2021 • edited Loading

ChristianOConnor commented Aug 23, 2021

ChristianOConnor commented Aug 25, 2021

andrewxhill commented Aug 25, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021 • edited Loading

jsign commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021 • edited Loading

jsign commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

ChristianOConnor commented Aug 30, 2021

jsign commented Aug 30, 2021 • edited Loading

ChristianOConnor commented Aug 31, 2021 • edited Loading

ChristianOConnor commented Aug 31, 2021 • edited Loading

jsign commented Aug 31, 2021

ChristianOConnor commented Aug 31, 2021

jsign commented Aug 31, 2021

ChristianOConnor commented Aug 31, 2021

jsign commented Aug 31, 2021 • edited Loading

ChristianOConnor commented Aug 31, 2021

ChristianOConnor commented Aug 21, 2021 •

edited

Loading

ChristianOConnor commented Aug 30, 2021 •

edited

Loading

ChristianOConnor commented Aug 30, 2021 •

edited

Loading

jsign commented Aug 30, 2021 •

edited

Loading

ChristianOConnor commented Aug 31, 2021 •

edited

Loading

ChristianOConnor commented Aug 31, 2021 •

edited

Loading

jsign commented Aug 31, 2021 •

edited

Loading