Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: IPLD/IPNS backups for pinner #297

Open
jimpick opened this issue Feb 28, 2019 · 1 comment
Open

WIP: IPLD/IPNS backups for pinner #297

jimpick opened this issue Feb 28, 2019 · 1 comment

Comments

@jimpick
Copy link
Member

jimpick commented Feb 28, 2019

The existing peer-base pinner will persist data to ~/.jsipfs, which works great.

But we'd like to deploy the pinner to Heroku, which has an ephemeral disk. On redeploys, the virtual machine will have a fresh disk, which will lose the pinned data.

As an extra layer, I'm going to try to write the pinner state out to IPLD on each state change, and call the ipfs-cluster API to pin it permanently. Additionally, we can use IPNS to keep track of the current head.

As a proof-of-concept, I've figured out how to save and retrieve the pinner state out-of-band from the IPFS datastore using the filesystem here:

https://github.com/jimpick/peer-pad/blob/jim-dev/pinner/index.js#L117

Next, I'm going to try to use IPLD and IPNS instead of fs to store the out-of-band backups.

@jimpick
Copy link
Member Author

jimpick commented Mar 16, 2019

Making some progress:

https://github.com/ipfs-shipyard/peer-base-pinner

I had to tweak IPNS (go-ipfs), but it can now save and retrieve pinned state across heroku reboots (the IPFS repo needs to be recreated after each reboot because of the ephemeral disks). I'll be added in some support to pin to ipfs-cluster, and with that, we should have a solution for durable pinning that will enable us to deploy the dev PeerPad to production.

It all needs lots of testing ... I think we'll be able to learn a lot from this pinning solution once we have it running in production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant