nocopy option doesnt work without moving files (defeating part of the purpose)

[PCSmith]

Version information: 0.4.10

Type: Bug / Implementation Flaw

Severity: high

Description:

The "filestore" capability should be considered baseline for this project IMO. The expectation cant be to duplicate all bytes you want to place on IPFS by keeping a copy in the datastore... Nor can it be to expect users to throw their existing directory organisational structure out the window to manually copy IPFS candidate files to a central location.

Need the ability to --nocopy any file in any location and have that file not move or be copied anywhere. Also, if this --raw-leaves thing is needed for this it should be done by default. Not sure if it is though.

Thanks for your work on this world changing project.

[whyrusleeping]

@PCSmith ipfs can only add files within its directory context to the filestore as a security measure. Think of it like a git repository. If i remember correctly, symlinks work fine, so you can symlink your /datamount into your homedir and use the filestore from there.

[kevina]

Note that the .ipfs default location is the home directory. This exposes the entire home directory that is likely to contain a lot of sensitive information. Thus, at least with the current setup, I do not see this is a very convincing argument. Apologies if I am missing something obvious.

[PCSmith]

A few thoughts: I'm not linux right now, as with the majority of your likely intended audience for this project, I'm Windows. The "home" directory isnt where most data is kept. I keep nothing in there in fact, and dont care to as thats on my SSD.

My use case is making a several hundred videos from my youtube channel accessible via IPFS. The video files are dispersed through a file system meant to organize them with their other assets (clips, adobe premiere files, audio, etc). Copying them would be a ridiculous waste of space. Moving them all away from their support structure and accompanying assets would be more than inconvenient.

I do not argue the security concerns and I'm glad you guys are keeping an eye on it. Though exceptions accepted through positive actions should be possible for usability. And please dont put Windows last on your list of considerations. I'd argue it should be near the front for adoptions sake, not because I dont use and love linux.

If symlinking my video tree into the ipfs path will work for this I will def give it a shot. Thanks guys!

[PCSmith]

Before I spend the time deleting my store and recreating everything let me make sure I understand how this will work on Windows.

The IPFS executable existes in a folder on drive X. Lets say X:\IPFS\IPFS.exe which has been added to my paths. The data store through the IPFS_PATH environment variable exists in X:\IPFS\Datastore. If I symlink my youtube video tree in on X:\IPFS\YouTubes then filestore can work with those files without copying or moving them?

[whyrusleeping]

@PCSmith Hey, thanks for the feedback. Getting feedback and people pushing for better windows support definitely helps us prioritize things.

The symlink as you describe it should work. Let me know if you run into any issues, I have a windows VM around now and should be able to help debug.

[PCSmith]

Is raw-leaves required?

[djdv]

Random anecdote, I've added over a terabyte of data across many different files and folders via nocopy, utilising symlinks in the IPFS_PATH without issues on Windows (outside of issues relating to filestore commands that aren't implemented yet). I made a shell extension for Windows that makes a symlink of the target's parent folder and this is the primary method I use for now.

---

2021 edit:
Key formats seem to have changed and I still get questions about this project.
The last published version is at /ipfs/zDMZof1m1fX98cTLyC2VLe9iDQQhWgDLu5foshBSsxSWHQNuiyYV
and the IPNS key is now /ipns/k51qzi5uqu5di8iluwqo958r5wf6vw7imzfww3zg1gi7br27ze7h3k93ddisr8 (it's the same keyfile after I imported it with ipfs key import from the old /ipns/QmaUgENG66kp6cyYUoiKREJWRaaQZmFt7EfFEnoMN1UvJZ key)

I haven't maintained this but if you replace the bundled ipfs.exe it probably still work. If you find it useful and want me to update it, reach out to me and I'll try to fix up the code so that the binary works and can release the code with it.
Or consider IPFS Desktop.

[whyrusleeping]

@PCSmith yes, raw leaves is required. In the near future we will be defaulting that option to true for normal adds as well.

[PCSmith]

crap -- I added my entire library without raw-leaves. Do I need to clean and redo? What happens without it?

[whyrusleeping]

If youre using the --nocopy option, it turns --raw-leaves on automatically for you.

[PCSmith]

oh! perfect. thanks. You can close this out I suppose. But I think my criticism still stands if your target audience is your typical computer user. Symlinks and environment variables are probably not going to work for them. If thats not the audience then disregard. I'm just having trouble figuring how this is going to mainstream without laymen being able to easily seed / pin things / publish things.

The example is Dtube -- most of these people are uploading videos without realizing that unless their vids pay that dude enough to keep their files pinned on his hosting platform that their videos are not going to live long unless they're running their own node and pinning everything themselves. Because the browser version of ipfs cant exactly pin most videos (50 mb limit right?), and even if it could it would only be running while that page was open. I guess we're OT at this point. Just rambling.

Is there someone available for the project to be interviewed on my channel? I'd love to promote the project.

[PCSmith]

btw djdv -- it is so freaking awesome that you were able to deploy that site with video and download to IPFS. Loving the possibilities here.

[whyrusleeping]

But I think my criticism still stands if your target audience is your typical computer user

Well, the typical computer user won't be using the command line either. When we have a nicer user interface for ipfs, this sort of thing could be more easily automated and hidden away from the user.

and even if it could it would only be running while that page was open

Not true actually, using an ipfs service worker, you could have a js-ipfs node running in the background being used by any website that needs it.

Is there someone available for the project to be interviewed on my channel?

I would be interested, but i'm going to be traveling for a few weeks so it might be difficult.

[PCSmith]

When we have a nicer user interface for ipfs, this sort of thing could be more easily automated and hidden away from the user.

Is anyone working on that or is that an area I might contribute?

and even if it could it would only be running while that page was open

oh yeah! I forgot about those.

I would be interested, but i'm going to be traveling for a few weeks so it might be difficult.

Awesome. Let me know how or who I can get in touch with my producer to set it up?

[whyrusleeping]

Is anyone working on that or is that an area I might contribute?

there are a lot of different projects, but nothing that nice yet. I think the biggest issue is nobody really knows what is needed or wanted. Who are the users? What are the use cases? how do we best support that, etc.

Let me know how or who I can get in touch with my producer to set it up?

Grab my email from a commit (spam avoidance)

[kevina]

@PCSmith can I have a link to your channel? Your GitHub profile doesn't say much about you.

[PCSmith]

I emailed you a while back why...

Channel is here:
https://steemit.com/@disenthrall
https://dtube.video/#!/c/disenthrall
https://www.youtube.com/DisenthrallMe
https://www.facebook.com/Disenthrall/

[whyrusleeping]

Hey, I got your email. Just been moving around a lot without stable internet since then

…

On Sun, Oct 8, 2017, 7:17 AM Patrick ***@***.***> wrote: I emailed you a while back why... Channel is here: ***@***.*** https://dtube.video/#!/c/disenthrall https://www.youtube.com/DisenthrallMe https://www.facebook.com/Disenthrall/ — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABL4HJUYLskhjX_W9h4dQ4rEU7xZtbMWks5sqEzhgaJpZM4PUJyC> .

[PCSmith]

no worries. I look forward to the talk. :)

2 side questions:

1. How do I update a nocopy file in ipfs when I move it?

2. Is there a way to search IPFS to see if a file has been nocopy pinned?

[kevina]

1. How do I update a nocopy file in ipfs when I move it?

You can't right now please see #4260

2. Is there a way to search IPFS to see if a file has been nocopy pinned

The best you can do now is ipfs filestore ls to get the contents of the filestore this will only list the leafs not the pinned roots.

[Voker57]

@whyrusleeping could you please explain the security importance of root restriction, from which attack vector does it protect?

[renich]

I am typing randomly here but, I think this is solvable in a not-so-convoluted way by:

# create systemd service unit
cat << 'EOF' > /etc/systemd/system/ipfs@.service
[Unit]
Description=InterPlanetary File System
After=network.target

[Service]
ExecStart=/usr/local/bin/ipfs daemon --enable-gc --migrate
ExecStop=/usr/local/bin/ipfs shutdown
Group=%i
Restart=always
Type=simple
User=%i

[Install]
WantedBy=multi-user.target
EOF

# create a user for this purpose
useradd --create-home --home-dir=/var/lib/ipfs/ --system --shell=/bin/bash ipfs


# login as that user
su - ipfs


# init
ipfs init


# create relevant dirs
## mounts
mkdir -m 2770 mounts
mkdir -m 2770 mounts/{foo,bar}

## ipfs and ipns
mkdir -m 2770 ipfs ipns


# configure ipfs
## enable filestore
ipfs config --bool Experimental.FilestoreEnabled true

## set ipfs and ipfs mount points
ipfs config Mounts.IPFS $( pwd -P )/ipfs
ipfs config Mounts.IPNS $( pwd -P )/ipns


# exit ipfs user
exit


# go back to the user
su - ipfs


# check peers
ipfs swarm peers

# mount whatever directories you want
## bind existing directories. You could, also, add the entry at /etc/fstab:
## /home/renich/foo /var/lib/ipfs/mounts/foo none bind
## note: remember that the directory has to be readable by the ipfs user now. It's entirely up to you how you do this. I can think of: 
## * common group between users and ipset
## * ACLs
## * bindfs UID and GID mapping
## * add the ipfs user to the user's group (not recommended but pretty much how it currently works)
mount -o bind /home/renich/foo ~ipfs/mounts/foo

## mount a drive
## you could, also, add it to /etc/fstab
## /dev/sdXi /var/lib/ipfs/mounts/bar btrfs defaults
mount /dev/sdXi ~ipfs/mounts/bar


# exit ipfs user
exit


# start the daemon
systemctl start ipfs@ipfs.service


# back to ipfs user
su - ipfs


# add stuff
ipfs add --progress --recursive --nocopy $HOME/mounts/foo
ipfs add --progress --recursive --nocopy $HOME/mounts/bar

I mean, it's not as easy as curl some-script | bash but it works more less.

[ghost]

Hey @renich please dont use IPFS for anything that could be regarded as copyright infringement. I've edited your comment slighty. For more info see https://github.com/ipfs/community/blob/master/code-of-conduct.md

Thanks for the script though, very nice.

[renich]

@lgierth sure thing. Just joking a bit. ;) Won't happen again.

[renich]

@lgierth btw, you missed a few. Updated again.

[PCSmith]

You are attempting to police content available on IPFS now? interesting... noted.

-Patrick
Intellectual property is not a valid form of property.

[whyrusleeping]

Nope, not policing content on ipfs. Just the community forums that we spend our time maintaining for the sake of the community.

[iain17]

I'm wanting to do something similar. I have a use case where every peer in my IPFS network shares a certain directory with files up to 10gb or more. Where this directory is located differs per user. I'm a little confused as to what --nocopy actually does now.... How can I add these files to IPFS without duplicating each file to the ~/.ipfs/ home directory?

The app will mostly be used on Windows so I doubt the symlink solutions discussed in this issue are available right?

Perhaps a possible solution would be to add a option to add hashes of the files to the network, telling the network that this peer has the files. Then when a peer asks for a hash which IPFS can't find in its own filestore, we have a callback to return a io.Reader of different locations where it could... Giving the end user some flexibility.

[Voker57]

@iain17 if users store data in their $HOME, they can use --nocopy, if not, they can add a symlink to their $HOME.

I'm still mystified on how that restriction improves security, however.

[iain17]

@Voker57 ah thanks for clearing that up. Same here. Do you know by any chance where in the code base this check if its inside of the home directory is done?

[jbarthelmes]

Relevant discussion: ipfs/go-filestore#25

@jbarthelmes:

@Stebalien: If an attacker could access the go-ipfs API, it could use the filestore to read an arbitrary file from the disk.

I still don't see the problem that the deleted 3 lines of code solve.

access the go-ipfs API

Which API do you mean? I'm thinking of these scenarios:

1. Local privilege escalation: attacker gains same read access as ipfs daemon user. Solution: restrict that user's rights. Access to $HOME is already enforced by the OS.
2. Remote attacker uses HTTP API to read from file system. Solution: Fix your network security.

Either way, restricting access to $HOME by default does not protect sensitive files like .ssh/id_* or .ipfs/config.

[Stebalien]

Unfortunately, someone could easily misconfigure their go-ipfs node and expose their IPFS API to other origins (e.g., allow the "*" origin). We plan on switching from origin to token/oauth-like security but until then, we're trying to be careful.

Either way, restricting access to $HOME by default does not protect sensitive files like

I agree. That's why this feature is still experimental and off by default. We're not removing it because there are legitimate use-cases and users who rely on it, but I don't want to make the situation any worse.

[jbarthelmes]

I made a quick fork at jbarthelmes/go-ipfs if you're bothered by this issue. It seems to just work with reading/writing absolute paths.

@mrambossek updated just for you

[Stebalien]

Note: If you like the filestore feature and want to make it better, I'd recommend implementing a stand-alone program to serve directories over IPFS. This program would: 1. Monitor for changes within some target directory. 2. Add & remove files from an "ipfs" directory when they're added/removed/changed in the target directory. 3. Publish the root CID using IPNS and/or DNSLink. This would sidestep the biggest issue with the filestore: if files referenced by the filestore are deleted or modified, go-ipfs will consider the datastore corrupt because a block it thinks should exist is missing.

[Voker57]

Note: If you like the filestore feature and want to make it better, I'd recommend implementing a stand-alone program to serve directories over IPFS. This program would: 1. Monitor for changes within some target directory. 2. Add & remove files from an "ipfs" directory when they're added/removed/changed in the target directory. 3. Publish the root CID using IPNS and/or DNSLink. This would sidestep the biggest issue with the filestore: if files referenced by the filestore are deleted or modified, go-ipfs will consider the datastore corrupt because a block it thinks should exist is missing.

Sounds like a lot of interfacing effort with unclear benefits for me, why not make IPFS monitor changes itself, and remove missing blocks automatically?

[Stebalien]

It doesn't interact well with go-ipfs pins/deduplication. You could later add other files that share blocks with these filestore files. If you then remove the filestore files, you'd lose the new files as well (or at least lose pieces of them).

It's possible to work around this but it's non-trivial.

[AzureCerulean]

@djdv ...> I made a shell extension for Windows that makes a symlink of the target's parent folder and this is the primary method I use for now.

I tried to access this, is it still available?

[Vort]

Symlinks are nothing better than making one more copy of the files: both are ugly hacks.
Security problems are solved somehow by DC++, torrents etc.
So it should be possible for IPFS too.

But for now IPFS is, sadly, unusable for me :(
(I wanted to add already seeded via torrent Libgen files (instruction), but looks like it is impossible without hacks right now)

Also if I tell software to share single folder, then I expect that it will do exactly this thing.
Not any file on HDD, not different folders etc.
If it is so hard for developers to implement, than it is a larger problem, than symlinks alone.

[jsarenik]

Adding myself to the list of people who use --nocopy on latest ipfs version just because according to https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#road-to-being-a-real-feature-2 this feature needs more people to use and report how well it works. It works well for me. Linux x86_64 glibc (Ubuntu 20.04). Thanks!

[kovan]

Does this even work when IPFS_PATH is set? I keep getting Error: cannot add filestore references outside ipfs root (.), no matter how I place the directories.

[whoizit]

I use bind to get around the problem outside ipfs root
in /etc/fstab:

# example, if .ipfs dir in /home/user
/mnt/share /home/user/share none bind

[djdv]

@AzureCerulean
Sorry for responding a year later, I must have missed this notification.
The IPNS key format seems to have changed or something. I updated the links but that project is no longer maintained.
IIRC the code for this predates any (stable) IPFS APIs so it just wraps the binary directly. This is a bad strategy that I wouldn't do now that the CoreAPI exists, but it also means that the extension likely still works. But I haven't tried it.
The hashes in that post were updated though, so they should actually resolve now (when my node is online) if you're still interested in it.

Judging by the comments above, It seems like the underlying --nocopy feature itself though has regressed in some way so even if the extension works, the command itself might not :^/
But if it does and it seems useful let me know and I might rewrite it and publish the sources.
The current/old version is a nasty mess of C++ and Go that communicate over my own binary protocol so it's probably easier to just rewrite it in modern C++ and communicate directly with the IPFS node over http.
(The extension was never meant to be anything more than a quick hack for myself but people really liked it so I published it)

[mrambossek]

so.. apart from the patched but outdated version here jbarthelmes/go-filestore@5a54f9a .. is there a solution for windows users now?
symlinks/hardlinks do not work for me; UNC paths obviously not either. regardless of what i do, i always get

929.25 MiB / 5.33 GiB [=============>-----------------------------------------------------------------]  17.02% 00m04s
Error: cannot add filestore references outside ipfs root (D:\ipfs)

(actual files / directory would be on e: or i:, so other drive)

[tinfever]

I'm having the same issue with the "Error: cannot add filestore references outside ipfs root" message with IPFS_PATH set. I wanted to add 600GB of files already downloaded via torrent but it appears I can't do so.

[mrambossek]

I was unable to get any help regarding this on the matrix/discord channel as well. Either devs are currently busy with other things or windows is not really a concern for them :( trying to find alternatives to ipfs now.. my main problem is that I want to have a (moderately large) tree of dirs and (rather large) files and the structure keeps changing .. if I just create, say, a torrent, this would mean a new overall torrent on every single change - a problem that ipfs supposedly has solved...

[Vort]

@mrambossek you may look at DC++ networks (AirDC++ client for example).
This technology is old, but it do exactly what is needed - you specify what directory you want to share and it becomes shared.

[FryingPanBrock]

@mrambossek you may look at DC++ networks (AirDC++ client for example). This technology is old, but it do exactly what is needed - you specify what directory you want to share and it becomes shared.

But then it would not be on IPFS, thus defeating the point. People like @mrambossek and I want to share files on specifically IPFS without duplicating said files, which could potentially be very large. I hope that the IPFS developers will take this issue more seriously in the near future. It is, to me, the only reason that torrents or Direct Connect still have a use case: your 200 gigabyte torrent will not take up 400 gigabytes of space on your hard disk.

I have a dream that one day IPFS can be used like a decentralised Dropbox without hassle.

[mrambossek]

so.. came back almost a year later and retried.. still same issue...
no love for windows people?

just to clarify the issue once more; on linux, you can mount --bind stuff all over the place, and solve it like that.
on windows, you are very limited in that regard; you cannot create "Hard Links" or "Junctions" to, for example, network paths.

symbolic links, even though they require admin rights to create, are NOT followed by ipfs when nocopy-adding; instead, the tiny lnk files are added. the --dereference-args parameter does not help either, both because it only works for single file arguments (not directories and their respective files) and also because when i try to add a single file with it, i get the dreaded "outside of ipfs" error again.

soooo.. for me (im not a dev) the most logical approach would be to add a configuration parameter like "additional paths" where you can add an array of paths that, if the file-to-be-added is "below", allow it to proceed.... would that be as relatively easy as i think it should be or am i missing something?

[pedroapero]

I'm running Kubo on Linux in a Docker container.

I tried with symlinks, but it added a 0B file (probably the link file and not the destination).

I tried mounting the disk in $IPFS_PATH/media, and I got:

• with small files: «Error: filestore is not enabled, see https://git.io/vNItf» (although the feature is enabled)
• with big files: it stales at 1.75 MiB.

[HostFat]

Any news on this? :)

[simonhorlick]

This issue has now been open for 6 years. I'm just wondering what the blocker is here? The code change itself is trivial, but what would need to happen for that to be accepted?

[HostFat]

Is there a way to donate specifically to completely fix this issue on every OS?