Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support storing UnixFS 1.5 Mode and ModTime #7754

Closed
wants to merge 25 commits into from

Conversation

kstuart
Copy link

@kstuart kstuart commented Nov 5, 2020

Feature Progress

  • Can ipfs add with preserved mode and/or last modification time
    • on files
    • on directories
  • Can ipfs add with custom mode and/or last modification time
    • on files
    • on directories
  • Can ipfs get restoring mode and/or last modification time
    • on files
    • on directories
    • in archives
  • Can ipfs files chmod to change mode
    • on files
    • on directories
  • Can ipfs files touch to change last modification time
    • on files
    • on directories
  • Automatically update the last modification time when file data is changed or truncated (e.g. ipfs files write)
  • Can add files and directories with mode and/or modification time using multipart-form data
  • ipfs files stat reports mode and last modification time

Related PRs

Implementation Notes

  • When adding files and directories without opting to store a mode or modification time the same CIDs are generated that would have been created before this feature was implemented (opt-in).
  • The Go runtime currently has no native support for restoring file mode and modification time on symbolic-links, support for restoring the last modification time has been added for Linux distributions and the following BSDs: freebsd, netbsd, openbsd, dragonflybsd.
  • Automatically updating a modification time will only occur if a modification time was previously stored.
  • When creating an archive, for compatibility, time resolution is to the second; Nanoseconds are not supported.

IPFS Add (CLI)

The ipfs add options --preserve-mode and --preserve-mtime are used to store the original mode and last modified time of the file being added, the options --mode, --mtime and --mtime-nsecs are used to store custom values, a custom value of 0 is a no-op as is providing --mtime-nsecs without --mtime.

The preserve flags and custom options are mutually exclusive, if both are provided the custom options take precedence.


Closes #6920

This commit introduces initial Mode and ModTime support
for single filesystem files and webfiles.

The ipfs add options --preserve-mode and --preserve-mtime are
used to store the original mode and last modified time of the
file being added, the options --mode, --mtime and --mtime-nsecs
are used to store custom values.

A custom value of 0 is a no-op.

The preserve flags and custom options are mutually exclusive,
if both are provided the custom options take precedence.
@welcome
Copy link

welcome bot commented Nov 5, 2020

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

  • The context for this PR is clear, with relevant discussion, decisions
    and stakeholders linked/mentioned.

  • Your contribution itself is clear (code comments, self-review for the
    rest) and in its best form. Follow the code contribution
    guidelines

    if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

  • A maintainer will triage and assign priority to this PR, commenting on
    any missing things and potentially assigning a reviewer for high
    priority items.

  • The PR gets reviews, discussed and approvals as needed.

  • The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

@kstuart kstuart changed the title preliminary support for unixfs Mode and ModTime Support storing UnixFS 1.5 Mode and ModTime Jun 26, 2022
@kstuart kstuart marked this pull request as ready for review October 31, 2022 14:07
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While having feature parity with JS and also UnixFS Specs (draft in ipfs/specs#331) would be nice, this was always a low priority, and slipped during triages. Let's see if we could land this.

@hacdias will look into this and ipfs/boxo#34 and see if we can rebase and ship this in 0.30 without investing too much time.

Some comments inline.

Comment on lines +74 to +78
preserveModeOptionName = "preserve-mode"
preserveMtimeOptionName = "preserve-mtime"
modeOptionName = "mode"
mtimeOptionName = "mtime"
mtimeNsecsOptionName = "mtime-nsecs"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 Is it possible to preserve mtime without preserving mode?

To avoid cross-implementation issues, we should have "zero values" documented in https://github.com/ipfs/specs/pull/331/files

Comment on lines +195 to +199
cmds.BoolOption(preserveModeOptionName, "Apply permissions to created UnixFS entries"),
cmds.BoolOption(preserveMtimeOptionName, "Apply modification time to created UnixFS entries"),
cmds.UintOption(modeOptionName, "File mode to apply to created UnixFS entries"),
cmds.Int64Option(mtimeOptionName, "Modification time in seconds before or after the Unix Epoch to apply to created UnixFS entries"),
cmds.UintOption(mtimeNsecsOptionName, "Modification time fraction in nanoseconds"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cmds.BoolOption(preserveModeOptionName, "Apply permissions to created UnixFS entries"),
cmds.BoolOption(preserveMtimeOptionName, "Apply modification time to created UnixFS entries"),
cmds.UintOption(modeOptionName, "File mode to apply to created UnixFS entries"),
cmds.Int64Option(mtimeOptionName, "Modification time in seconds before or after the Unix Epoch to apply to created UnixFS entries"),
cmds.UintOption(mtimeNsecsOptionName, "Modification time fraction in nanoseconds"),
cmds.BoolOption(preserveModeOptionName, "Apply existing POSIX permissions to created UnixFS entries"),
cmds.BoolOption(preserveMtimeOptionName, "Apply existing POSIX modification time to created UnixFS entries"),
cmds.UintOption(modeOptionName, "Custom POSIX file mode to store in created UnixFS entries"),
cmds.Int64Option(mtimeOptionName, "Custom POSIX modification time to store in created UnixFS entries (seconds before or after the Unix Epoch)"),
cmds.UintOption(mtimeNsecsOptionName, "Custom POSIX modification time (optional time fraction in nanoseconds)"),

Comment on lines +1377 to +1379
Tagline: "Change mode permissions",
ShortDescription: `
The mode argument must be specified in unix numeric notation.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Tagline: "Change mode permissions",
ShortDescription: `
The mode argument must be specified in unix numeric notation.
Tagline: "Change optional POSIX mode permissions",
ShortDescription: `
The mode argument must be specified in Unix numeric notation.


var filesTouchCmd = &cmds.Command{
Helptext: cmds.HelpText{
Tagline: "Change file modification times.",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Tagline: "Change file modification times.",
Tagline: "Change optional POSIX modification times.",

}

const (
defaultStatFormat = `<hash>
Size: <size>
CumulativeSize: <cumulsize>
ChildBlocks: <childs>
Type: <type>`
Type: <type>
Mode: <mode>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 seems <mode-octal> would only be used with --format.

Perhaps it will be useful to show it here as well?

Suggested change
Mode: <mode>
Mode: <mode> (<mode-octal>)

@lidel lidel requested a review from hacdias May 17, 2024 15:42
@hacdias hacdias removed their assignment May 27, 2024
@hacdias hacdias removed their request for review May 27, 2024 12:41
@lidel lidel mentioned this pull request May 28, 2024
32 tasks
gammazero added a commit that referenced this pull request Aug 13, 2024
Replaces #7754 written by @kstuart

- ipfs/boxo#653
- ipfs/boxo#658

- [X] Can `ipfs add` with preserved mode and/or last modification time
  - [X] on files
  - [X] on directories
- [X] Can `ipfs add` with custom mode and/or last modification time
  - [X] on files
  - [X] on directories
- [X] Can `ipfs get` restoring mode and/or last modification time
  - [X] on files
  - [X] on directories
  - [X] in archives
- [X] Can `ipfs files chmod` to change mode
  - [X] on files
  - [X] on directories
- [X] Can `ipfs files touch` to change last modification time
  - [X] on files
  - [X] on directories
- [X] Automatically update the last modification time when file data is changed or truncated (e.g. `ipfs files write`)
- [X] Can add files and directories with mode and/or modification time using multipart-form data
- [X] `ipfs files stat` reports mode and last modification time

**Note:**
- [X] Adds support to `kubo/core/rpc` (may require additional tests).

- ~ipfs/interface-go-ipfs-core/pull/66~ replace by this PR
- ~ipfs/go-unixfs/pull/85~ replaced by: ipfs/boxo#658
- ~ipfs/go-mfs/pull/93~ replaced by: ipfs/boxo#658
- ~ipfs/go-ipfs-files/pull/31~ replaced by: ipfs/boxo#653
- ~ipfs/tar-utils/pull/11~ replaced by: ipfs/boxo#653

- When adding files and directories without opting to store a mode or modification time the same CIDs are generated that would have been created before this feature was implemented (opt-in).
- The Go runtime currently has no native support for restoring file mode and modification time on symbolic-links, support for restoring the last modification time has been added for Linux distributions and the following BSDs: freebsd, netbsd, openbsd, dragonflybsd.
- Automatically updating a modification time will only occur if a modification time was previously stored.
- When creating an archive, for compatibility, time resolution is to the second; Nanoseconds are not supported.

The `ipfs add` options `--preserve-mode` and `--preserve-mtime` are used to store the original mode and last modified time of the file being added, the options `--mode`, `--mtime` and `--mtime-nsecs` are used to store custom values, a custom value of 0 is a no-op as is providing `--mtime-nsecs` without `--mtime`.

The preserve flags and custom options are mutually exclusive, if both are provided the custom options take precedence.

---

Closes #6920
gammazero added a commit that referenced this pull request Aug 13, 2024
This commit introduces initial Mode and ModTime support
for single filesystem files and webfiles.

The ipfs add options --preserve-mode and --preserve-mtime are
used to store the original mode and last modified time of the
file being added, the options --mode, --mtime and --mtime-nsecs
are used to store custom values.

A custom value of 0 is a no-op.

The preserve flags and custom options are mutually exclusive,
if both are provided the custom options take precedence.

Majority of of the code was from #7754 written by kstuart

Replaces #7753

Closes #6920
@lidel
Copy link
Member

lidel commented Aug 13, 2024

Continued in #10478

@lidel lidel closed this Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BOUNTY] Implement UnixFSv1.5 in go-ipfs
3 participants