-
-
Notifications
You must be signed in to change notification settings - Fork 345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Little FS improvements and IFS revisions to support user metadata #2308
Merged
slaff
merged 11 commits into
SmingHub:develop
from
mikee47:update/littlefs-improvements
Apr 20, 2021
Merged
Little FS improvements and IFS revisions to support user metadata #2308
slaff
merged 11 commits into
SmingHub:develop
from
mikee47:update/littlefs-improvements
Apr 20, 2021
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Change `file id` to `file handle` Change `file name` to `file path`
Adds complexity for little gain.
…ind) Parts of oix_hdr not initialised so may leak sensitive information to disk
Fixes - Revise HYFS to track file path SPIFFS getFilePath() call sometimes returns weird results. - Fix SPIFFS FileSystem::getFilePath return code - Fix FileDir weirdness The generic `DirHandle` type points to an `IFS::FileDir` struct, which each filesystem defines internally as required. It looks like the compiler sometimes mixes these up and causes problems, though this has only been seen when compiling in debug mode for Host. So instead, redefine `DirHandle` as an abstract type, move the `FileDir` definition inside the appropriate filesystem namespace and add the `GET_FILEDIR()` macro to deal with checking and casting. - Fix memory leak in test module (thanks valgrind) - Set lasterror in `File::readContent()` - Fix `FileSystem::makedirs()` Improvements - Implement Host `flush`, `rename` and `remove` methods - Helpers return FileSystem* instead of IFileSystem* for easier use - Add optional `IFileSystem::setProfiler` method to enable flexible debugging and performance evaluation - Revise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS) Changes to core interface - Remove `IFileSystem.fopendir(Stat&)` and `IFileSystem.fopen(Stat&)` The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration. However, they are difficult to implemenent effectively and the additional complexity is hard to justify. POSIX defines a whole raft of such methods such as `openat`, `mkdirat`, etc. which add an additional `dirfd` parameter to specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background. This may be re-visited in the future if the need arises. - Add generic get/set attribute methods These support LFS file/directory attributes as well as standard attribute types via `AttributeTag`. There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.
mikee47
changed the title
Update/littlefs improvements
Little FS improvements and IFS revisions to support user metadata
Apr 19, 2021
slaff
added
3 - Review
3.5 - Approved - Waiting for CI results
and removed
3 - Review
3.5 - Approved - Waiting for CI results
labels
Apr 19, 2021
slaff
pushed a commit
that referenced
this pull request
Sep 27, 2021
) **Main changes** This PR adds the following methods to `IFileSystem`: * setProfiler This allows applications to hook into the low-level read/write/erase calls for debugging and profiling. * fsetxattr * fgetxattr * setxattr * getxattr The `setacl`, `settime`, `setattr` and `setcompression` methods have been moved to `FileSystem` and call the above new methods to do their work. Additional overloads have been provided in FileSystem (and via methods of File objects): * setAttribute * getAttribute These work with file handles or paths and have various convenient overloads. * setUserAttribute, getUserAttribute These are to access user-defined attributes. For LFS volumes, each item is limited to a maximum size of 1023 bytes. SPIFFS is less flexible (by design). Metadata is stored in the object headers (after filename, etc.) and is restricted by the setting of `SPIFFS_OBJ_META_LEN`. The first 16 bytes are used for system attributes (e.g. modified time), with user attributes occupying any remaining space. The current default setting (16) is unchanged, so if user metadata for SPIFFS volumes is required then this must be increased accordingly. Each user attribute requires a 2-byte header (tag + size). * fopen * fopendir Removed - see below. **Fixes** - Revise HYFS to track file path SPIFFS getFilePath() call sometimes returns weird results. - Fix SPIFFS FileSystem::getFilePath return code - Fix FileDir weirdness The generic `DirHandle` type points to an `IFS::FileDir` struct, which each filesystem defines internally as required. It looks like the compiler sometimes mixes these up and causes problems, though this has only been seen when compiling in debug mode for Host. So instead, redefine `DirHandle` as an abstract type, move the `FileDir` definition inside the appropriate filesystem namespace and add the `GET_FILEDIR()` macro to deal with checking and casting. - Fix memory leak in test module (thanks valgrind) - Set lasterror in `File::readContent()` - Fix `FileSystem::makedirs()` - Fix LFS reported volumesize **Improvements** - Increase LFS_CACHE_SIZE from 16 to 32 bytes Results in signficant reduction in read counts - Switch to littlefs fork, manage file attributes manually Only changed attributes (and those with non-default values) get written out File time only gets updated if file is changed, opening in write mode is insufficient - LFS mkdir sets mtime on success - LFS `mkdir` succeeds if directory already exists - Add LFS inspect sample - Implement Host `flush`, `rename` and `remove` methods - Helpers return FileSystem* instead of IFileSystem* for easier use - Add optional `IFileSystem::setProfiler` method to enable flexible debugging and performance evaluation - Revise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS) - Remove `IFileSystem.fopendir(Stat&)` and `IFileSystem.fopen(Stat&)` The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration. However, they are difficult to implement effectively and the additional complexity is hard to justify. POSIX defines a whole raft of such methods such as `openat`, `mkdirat`, etc. which add an additional `dirfd` parameter to specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background. This may be re-visited in the future if the need arises. - Add generic get/set attribute methods These support LFS file/directory attributes as well as standard attribute types via `AttributeTag`. There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.
mikee47
added a commit
to mikee47/Sming
that referenced
this pull request
Feb 12, 2022
Opening on directory handle removed in SmingHub#2308
mikee47
added a commit
to mikee47/Sming
that referenced
this pull request
Feb 13, 2022
Opening on directory handle removed in SmingHub#2308
slaff
pushed a commit
that referenced
this pull request
Feb 15, 2022
This PR fixes a few problems related to handling of file/directory Access Control entries. **IFS** - Fix FileCopier class not copying directory attributes - In filesystem builder, allow minifcation to fail, but print warning (to support e.g. JSON template files) **LittleFS** - Fix `readdir()` not setting default ACL - Translate 'NOENT' lfs error code into 'NotFound' IFS error - Fix missing directory attribute (imperfect fix in #2487) Also remove unused FileStream method. Opening on directory handle removed in #2308.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Main changes
This PR adds the following methods to
IFileSystem
:This allows applications to hook into the low-level read/write/erase calls for debugging and profiling.
The
setacl
,settime
,setattr
andsetcompression
methods have been moved toFileSystem
andcall the above new methods to do their work.
Additional overloads have been provided in FileSystem (and via methods of File objects):
These work with file handles or paths and have various convenient overloads.
These are to access user-defined attributes.
For LFS volumes, each item is limited to a maximum size of 1023 bytes.
SPIFFS is less flexible (by design). Metadata is stored in the object headers
(after filename, etc.) and is restricted by the setting of
SPIFFS_OBJ_META_LEN
.The first 16 bytes are used for system attributes (e.g. modified time), with user attributes
occupying any remaining space.
The current default setting (16) is unchanged, so if user metadata for SPIFFS volumes
is required then this must be increased accordingly.
Each user attribute requires a 2-byte header (tag + size).
Removed - see below.
Fixes
SPIFFS getFilePath() call sometimes returns weird results.
Fix SPIFFS FileSystem::getFilePath return code
Fix FileDir weirdness
The generic
DirHandle
type points to anIFS::FileDir
struct, which each filesystem defines internally as required.It looks like the compiler sometimes mixes these up and causes problems,
though this has only been seen when compiling in debug mode for Host.
So instead, redefine
DirHandle
as an abstract type, move theFileDir
definition inside the appropriate filesystem namespaceand add the
GET_FILEDIR()
macro to deal with checking and casting.Fix memory leak in test module (thanks valgrind)
Set lasterror in
File::readContent()
Fix
FileSystem::makedirs()
Fix LFS reported volumesize
Improvements
Results in signficant reduction in read counts
Only changed attributes (and those with non-default values) get written out
File time only gets updated if file is changed, opening in write mode is insufficient
LFS mkdir sets mtime on success
LFS
mkdir
succeeds if directory already existsAdd LFS inspect sample
Implement Host
flush
,rename
andremove
methodsHelpers return FileSystem* instead of IFileSystem* for easier use
Add optional
IFileSystem::setProfiler
method to enable flexible debugging and performance evaluationRevise hybrid filesystem to allow use of alternative writeable filesystem (e.g. Little FS)
Remove
IFileSystem.fopendir(Stat&)
andIFileSystem.fopen(Stat&)
The goal of these methods was to provide more efficient means to open files/directories discovered during enumeration.
However, they are difficult to implement effectively and the additional complexity is hard to justify.
POSIX defines a whole raft of such methods such as
openat
,mkdirat
, etc. which add an additionaldirfd
parameterto specific the working directory. See https://man7.org/linux/man-pages/man3/open.3p.html for some background.
This may be re-visited in the future if the need arises.
These support LFS file/directory attributes as well as standard attribute types via
AttributeTag
.There is no POSIX specification for this but the linux convention (setxattr, etc.) are appropriate.