-
-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow listing of directory contents #39
Conversation
It is still WIP, just first proof of concept implemented and tests are missing. BREAKING CHANGE: Other extending drivers implementing DriverContract now need to implement new method list(location: string): DirectoryListingContract<this>
Hey @Ruby184 . Great proposal Can you also investigate and add the listing support to s3 and gcs? I would want all the core drivers to have feature parity |
Thanks @thetutlage, sure I will add the support also to these drivers. I have added the section according to implementation at the end of the proposal ( |
…ose it on DriverManager
… for drivers To solve problems with current implementation and centralize path manipulation we are introducing new class `PathPrefixer` which also changes behaviour for resolving paths. BREAKING CHANGE: Absolute paths are treated differently and are always prefixed with disk root path to prevent accessing files beyond the disk root
Looks good at the surface. I might share small modification once I try it in real. One thing, what's the benefit/use case of having |
I have added |
…items from direcory
…n in normalizePath
adonis-typings/drive.ts
Outdated
* Shape of the directory listing constructor, we export the constructor for others to add macros and getters to the class. | ||
*/ | ||
export interface DirectoryListingConstructorContract | ||
extends MacroableConstructorContract<DirectoryListingContract<DriverContract, DriveListItem>> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want directory listing class to be macroable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is a good idea for good DX to allow chainable one line calls which are readable. For example, maybe somebody wants to list files matching a glob pattern, so he can add a nice chainable method and use it like this:
// get json files in directory
const files = await driver.list('some/dir').glob('*.json').toArray()
filter( | ||
predicate: (item: T, index: number, driver: Driver) => Promise<boolean> | boolean | ||
): DirectoryListingContract<Driver, T> | ||
|
||
/** | ||
* Transform generated items of listing with the given mapper function. | ||
*/ | ||
map<M>( | ||
mapper: (item: T, index: number, driver: Driver) => Promise<M> | M | ||
): DirectoryListingContract<Driver, M> | ||
|
||
/** | ||
* Do recursive listing of items. Without the next function it will do listing of leaf nodes only. | ||
* For advanced usage you can pass the next function which will get as parameter current item and it should | ||
* return the next location for list or null if the recursion should stop and yield the current item. | ||
* For advanced usage you can also limit the depth of recursion using the second argument of next function. | ||
*/ | ||
recursive( | ||
next?: (current: T, depth: number, driver: Driver) => Promise<string | null> | string | null | ||
): DirectoryListingContract<Driver, T> | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we really need these additional functions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if you mean filter
and map
it is good for developers which prefer functional programming with higher-order functions over imperative style using for loop.
If you mean pipe
it is added so you can add transforming functions - when extending listing with custom method.
Okay, So I have been through the PR for the most part. Some bits looks complicated to me (because my understand of iterables is limited) Whenever things looks complicated to me, I try to question the use case we are trying to serve. So let's do that in this case as well.
Are my assumptions correct? |
Yes, it is the main point and use case for
Sure, sometimes we want the list of all files in given directory no matter how deep they are in given directory.
Yes, filtering files is a common use case. With current implementation you can do it in many ways according to what you prefer. // 1. just use the for loop and if
const files = []
for await (const item of driver.list('some/dir')) {
if (item.location.endsWith('.json')) {
files.push(item.location)
}
}
// 2. use filter higher-order function
const files = []
for await (const item of driver.list('some/dir').filter((i) => i.location.endsWith('.json'))) {
files.push(item.location)
}
// 3. do not use for-await-of at all and just use higher-order functions and convert result to array
const files = await driver.list('some/dir').filter((i) => i.location.endsWith('.json')).map((i) => i.location).toArray()
Yes, async iterables are great when you just need to fetch chunks of data on demand, as you said. It is maybe a little slower as you need to wait for each call for the next chunk of data, but not all the data are in memory the whole time. As usage is not as straightforward as with arrays, I tried to add nice API and allow working with iterables for people who do not like it and prefer working with directory listing as arrays. So it is a compromise. |
Cool. Thanks for the answers. So what I am thinking is when exactly am I going to use iterables? If I am creating a JSON API, I will have to anyways collect the tree inside the buffer and then send it as response. Same is the case when rendering the tree inside edge templates |
Yeah, you are right. The main use case for iterables is to perform operations on the files, for example you want to delete all the files in the directory (or maybe move all files to other location etc.) for await (const item of driver.list('some/dir').recursive())) {
await driver.delete(item.location)
} |
Yup, that was the thing I was missing. Totally makes sense. Now, how should we approach with the same API for |
As I have mentioned in the proposal, |
@Ruby184 Thanks for being patient :) The PR looks good to me. So let's merge it. A couple of questions before I hit the merge button
|
I like the idea the developers could add the methods what they consider as needed, for example mentioned
I think we should handle it by releasing the major version of the package as Semantic Versioning recommends for such changes. |
I'm not sure it's a breaking change. The generic has a default value. |
Yeah this one is not a breaking change and should be BC but there are 2 other breaking changes: BREAKING CHANGE: Other extending drivers implementing DriverContract now need to implement new BREAKING CHANGE: LocalDriver is now treating absolute paths differently and are always prefixed with disk root path to prevent accessing files beyond the disk root |
We can make it non-breaking by making the method optional (the default implementation could throw an error).
We could argue that this is a security fix and as such may be breaking in a patch/minor release. |
Yes, but I do not think it is a good idea as we cannot add default implementation which will throw error to extended drivers outside the core because
Yeah, I agree with that this is considered as security fix. And I do not expect people were using absolute paths with these methods when using local driver. |
The package is part of adonisjs/core. I don't know how often @thetutlage intents to make major releases of the core, but as a general thought: it would be unfortunate to require major releases when a new API is added (this new method doesn't make old code throw or change behavior). |
The absolute paths one is a bug fix and usually bug fix breaks existing code, since the old behavior was unexpected. Also, I agree with @targos here, this is just a breaking change for other packages relying on Drive base interface and not the end user code. So, I will prefer not making a breaking change. So, to conclude. Let's make the |
Yeah, sure, I got it. Actually I have got an idea, let me try to make some changes. I think I can make it optional and prevent the need of checking for existence in user code when using core drivers. |
I don't think anyone will use it immediately. If you don't need then just remove it. I try to keep the API scope as slim as possible and expand as I get feature requests |
@thetutlage @targos All should be good and ready to merge. |
All looks good to me. One final thing, can we have tests for other listing methods as well? Like |
@thetutlage I have added the tests you wanted :) |
…pends on filesystem
Looks all good to me and I am ready to merge. What you think @targos ? |
Sorry I don't have time to look at it again. I trust your judgment 👍 |
@Ruby184 Thanks a ton for taking the time to create this PR and also showing patience with all the back and forth we had. 🎉 |
Good evening everyone, really sorry if this is not where I have to express my concern. Indeed, I had to use the list functionality on Drive.use('s3').list!('images').toArray(), |
Hello @Akpagni5547, implementation for |
Proposed changes
I propose to add listing of directory contents for drivers. To allow flexibility, we should implement it using async iterators which are natively supported from Node.js 10.x. But also to allow easy manipulation we should wrap async iterables and add some nice API for transforming the listing output and maybe allow extensions.
DirectoryListing class
To wrap listing and allow transformations of the output we are using the class
DirectoryListing
which:AsyncIterable
so it can be used directly using for-await-of looppipe
listing through series of transformation functions which takes current async iterable and returns one with transformed outputfilter
andmap
toArray()
function to convert async iterable to array if we do not want to use for-await-of.DriveManager
instance asDirectoryListing
to be extended from outsideDriveListItem interface
It is a type of items returned from driver listing. Currently it is specified as follows:
Drivers can extend the interface and provide type for
original
or add additional properties:To prevent repeation I have introduced optional type parameter to
DriverContract
(it defaults to DriveListItem if not provided):Driver list implementation
Disk driver needs to implement
list(location: string): DirectoryListingContract<this, DriveListItem>
function which returns instance ofDirectoryListing
class. It accepts to constructor instance of disk driver (this) and async generator function which yields entries in given location. Because generators cannot be arrow functions we use driver instance (this) so we can bindthis
inside generator function to it.See example of implementation of
LocalDriver
(note that we are usingthis.adapter
inside async generator):Usage
Example usage of driver list function inside user code:
PathPrefixer
To solve problems with current implementation and centralize path manipulation we are introducing new class
PathPrefixer
which:/
as separator/home/user
using/some/directory/
as location has the same behaviour assome/directory
and both resolves to path/home/user/some/directory
not/some/directory/
in first case as before)../some/directory
. Note thatsome/../directory
is OK as it resolves todirectory
''
prefixprefix
config option for cloud providers to create disk where objects are always transparently prefixed with given prefixTypes of changes
What types of changes does your code introduce?
Put an
x
in the boxes that applyBREAKING CHANGE: Other extending drivers implementing
DriverContract
now need to implement newmethod
list(location: string): DirectoryListingContract<this, DriveListItem>
BREAKING CHANGE:
LocalDriver
is now treating absolute paths differently and are always prefixed with disk root path to prevent accessing files beyond the disk rootChecklist
TODO
DirectoryListing
Macroable so it can be extended by users and packagesrecursive
transformer toDirectoryListing
Further comments
More description of further changes needed to related packages.
Implementaion of list for s3 and gcs
After we agree on the proposal of the API, merge PR, and release a prerelease version of
@adonisjs/drive
and@adonisjs/core
, I will open a PRs for@adonisjs/drive-s3
and@adonisjs/drive-gcs
for implementinglist
from changedDriverContract
interface.These cloud drivers are similar in concept of storing objects and not having traditional directories. But directories could be handled with prefixes of object keys and delimiter. Here is an example of implementation of list for s3 driver (not tested):