[epic] vinyl-fs version 3.0.0

[phated]

I'm creating this issue to get some more visibility on outstanding issues in vinyl-fs for it to reach 3.0 (which we'd like to ship with gulp 4).

If you are able to tackle any of these, the help would be very much appreciated.

• Decouple opening and closing of file descriptors from src and dest - Decouple opening and closing of file descriptors from src and dest vinyl-fs#158
• Support changing uid/gid from the vinyl object to the file on disk - Support changing uid/gid from the vinyl object to the file on disk vinyl-fs#157
• Support setting setuid/setgid/sticky bits - Support setting setuid/setgid/sticky in updateMetadata vinyl-fs#156
• Support functions for all options - Support functions for all options vinyl-fs#139
• kebabCase the filenames - kebabCase the filenames vinyl-fs#152
• vfs.dest(outfolder, { base.. }) ignores outfolder? - vfs.dest(outfolder, { base.. }) ignores outfolder? vinyl-fs#141
• The mkdirp module doesn't handle modes in the best way - The mkdirp module doesn't handle modes in the best way vinyl-fs#165
• Account for process.umask - Account for process.umask vinyl-fs#166
• Test refactor - (Don't Merge) Test refactor vinyl-fs#194
• Separate prepareWrite into own module - Proposal to allow files to pass through dest without writing them to disk.  vinyl-fs#193
• Fix Symlink TODOs (issues need to be created)
• Sink the symlink stream - Needs to sink the symlink stream vinyl-fs#195
• Get https://github.com/gulpjs/vinyl-sourcemap up-to-snuff
• Rebase vinyl-fs to generate conventional changelog (Rebase review!!! vinyl-fs#284)
• Finalize documentation

[phated]

Updated! We've had some excellent contributions by @hgwood and @tmcgee123
Some of the improvements brought to light a couple of other things we need help with.

[phated]

Almost all these are wrapping up. We need to get some more tests in place for the changes and solidify the "Support functions for all options" stuff.

Still looking for someone to tackle the uid/gid stuff.

[odino]

What about a smaller update to vinyl? With this issue on minimatch everyone gets lots of sec. alerts :)

[phated]

@odino no, those alerts don't matter. If you are passing user input directly to gulp, you have bigger problems.

[odino]

That's true, but those alerts will keep coming due to gulp requiring the
library. So even if this has never been exploited people are gonna
"wtf?!?!" anyway unless they really dig into it and figure out that this is
not an issue at all.

If there's an easy way to upgrade to a version that has the patch I would
suggest to go for it, else if its a big PITA then never mind :)
On Jul 12, 2016 1:40 PM, "Blaine Bublitz" notifications@github.com wrote:

@odino https://github.com/odino no, those alerts don't matter. If you
are passing user input directly to gulp, you have bigger problems.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1604 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAUC5BPlBGK1FkEyh0WVf2e_q6GkZku2ks5qU2ElgaJpZM4IIqCY
.

[strugee]

@phated the UID/GID item should be checked off, no?

[ilanbiala]

@phated any update on ^^?

[atomantic]

re @odino and security fix. this could be fixed just by updating the version of vinyl-fs that gulp installs. Currently you get vinyl-fs@^0.3.0 which translates to a really old version of vinyl-fs (0.3.14).

This is a problem for anyone requiring nsp check to pass for a build to succeed.

[Den-dp]

this could be fixed just by updating the version of vinyl-fs

@atomantic a lot of things was changed since vinyl-fs@0.3.14 https://github.com/gulpjs/vinyl-fs/blob/master/CHANGELOG.md#v0314-20150921-2357-0000 so it is impossible just bump the version.
In fact, migration to vinyl-fs@3.0.0 is a thing and the whole point of this issue.

[phated]

@strugee @ilanbiala thanks, I've updated the list. However, a few more things had to be added to account for things I've found during the test refactor.

[atomantic]

got it. Yeah, I noticed tests pass as long as I upgrade vinyl-fs to latest version and include glob-watcher 0.0.8--but that's still a really old version of glob-watcher. glob-watcher@3.0.0 has some differences that break it.

[phated]

@atomantic the gulp tests are just a smoke test because each underlying library has a full test suite that we aren't going to duplicate. Extreme amounts of stuff has changed and you should not assume everything will just work.

[phated]

I've submitted the test refactor for review at gulpjs/vinyl-fs#194 - I'd like to get feedback before I push it so feel free to comment.

[pgilad]

Hey - any news on this? gulp is bringing in an outdated version of minimatch (with security issues)..

[phated]

@pgilad I'm pretty diligent about updating this thread when there are updates, so no. You can follow the linked issues to see progress on the outstanding items (and even contribute). The minimatch warning you mention doesn't mean anything because you aren't passing raw user input to the gulp methods (or you shouldn't be).

[addyosmani]

@phated First, thanks for all the awesome contributions you continue to make to Gulp. Second, looking at the prepareWrite note it appears this is done as of gulpjs/vinyl-fs@cc99707 and the associated tests also landed. Was there more work to do before it could be checked off the list?

[phated]

@addyosmani unfortunately that commit didn't solve everything and caused problems with the symlink method so it was never merged. There are some more option normalizations that are in progress, so I have been focusing on the glob-stream and vinyl dependencies.

[phated]

Thanks but we have many issues opened throughout the organization.

You can see everything labeled as "help wanted" at https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+org%3Agulpjs+label%3A%22help+wanted%22+

We are tackling some of them this weekend at HackIllinois

[CKGrafico]

Amazing work, I'll wait to next week. Good luck at HackIllinois

[phated]

This is getting really close. I need help with gulpjs/vinyl-sourcemap#19

[anru]

Any progress on this ? Looks like almost things from list are already done. May be I can help somehow ?

gulp still have very old vinyl-fs dependency, and I get following warning each time I install/upgrade gulp:
npm WARN deprecated graceful-fs@2.0.3: graceful-fs version 3 and before will fail on newer node releases. Please update to graceful-fs@^4.0.0 as soon as possible

├─┬ gulp@3.9.1
│ └─┬ vinyl-fs@0.3.14
│ ├─┬ glob-watcher@0.0.6
│ │ └─┬ gaze@0.5.2
│ │ └─┬ globule@0.1.0
│ │ └─┬ glob@3.1.21
│ │ └── graceful-fs@1.2.3

[avegancafe]

@anru If you check out that link phated posted there's a bunch of issues left in the org, I think those are all related to the upgrade

[mcdado]

@anru I believe the roadmap is here: #1945

More than the warning the main issue of the current version is the incompatibility with Node > 6. Later this year 8 will be LTS and then it will become more problematic to have gulp as a dependency.

I previously said I'm willing to help, I'll probably take a look at doc related issues.

[anru]

#1945 is gulp 4 roadmap. And may be we could release gulp 3 with updated vinyl-fs dependency. Or is it will be more simple to help with gulp 4 where vinul-fs is already updated. What will be more quick/easy ?
I'm asking because there are no incomplete issues in issue description, except Fix Symlink TODOs (issues need to be created)

[strugee]

@anru @mcdado see #1571. The gulp team is aware of the graceful-fs problems, but it's impossible to upgrade Vinyl without breaking changes. Hence, gulp 4.

[subesokun]

Any updates on this? The NSP check fails on all my projects which are using Gulp when scanning also dev dependencies :( https://nodesecurity.io/advisories/118

[ghost]

@subesokun
It seems like that vulnerability requires an attacker to be able to pass an arbitrary glob string to minimatch... which seems like a bad idea in general.

[avegancafe]

Is the "Fix Symlink TODOs (issues need to be created)" item the only thing that's left to do here?

[phated]

@kyleholzinger yes, it is a tough problem we are trying to solve over on the vinyl-fs repo.

[nwhitmont]

What can I do to help move this forward?

[phated]

@nwhitmont the work is happening at gulpjs/vinyl-fs#254

[phated]

We've wrapped up the symlink/junction work. There's just some documentation/rebasing/cleanup work to do on vinyl-fs before the 3.0 release.

If you are interested in this stuff, please go try out the master branch and give us feedback - there are some big changes over there.

[phated]

Documentation has been finalized. Rebase has been completed but it needs review. Check out gulpjs/vinyl-fs#284 if you are interested in reviewing.

[phated]

Based on the reactions, I expected more review/feedback on the rebase ☹️

Anyway, 3.0.0 has been published. Changelog at https://github.com/gulpjs/vinyl-fs/releases/tag/v3.0.0

[CKGrafico]

Thanks for your awesome job @phated

[mcdado]

I tried to have a look but it was a bit overwhelming not knowing anything of the internals.

I second @CKGrafico, thank you and the whole team!