files: list all required files for build tools

[desandro]

Since resolving main in #43, we've thrown a wrench in the gears for build tools, most prevelantly Wiredep. Wiredep needs a list of all the files to be included when you install the package. Previously, main was being widely used for this purpose. Now main should no longer lists all the files (The entry-point files necessary to use your package. One per filetype). We need another field.

These two proposals are my first shot at it. Your thoughts appreciated 🍉

Proposal 1: files flat list

files is an array of all the files required to use the package. Filepaths may include globs.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

We recommend listing source files in files, rather than compiled distribution files.

"files": [
  "js/motion.js",
  "js/run.js",
  "js/walk.js",
  "sass/motion.scss",
  "sass/run.scss",
  "sass/walk.scss",
  "img/motion.png",
  "img/walk.png",
  "img/run.png",
  "fonts/icons.woff2",
  "fonts/icons.woff"
]

Most likely you would list globs for multiple files of the same type.

"files": [
  "js/*.js",
  "sass/*.scss",
  "img/*.*",
  "fonts/*.*"
]

Pros: Simple specification, easier for developers to adopt. Don't have to worry on Bower spec side about asset-type specification. Cons: Aside from file extension, it's not clear what files do what. For example, a package make require icon.svg — is this file a font or should be it used as an image?

We leave it up to build tools on how they make use of the array. Some might only be concerned about .js and .css files. For globs, they'd have to expand the glob and do the logic on their side.

Proposal 2: asset-type object

Taking from #21

files is an object listing all the files required to use the package. files are listed as arrays for each type of asset.

• scripts: Script files like JavaScript .js or CoffeeScript .coffee
• styles: Style files like CSS .css, Sass .scss, or LESS .less
• images: ...
• fonts: ...
• data
• templates
• (Others — we'd have to determine this spec)

Filepaths may include globs. yada yada yada ....

"files": {
  "scripts": [
    "js/motion.js",
    "js/run.js",
    "js/walk.js",
  ],
  "styles": [
    "sass/motion.scss",
    "sass/run.scss",
    "sass/walk.scss",
  ],
  "images": [
    "img/motion.png",
    "img/walk.png",
    "img/run.png",
  ],
  "fonts": [
    "fonts/icons.woff2",
    "fonts/icons.woff"
  ]
}

Pro: Clear about what files do what. Cons: More complex. We'd have to figure out the list of types. Might be confusing files that do not have an explicit type, like if there's a video file and we don't have a videos field.

Fun fact: recommended way back in 2013 bower/bower#935 (comment)

---

files would live along side main, not overwrite it. I'm not sure if it should be required or part of the bower init flow at this point.

Currently, I prefer Proposal 1. Easier for authors and for Bower spec.

Related issues

• Changing bower.json main as entry-point files, not distribution files taptapship/wiredep#180
• bower.json not including css file in main twbs/bootstrap#16663
• Wiredep bug - item.js missing in bower.json main metafizzy/outlayer#33

[HeroBart]

So the new "files" will hold all source files, and "main" will hold compiled versions again?
If so, I'd vote for proposal 1, it's simple enough.

[kkirsche]

With proposal 2 I think you have the potential to accept valid MIME types as the category list. By doing this, it's not up to you to figure out what kinds of files people need and you offload that to a different organization where it is getting close monitoring. Personally, I prefer proposal 2 as I feel there is more clarity which would be useful to users who are new or inexperienced with a project.

[alber70g]

Does this mean that the dist/movement.css used in the example of the current spec can be in main? Because if that's the case, nothing really changes. Except the hard-exclude of movement.css is out of the way.

[kkirsche]

@alber70g It doesn't read that way to me — this would put them in a separate files block. main would retain source files only is my understanding of the OP's post.

[HeroBart]

So we can expect half a year of wiredep not working and dependency mayhem... Nice.

[alber70g]

files is an array of all the files required to use the package. Filepaths may include globs.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

We recommend listing source files in files, rather than compiled distribution files.

The second line implies that we should keep only ONE file per filetype in main
The last line implies that we should put into files what is now in main in a sense that .less (or scss) is not allowed to be in main (which is the case, by looking at the current spec).

[nmalaguti]

Who is files meant to be consumed by? Any sufficiently advanced packer should be able to parse the assets listed in main and resolve all other dependencies (scripts, styles, images, fonts, etc.).

The problem is that naive dependency resolvers like wiredep don't have much file specific knowledge, they just want to resolve paths and group assets by extension. Depending on the file extension, they are injected into html or returned in an object (organized by extension/dependency) for further tools to consume.

There are several kinds of information I would find useful from a package (property names aren't important here):

• main: entry points for various file types that can be directly referenced from a browser. Not all file types support imports (e.g. ES5) and therefore concat + minification may be desirable for some.
• metadata: e.g. .d.ts or .less files with typing or variable information, but that compile down to nothing. Useful when doing development.
• sources: a list of source entry point files in the component that are needed for compilation (if no sources use imports/exports, then all source files should be listed). Don't list intermediate representations (e.g. ts -> js -> min.js, the js files should not be listed).
• assets: a list of static assets that are referenced by various other files (e.g. fonts, images, etc.) that should be copied into production so that unneeded files (sources, metadata) can be filtered out.

It may also be useful to offer different collections of files in the package that can be combined together with main as needed by the consumer (think various sizes, legacy support, languages):

{
  "main": [
    "js/motion-base.js"
  ],
  "options": {
    "lite": [
      "js/motion-lite.js"
    ],
    "full": [
      "js/motion-full.js"
    ],
    "huge": [
      "js/motion-huge.js"
    ],
    "legacy": [
      "js/motion-legacy.js"
    ]
  }
}

There would likely need to be some way for the consumer to specify which option(s) they want to be combined with main.

I think this gives enough information for a consumer to take the bower.json and pipe the various file lists into their tools of choice.

• metadata is used by the consumer to build their own projects/do development
• sources can go to a compiler (and then main should be ignored as you don't want to include stuff twice)
• main + options are injected as paths into html (or whatever you want to do with file paths)
• assets are retained when shipping to production so all unneccessary files can be stripped out (I already minified all js and therefore don't need to ship it)

[cvrebert]

• Re: Proposal 1: files flat list:
  • Couldn't tools just do ls -R . and then subtract out the ignore entries and get the same result? If yes, I don't see the point; Bower or the tools could just generate the list programmatically at runtime instead.
• Re: Proposal 2: asset-type object:
  • Better, but I think it would be good to tie precompiled CSS to the preprocessor file that it originated from, which this proposal doesn't do.
  • Example/strawman:

"files": {
  "styles": {
    "dist/css/bootstrap.css": "less/bootstrap.less",
    "dist/css/bootstrap-theme.css": "less/theme.less"
  }
}

Other issues:

• What about bootstrap.min.css.map?
  • Can we assume the <foo>.map naming convention and not explicitly list these and just have tools check for them?
• What about bootstrap.min.css and bootstrap.min.js?
  • Can we similarly assume the <foo>.min.<bar> naming convention?

[ezraroi]

I think proposal 1 is better, build tools will easily adopt it I believe

[stephenplusplus]

Proposal one works for wiredep 👍 I wouldn't vote for globs, however. It would just slow things down and the publisher should be able to explicitly list what files are important per release. They've been doing it with main already anyway.

[sheerun]

+1 for Option 1 files, that is compatible with both wiredep and NPM specification...

Mime-types of Option 2 can be easily extracted automatically..

I'd go without globs, but with special behavior of npm:

The "files" field is an array of files to include in your project. If you name a folder in the array, then it will also include the files inside that folder. (Unless they would be ignored by another rule.)

[cvrebert]

@sheerun What becomes the purpose/use of ignore then?

[sheerun]

@cvrebert It would be the same as .npmignore, but without separate file:

You can also provide a ".npmignore" file in the root of your package, which will keep files from being included, even if they would be picked up by the files array. The ".npmignore" file works just like a ".gitignore".

[cvrebert]

@sheerun Alrighty.

@paulmillr Does Brunch have any thoughts about the proposals?

[donaldpipowitch]

I'm fine with globs.

However... we should note what problems we want to solve here... and which not.

1. What is with minified/optimized code/assets?
2. What is with "dynamic" code/assets as seen in https://github.com/angular/bower-angular-i18n where we only need one locale-specific file?
3. What is our solution to different builds (compatible-modes for older browsers, etc.)? A current problem for RxJS Align bower.json with official spec Reactive-Extensions/RxJS#724.
4. What is with source maps?

Do we answer these questions here or in a different place? I would like to solve real problems here.

[HeroBart]

By the way, while we're at it, why don't we choose a more descriptive name, such as "sourceFiles" or "source".

[sheerun]

I've just read why you want files introduced..

I think Bower made mistake by encouraging to include only source files in main. Maybe we should roll-back a little and accept this pull-request: https://github.com/bower/bower.json-spec/pull/46/files

That said, I think files is good idea anyway, as a more explicit way of listing all files to include in package (not main entry-points, that's what main is for). It can be useful for packages without main files, like CSS libraries.

[HeroBart]

+1 for @sheerun, please do!

[donaldpipowitch]

Would this exclude Less files from "main"? I don't know if I like that :/

[sheerun]

@donaldpipowitch No, less can be library entry-point as well. If you use less, you probably would be more interested in consuming .less main files. If you use sass, you would be interested in consuming .sass or .scss. If you use plain html, you want to use .css main files. Bower for better of worse tries to be language-agnostic..

[donaldpipowitch]

Okay, thanks. I asked because of this line in the pull request:

"This means that the files can be used without compiling for use by the consuming party."

[benschwarz]

@calidion maybe a good idea to respect people and their ideas rather than posting what you just did.

[benschwarz]

@calidion You're way off-base. None of that gives you a right to treat people like dirt. Seeing as an apology is unlikely forthcoming, I'm going to mute this issue.

[calidion]

@benschwarz I apology for what i have said, but I hope as a base system, considering more on every changes you made, and what would be the subsequences for client projects.

[stevemao]

Just thought https://atom.io/docs/v1.0.3/hacking-atom-package-word-count#package-json might inspire you guys :)

[alexsaare]

+1 for files with flat list

As an aside - the changes to main also break this grunt task - https://github.com/yatskevich/grunt-bower-task - I think "files" is actually what this copy functionality needs rather than main.

[desandro]

After letting things cool off for a while... 🐧 I think ditching this proposal and reverting main back to including all files may be the way to go. This is already the common place practice. This will get Wiredep support back on track. As @notbrain points out, advocating for adopting a new field will likely have its own difficulties.

[sheerun]

As for me I'm convinced that listing only one file per extension in main, and extracting all the needed dependencies from them (using e.g. webpack) is the way to go. Listing needed files manually is too error prone and doesn't capture e.g. dependencies between assets..

[calidion]

Technologies are like products, when they fit for demands, they will grow, otherwise, they will die.
compatibility is the key to success for technologies, Anyone who breaks it will be published.

[kkirsche]

But breaking compatability is how new standards and best practices are made. To only look at the short term is short sightsighted and misses the big picture goals

[notbrain]

@kkirsche but what this boils down to is a naming convention preference. The main files list can be used as the de-facto standard for compiled/assets (as @stephenplusplus points out) and a new list files or whatever it will be called can be used to list all core source files. I'd call it sources or rawsrc or something since main still makes more sense as a list of files that allow me to use a package as a library, not as a main entry point to files that need further processing. We shouldn't need to process main files, especially if up until this change we didn't need to.

[alexsaare]

@desandro Whilst it might be simple for bower to take this approach, those who utilise bower will have to deal with the breaking change in specification. It doesn't just affect the direct consumers of bower, but those who use components which use bower (think grunt/gulp tasks). This is mostly a pain because there's no good mechanism for communicating the new specification to all consumers.

For me, Bower is a tool which allows devs to get on with the work that really matters to them... If any tool takes more effort to maintain than the primary goal, people just stop using the tool. This will be a real pain for CI builds.

It took a long time to settle on a specification. Personally I feel it would be unwise to change it now rather than extend it as originally proposed in this post.

[kkirsche]

Is there a reason to not just revert this change and re-release this version then do a major version bump to forcefully deprecate the old spec while moving to a new version with a new spec? I can only understand the frustration when there isn't a major version change. With one, sorry but breaking changes are the entire reason for major version numbers.

[desandro]

@kkirsche

Is there a reason to not just revert this change and re-release this version

Just to clear something up: we already changed main #43 this change would be add a new field files

@sheerun

As for me I'm convinced that listing only one file per extension in main, and extracting all the needed dependencies from them (using e.g. webpack) is the way to go. Listing needed files manually is too error prone and doesn't capture e.g. dependencies between assets.

Thanks for this confirmation. Current main spec follows this.

---

Now I'm flip-flopping on my previous comment. "main listing one-per-type" is a good change. It does break backwards compatibility with some tools like Wiredep. However, by adding a new field, these tools can maintain compatibility with packages that use main the old way. A fix would look like checking for files before using main.

// use files field if there, otherwise fall back to main field
var files = manifestJson.files || manifestJson.main

[KrisSiegel]

I was looking up to see if bower contained something similar to npm's file list to simplify my module (I'd rather be explicit with what I want included versus maintaining an ever growing list of ignores) which ended with my stumbling across this thread. This issue has been open since June and adding a files property would bring yet another thing of bower's to parity with npm; why wasn't that simply done and merged in? This is just a huge issue of arguments over using main or source or sourceFiles or min vs full vs other assets.

Just make a file list that includes everything that should be downloaded by bower. When a developer wants to use one file versus another, in their deployment scripts they can decide what is or isn't taken and that's it. That's how most other package managers do it and it's easy. The simpler the better.

Besides the more parity with how npm works the easier bower/bower#1520 is to close (in one direction or another).

[sheerun]

npm's files is something different than proposed here (it's simply an inverse of ignore)

We could include it as well, but I don't like attaching it the meaning of "of all the files required to use the package". I especially don't like the idea of build tools using this list to select files to auto-require in project.

If we introduce files as just inverse of ignore, I'm perfectly fine with that.

If wiredep wants to interpret this list as all files to include in project, I cannot stop them from doing so, but I think it's a wrong approach. It's better to extract dependencies with tools like webpack.

How files would be implemented in bower? Any files not listed in files would be removed after downloading project. It means files needs to list even optional dependencies. Just like npm does it.

Additionally we'll make it clear files don't specify the order to load these files (some people assume main does it...). files will be non-ordered list files to include in the package. It can include globs.

Are you OK with it?

[KrisSiegel]

attaching it the meaning of "of all the files required to use the package"

But that's not what it means in the npm world; those are simply the files bundled with an npm install and nothing more. Yes some are required but not all. You typically pull down more files than what is required (documentation, sometimes examples, etc).

If we introduce files as just inverse of ignore, I'm perfectly fine with that.

👍 ship it :). I'd work on the merge request but I'm unlikely to have time for a few more weeks (though if no one does it I'll certainly just do it).

[KiT106]

+1 for Proposal 2

[OmgImAlexis]

@desandro after a year and a half has anything happened to this?

[desandro]

@OmgImAlexis No

[kkirsche]

It's a bit rude to link to that. While true, no reason to have to come off as a jerk about saying no.

[OmgImAlexis]

The reason I ask is I ended up in in a 20+ page rabbit hole to find this page with multiple different projects relying on each other's issues. I wouldn't have asked otherwise.

[sheerun]

@OmgImAlexis A year later, I'd say the recommended approach for developers is to use webpack to automatically discover assets that really need to be bundled. If you're package author, you can still use ignore field of bower.json to exclude unnecessary files.