Assets / packages management

[samdark]

Publishing a code snippet

\Yii::app()->clientScript->registerScript('name', 'some encoded js');

Packages

There should be no way to publish a file without defining a package for it in order
to support combining and processing assets. All the core classes, extensions and
modules should use packages.

Package definition

return array(
    'package1' => array(
        'basePath' => '@app/scripts'
        'map' => array(
            '*.js' => 'all.js', // append all JS into single file
            '*.ts' => 'all.js', // append all TypeScript into single file
        ),
        'process' => array(
            // how to deal with *.ts
            '*.ts' => function($fileName, $outputDir) {
                exec('…');
            }
        ),
    ),
    'package2' => array(
        'basePath' => '@app/styles',
        'map' => array(
            '*.less', // do not map, just include
        ),
        'process' => array(
            '*.less' => function($fileName, $outputDir) {
                exec(…);
            }
        ),
        'depends' => array(
            'package1',
        ),
    ),
);

Asset mappings

The idea of mappings is to be able to map one or more source files into same of less
number of destination files. Typically we have a number of source assets such as
a.js, b.js, c.js, d.js. We need to be able to map:

1. Every *.js to all.js.
2. a.js and b.js to ab.js while c.js and d.js to cd.js.

If all these from point 2 are separate packages, then if a is required ab.js
will be included in the page source.

By default each asset maps to the same named file.

Processing

1. Determine the order of the scripts based on dependencies.
2. Build ordered list.
3. Iterate list, write contents of each file from the file itself to the corresponding map file.

Adding package on the fly

\Yii::app()->clientScript->addPackage('name', array(...definition...));

Adding package to the page

\Yii::app()->clientScript->registerPackage('name');

When doing it:

• Check if package already registered. If yes, skip it.
• Check if package has dependencies. If yes, process these one by one first.
• Build a list of files from 'map'.
• Iterate over 'map'. For each record:
  • Find matching files. Check if these are matching processing rules. If yes,
    process file.
  • After file is processed (or processing is skipped), write it to the corresponding
    file from 'map' or the the same named file if not defined.

Package processing

Common tasks needed for resources:

1. Process resource content, write to the corresponding file. Processing can be either
   internal or external. Should be customizable per resource. Should support wildcards.
2. Map multiple resources into single file. Support either just mapping or actual
   writing.
3. Ability to specify how resource (or a wildcard) should be published
   (i.e. define a place on the page where it should be written).
4. Ability to define how exactly resources are included and in which order.

Extensions

If extension registers any resources it should be specified in application's
extensions section of the config. Each record will point to extension's config.

The whole process should be described in the extension's readme.md. Will be automated
in future in case of installing via Composer.

Modules

We have a list of currently enabled modules in the application config (modules)
so we can get package definitions from module config file located at moduleName/config/packages.php.

[qiangxue]

What does "process" do? Can you show a detailed example? Putting anonymous functions in configuration array is not a very good approach because anonymous functions cannot be serialized and thus the whole configuration array cannot be serialized and cached.

[samdark]

Process takes the content of the source file, converts it and gives it back. It can be minification or preprocessing (LESS, TypeScript). Since we can't predict what processing is required we should provide a way customizing it.

Another variant I have in mind is to introduce IAssetProcessor interface and implement basic stuff, like CSS and JavaScript minification, out of the box. In config files there will just an array of such class aliases.

[qiangxue]

Most likely minification/preprocessing requires some external tools/commands, and writing such anonymous functions means redundant code and unnecessary dependency.

Some predefined processor classes seem fine to me. But then we need to decide: what are these processors? should they be applied by default on certain script files? If not, what will happen?

[samdark]

The most useful ones are:

1. Compress CSS.
2. Compress JS.

Also it would be handy to have:

1. Transform LESS into CSS.
2. Transform SCSS into CSS.
3. Transform TypeScript into JavaScript.

I don't think any of these should be applied by default. If no processing is applied file contents are used as is.

[qiangxue]

I think compressing CSS/JS should be done outside rather than via processor.
We may limit the scope of a processor to be: converting .* files into CSS or JS files.

[samdark]

Well, this "outside" thing should be configurable as well and I thought about using the same package config for it.

What's bad in handling compression in the same way btw.? I see no difference in translating LESS into CSS or translating CSS into compressed CSS.

[qiangxue]

After compressing, you still need combining to make it truly meaningful.
And toggling between compressing and uncompressing is often determined by the deployment mode - you don't want to see compressed scripts while you are developing. So in the end, if you do compressing for some of the packages via the processor while not for some other packages, you end up with mixed results.

[samdark]

For this purpose there will be getIsEnabled and setIsEnabled in each IAssetProcessor so each processor can be configured like the following:

array(
  '*.ts' => array(
    'class' => '\yii\asset\processors\TypeScript',
    'enabled' => !YII_DEBUG,
  ),
),

By default enabled is true.

To do the same for combining scripts we can introduce combineAssets flag in clientScript component.

[qiangxue]

These are global settings, a different story. What about settings per package that we are talking about? Do you still think compressing should be a processor? IMO, the definition of processor is to turn non-standard scripts into standard ones (js/css). Then we rely on external tools (depend on some global settings) to combine/compress them.

[samdark]

Yes, it should be a processor. There are lots of ways you can compress CSS and JavaScript. Some give better compression rates, some are safer, some are with options.

Processor is just something that converts input into something different. I think compressor fits there very well. I don't think we should write processor routine itself but use existing ones so the implementation of IAssetProcessor will be, most probably, just calling a few external commands with config translated into parameters for these commands.

[samdark]

btw., processing isn't just CSS and JavaScript. We will not provide it out of the box but this processing can probably be applied to image resources (i.e. stripping down metadata, recompressing PNGs etc.)

[qiangxue]

I thought more about the above proposal. One thing I'm still not clear is how we will call the processors: should it be done on-the-fly or offline?

Also, the proposal above mainly applies to native application code. For extensions, it has two crucial problems:

1. unable to specify package dependency between two extensions since package names are something internal.
2. unable to specify the script mapping since the target scripts are in the application domain which is out of control of individual extensions.

Since script and asset management is mainly meaningful for extension, we need to introduce more rules to resolve the above problems. Here is I'm thinking about:

1. each extension can only have one package, and the package name should be the same as the extension name.
2. the package should be declared as a separate configuration file with a fixed file name under the root folder of the extension;
3. the script combining and compressing is a global process, and we should rely on some global configuration to specify the mapping from individual packages to combined scripts;

[bwoester]

I think there are good arguments for both on-the-fly processing and offline processing of assets. During development, it's super comfortable to hack on your LESS/SCSS files or CoffeeScript sources, reload the browser and see the changes. On my live server, I definitely don't want that, though. Especially not, if the processing includes any exec calls. They might even be disabled. So I wouldn't restrict the use of processors to either one or the other approach.

[creocoder]

each extension can only have one package, and the package name should be the same as the extension name

Why? What if this some complex extension?

the package should be declared as a separate configuration file with a fixed file name under the root folder of the extension

Why separate file?

[samdark]

Why separate file?

The reason is not to instantiate all the extensions while building combined script / css.

Why? What if this some complex extension?

Any examples?

[bwoester]

each extension can only have one package, and the package name should be the same as the extension name

Why? What if this some complex extension?

Any examples?

Probably modules. If you think of them as "embedded applications" like say a wiki, a forum or an image gallery, then I guess they might very well grow large enough to justify several packages. Basically, if any application is considered to use several packages, then extensions should have the same freedom. And what's wrong if they define packages like:

• gallery.categoryList
• gallery.category
• gallery.settings

And I wouldn't consider those package names "internal". If an extension is released, it should be documented what packages it defines and what purpose they serve. They're part of the extension's API.

I'm not sure about the thing with package configurations in a separate file. After all, the configuration is data, and where that data comes from shouldn't matter. Could also come from app config or user input. If we're only talking about defaults, then I think a separate file is good.

Oh, and you don't plan to implement some fancy async lazy on demand loading of scripts, do you?
The whole idea is to collect everything you need, compile it into as little scripts/ stylesheets as possible and to reduce the number of requests, correct? Obviously, you wouldn't want to map libs like jQuery into different packages, but I guess that is configuration work then...

[samdark]

@bwoester modules are a special case where no such problem exists since we know for sure which modules are enabled.

How files are combined will be configurable.

[creocoder]

Any examples?

Extension with 5 widgets for example. So it needs 5 packages. This is not compatible with

each extension can only have one package

[samdark]

You can implement it as a set of extensions. Each with one widget.

[bwoester]

Why the restriction at all? What's the problem with extensions defining multiple packages?

[samdark]

If it will never be used and there's a better way we prefer not to allow doing it.

[schmunk42]

It would be nice, if you could setup a small glossary about the terminology which is used for Yii2.
Like a package is usually referred as a collection of client-script files, but also sometimes as a composer package.

Some words I'd like to see clarified: application, assets, configuration, installation, extension, namespace, alias, package, module, extensions/, vendor/.

[qiangxue]

We will use the name asset bundle instead of package. Other than this, there're very few new terms introduced in Yii 2 as we want to make the transition from Yii 1 to Yii 2 as smooth as possible.

[samdark]

@schmunk42 https://github.com/yiisoft/yii2/wiki/Glossary feel free to add new terms and edit / request descriptions.

[qiangxue]

Finally I have done the design and implementation of asset and script management, except the implementation of concrete asset processor classes. I'd like to share with you some of the designs so that you may have better idea about the new extension strategy.

First, we will embrace Composer as a way to install extensions in Yii 2. By default, all extensions will be installed under the "@app/vendor" directory. Here "@app" stands for the application directory. Each extension has a unique name, in the format of "CompanyName/ProductName", following the convention recommended by Composer.

We will introduce the new concept called asset bundle, which is similar to the package concept in 1.1. Each asset bundle is represented in terms of an AssetBundle object which may be configured using the following array:

'bundleName' => array(
   'basePath' => ...,
   'baseUrl' => ...,
   'js' => array(...list of js files...),
   'css' => array(...list of css files..),
   'depends' => array(...list of bundles that this bundle depends on...)
)

For application code, asset bundles are declared by AssetManager::bundles. For an extension, they are listed in the assets.php file under its root source code directory.

To use a bundle, simply call $this->page->registerAssets($name), where $this refers to the current View object (in a view file).

Conversion from special types of asset files (e.g. LESS, Sass, TypeScript) is supported via the processor design that Alex proposed. In particular, AssetManager::processors takes the configurations for supported processors. A processor will convert an asset file into a JS or CSS file when the bundle containing the asset file is being used.

Script combining and compression is done via an offline command that will be included in the core release. The command will do the actual job and generate a new list of bundle declarations. By setting AssetManager::bundles with this list, pages will automatically use the combined/compressed versions of the scripts.

[qiangxue]

Done.

[schmunk42]

btw: 💯 👍 for the stuff I see so far