Javascript Bundling produces huge 13MB js files

[VegasJames]

Steps to reproduce

1. Install Magento via composer. v2.0.5
2. Use admin to enable Javascript Bundling. (Stores -> Configuration -> Advanced -> Developer -> JavaScript Settings) and set Merge JavaScript Files and Enable Javascript Bundling to Yes
3. Left "Minify JavaScript Files" to No so the file could be inspected, but even with it as Yes the bundle0.min.js files is 8MB.
4. Had to run bin/magento setup:di:compile
5. Then did bin/magento setup:static-content:deploy
6. Observe the size of the generated bundle0.js file.

Expected result

Before bundling Chrome showed about 515KB of javascript being loaded, bundling the files in to one should result in a file of a similar size, not 25 times as big.

Actual result
ls -lh pub/static/frontend/Magento/blank/en_US/js/bundle/
total 13M
-rw-rw-rw-. 1 magento apache 13M May 14 07:23 bundle0.js

I've investigated the content of the file somewhat. After a lot of valid javascript we get to this bit:
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJz (truncated)
This goes on and on with random characters for 1,414,372 bytes.
The rest of the file then appears to be valid javascript, so unfortunately the above base64 only explains about 10% of the bloat, so I'm not sure where the rest comes from, but certainly this seems wrong.

Additionally, this is more of a feature request, but can the bundling leave out jquery and jquery-ui?
These are the two largest files accounting for 700KB of javascript and almost 200KB of data transferred, and they also could be fetched from Google CDN instead:
https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js
https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.4/jquery-ui.min.js

[pboisvert]

Chuck--can you get the team to investigate given the potential for performance impact if reproducible?

[rudik99]

I also suffered the same problem on the same version installed via composer.

[viktor-zhuromskyy]

Same thing here. 👎

[viktor-zhuromskyy]

Not even that! On 2.0.5 after all static content is compiled without any errors, there is a fatal error on whole Magento installation dues to errors in DOM bundling. With the JS Bundling OFF everything works normal.

Please see screenshots.

[choukalos]

I'll check with the team re bundle size. I believe we implemented a very simple bundling algorithm so it's possible we're pulling in some additional files in the bundle.

Is anyone else seeing bundle.js errors on 2.0.5? @devdesco-ceo did you create a separate Github issue for the bundle error on 2.0.5?

[vkorotun]

@VegasJames It seems like your theme doesn't inherit from Blank or Luma Magento native themes, so exclude list for JS files bundling is missed. You need to configre the exclude list for your custom theme. As a reference example you can have a look into configuration for Blank or Luma theme:

• /app/design/frontend/Magento/blank/etc/view.xml
• /app/design/frontend/Magento/luma/etc/view.xml

See "exclude" node content.

[choukalos]

@VegasJames Please close or let us know if that didn't address your issue.
@devdesco-ceo What themes do you have installed?

[viktor-zhuromskyy]

I have Porto theme installed and my theme is a child theme of Porto which is based on Blank.
my /etc/view.xml file has all proper exclude statements for the JS files.

[VegasJames]

incorrect, this is a stock Magento install. the bundle file inside both the
blank and Luma theme directories is 13MB

On Wednesday, May 18, 2016, Chuck Choukalos notifications@github.com
wrote:

@VegasJames https://github.com/VegasJames Please close or let us know
if that didn't address your issue.
@devdesco-ceo https://github.com/devdesco-ceo What themes do you have
installed?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4506 (comment)

[TommyKolkman]

I can second VegasJames's problem on 2.0.7. Also, my theme inherits correctly.

[atIOCrON]

@vkorotun we have the same issue verified with Magento 2.0.6 using the default Luma theme:

Without JS Bundling (453.8kb): http://tools.pingdom.com/fpt/#!/czoIzm/https://new.betterbatt.com.au/
With JS Bundling (2.1mb): http://tools.pingdom.com/fpt/#!/c8CxvG/https://new.betterbatt.com.au/

As you can see JS Bundling increases page size by 4x.

I also noticed that JS Bundling when enabled blocks the selection of role resources when adding a custom admin role.

[erikhansen]

From what I've read here and from my independent testing, it seems like the "Enable Javascript Bundling" is currently unusable and should not be recommended for production environments. The way bundling has been implemented for M2 negates much of the benefits gained by RequireJS.

I would recommend the following:

• To prevent developers from enabling bundling without understanding its ramifications, in the next patch release, remove the "Enable Javascript Bundling" option or at least add a disclaimer beneath the option alerting the admin that the feature is not yet recommended for production.
• Refactor the JS bundling functionality so that rather than bundling all of the JS files into a massive ~8MB file, the bundler should only bundle JS files that are known to be required by all pages. The logic behind how the bundler should know what files to include is up for discussion, but the <exclude> logic in view.xml could possibly be inverted to use an <include> logic so that only files/directories in this node would be bundled.

[hostep]

And let me just add one small addition: if your webserver is capable of running HTTP/2 (which will become more and more the case) it is really not recommended to enable bundling. Since HTTP/2 hasn't got performance problems with transferring a lot of smaller files.

[southerncomputer]

Expected '}'
File: bef36da44feee702d670095d789f94f7.min.js, Line: 136, Column: 49365

Getting this error on IE and Firefox plus admin section no longer functioning after enabling bundling on 2.0.6 !

This seems about as reliable as the old magento 1 compiler ! Having to load +140 .js files due to this feature not working is severely bad!

[erikhansen]

@southerncomputer Your browser will cache the JS files, so while the first page will load more slowly, subsequent pages will use the cached JS files. And since the JS is being loaded asynchronously via RequireJS, the page will render before the JS files are loaded.

That being said, it does seem a bit much to have ~118 JS files loaded for a vanilla M2 homepage—thus why I'd like to see a smarter bundling feature.

[southerncomputer]

#3040 this is a duplicate ?

[erikhansen]

@southerncomputer No, #3040 is not a duplicate issue of this issue. Although the two issues are somewhat related in that they deal with JS minification/bundling.

However the issue you described in your comment above might be caused by the issue reported in #3040.

[southerncomputer]

https://github.com/magento/magento2/blob/6ea7d2d85cded3fa0fbcf4e7aa0dcd4edbf568a6/lib/web/mage/backend/validation.js

this file throws an error in the bundle regardless if you disable jshrink or not and have bundling enabled. I will try to manually minify this file to see where the file is causing issues in bundling.

The result is in IE11/FF it will throw an unmatched } error on the developer tools console, thus rendering the big bundle unusable for the most part!

[Zaylril]

I have to agree with @erikhansen in that in it's current state the bundling feature is simply not usable. Just think here for a second for a mobile user on 3G, even with the 13meg file gzipped it comes out at a whopping 3.3meg. Add this to the rest of the page and you are looking at 5meg+ for an initial page load. Not great.

Surely a much better solution would be to only bundle files which would be used on every page - or have a config where you could add files you wanted to be bundled together - whether this be automatic or not - meaning you would get some benefit from bundling files for a common.js file and then load the rest as and when they are needed.

Additionally I'm also not sure why the requireJS optimizer is not being implemented - where it combines related scripts together to build multiple bundles which are included as and when they are needed vastly reducing the number of individual files loaded, and greatly increasing performance.

Of course there is a balance to be had between file size and the number of files loaded - but as already mentioned when http2 is implemented this becomes a non-issue, however I can't really see this happening for everyone and all devices in the near future.

I think the only solution is really to create a smarter bundling feature - similar to that of the requireJS optimizer or implementing a build step where smarter bundling occurs. In this way you would reduce the number of requests down for each page and also reduce the file size as well - as was the initial intention of loaded each file separately. This really needs fixing though - especially on the checkout where there is something like 250 JS files loaded - not to mention the .html templates being loaded via AJAX too. Not the best for performance. The worst thing about this also is you can't really implement any kind of server cache on all those JS files to speed it up - as it totally relies on the browser.

[erikhansen]

@Zaylril Thanks for adding to the discussion.

@alankent I'd be curious to hear your thoughts on this issue.

[southerncomputer]

2.1.0-rc2 now creates 5 bundle0.js .. bundle5.js - not that is saving much space but breaks up the files to concurrently load?

[Khaleel]

I agree. Webpack. Gulp. Many better tools than RequireJS there is still so much bloat in Ajax calls, phtml, knockout and then requireJS - cannot do anything until requireJS has fired every ajax and knockout call.. then mixed in with localStorage - get back to American developers

[dunarri]

@viktor-zhuromskyy That is the size it should be, is that the result on v2.1.3?

[dunarri]

I'm the OP (changed name) and have retested the issue on the latest Magento v2.1.5.

With the merge and bundling options turned on (also note Magento has to be in production mode to use these settings) I now get 61 requests and 1.1MB transferred on a category page for about 5s load time.

The merged js file is still kinda large at 3.4MB uncompressed and this can be seen by
ls -lh pub/static/_cache/merged/

By comparison, testing the same page with merge and bundling turned off I get 200 requests, 131 of which are JS, and 1.0MB transferred. So it would appear they have addressed the issue.
Here's the kicker though, with merge and bundling turned off my page load time was only ~3s. So it doesn't achieve the desired result of faster page loads anyway. And I'm running HTTP/1.1, I'd imagine it gets even better with HTTP/2.

Also, as I stated originally the bundle includes the large JS libraries that could be offloaded to CDN (jQuery, jQuery-UI, Knockout, RequireJS).

In conclusion, don't use the bundling feature.
If you want to improve page load, get large JS libraries from a CDN and upgrade your web server to support HTTP/2.

[pynej]

The optimal setup is something like so:

HTTP/2 Nginx Proxy to Varnish Cache to Apache
With compression, minification, and etags/expiration manually set in apache as magenta doesn't so them all properly. At this point you can turn of file compilation but in reality the site really shouldn't be needing 131 separate JS @ 1MB files to build a simple page.

The only real issues is that due to how Magento does it's url rewriting rules I can't seam to get full site SSL working, and not just the admin/checkout pages.

[mombrea]

@pynej for full site ssl, try setting both the base url and base secure url to https

php bin/magento setup:store-config:set --base-url="https://example.com/"
php bin/magento setup:store-config:set --base-url-secure="https://example.com/"
php bin/magento cache:flush

[pynej]

Oh I'v tried that, The system gets really pissy and force rewrites URL's now matter how I do it when in production mode due to the HTTP Proxy layer. Oddly it seamed to work in standard mode.

[zetlen]

@dunarri and anyone else with these unacceptably large bundles: do you still have the inline sourceMappingUrl followed by lots of base64 at the end of your bundled file?

[benheart]

Hello guys, is there anyone solved this issue? I minified and merged the Javascript files and enable Javascript Bundling, but the bundled file is still very large(1M+), so home page load very slowly. I consider that can we use asynchronous js or move bundled js file to the foot of html body? But I don't know how to do that, if someone has good methods to solve this issue, please tell me, thanks very much!

[Khaleel]

@hwj727 One of the things we are doing:

• Minifiying all HTML
• Minifying all CSS
• Serving CSS on CDN
• Serving Images on CDN

I am going to try serving JS on the CDN too but there is a lot of JS which has PHP inside of it - such as Knockout files etc - so not sure about that. Magento bundling is poor

To place files at the bottom will only help a small amount as it is knockout dom rendering so it will make you site look funny - this also affects data-mage-init calls.. seriously.. Magento needs to get their shit together with this Front End.. especially the cart.. no documentation on serious front end development at all..

To move JS at the bottom edit the XML files in Magento Vendor files and theme XML files

[zetlen]

@hwj727 and @Khaleel: I'm sorry you're having so much trouble. Can I ask if either of you see the sourceMappingUrl section in your large JavaScript bundles?

[zchking]

2.2M after merge Magento 2.1.6, I can accept it:)

[Khaleel]

Even with bundling on Magento 2 are not using RequireJS optimiser so HTTP calls will still be over 40+ to 50+ for calling JS files (async or not async) the M2 default still has a heavy HTTP call..

[SchumacherFM]

Same here ... it's just a big PITA

[lingwooc]

I have 2.6mb merged .min.js which isn't too bad. It doesn't gzip though which rather spoils it. I have mod_deflate setup in the magento root .htaccess and it works for other files in _cache/merged. Any fixes?

Thinking about it, why aren't static files pre-gzipped when they are deployed rather than done on the fly each time?
edit: I'm using a cdn now which resolves the gzipping issue (as well as being generally better)

[magento-engcom-team]

@VegasJames thank you for your report.
The fix for this issue is already fixed in 2.0.11 release

[antonkril]

More efficient r.js-based bundling solution: https://github.com/antonkril/magento-rjs-config

[thaddeusmt]

FWIW, commenting on a closed bug... in 2.2.2 with JS bundling on (but not minification) the single JS file being output is 5.3MB right now. And our theme has bundle_size = 1MB set.

[antonkril]

Official bundling recommendation
https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html

[erikhansen]

@antonkril Thanks for sharing that detailed writeup about how to optimized Magento's bundling. I look forward to going through those steps for some of the Magento sites I'm working on. Due to the work involved with creating customized bundles, I expect the most merchants/developers will not implement them.

1. Do you anticipate a future core Magento release to ship with this type of configuration built-in?

2. Do you know of any Gist or Github pages that share bundle configurations for a vanilla Magento 2.2.x site?

[SKovbel]

2 ways

1. Duplicate each row with .min. in view.xml, exclude section

        <item type="file">Magento_Customer::js/zxcvbn.js</item>
        <item type="file">Magento_Customer::js/zxcvbn.min.js</item>

2. Fixing function isExcluded, ignore .min. suffix in comparasion
   if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {

diff --git a/vendor/magento/module-deploy/Service/Bundle.php b/vendor/magento/module-deploy/Service/Bundle.php
index cf0d7a551..f661a69d9 100644
--- a/vendor/magento/module-deploy/Service/Bundle.php
+++ b/vendor/magento/module-deploy/Service/Bundle.php
@@ -193,7 +193,7 @@ class Bundle
         $excludedFiles = $this->bundleConfig->getExcludedFiles($area, $theme);
         foreach ($excludedFiles as $excludedFileId) {
             $excludedFilePath = $this->prepareExcludePath($excludedFileId);
-            if ($excludedFilePath === $filePath) {
+            if ($excludedFilePath === $filePath || $excludedFilePath === str_replace('.min.js', '.js', $filePath)) {
                 return true;
             }
         }

[webspeaks]

For me the culprit is JS bundling. I was seeing huge JS file (minified and bundled) even in admin side. Admin was too slow. Similarly frontend was slow due to huge JS file size.

Bundling enabled:
Admin minified and bundled js size: 17.2MB

Bundling disabled:
Admin minified and bundled js size: 100kb

That's a disastrous different for me. I am using Magento 2.3.0. So I turned off js bundling (even in production mode) and life is smooth now.

Disable js bundling:
php -f bin/magento config:set dev/js/enable_js_bundling 0

[magento-engcom-team]

Hi @VegasJames. Thank you for your report.
The issue has been fixed in #24506 by @hostep in 2.3-develop branch
Related commit(s):

• 41ef34feca73eea48f3dd1744bce529d36909c91
• cbe25f4960b59feed344eebee25e3dbe2f3efaaa

The fix will be available with the upcoming 2.3.4 release.