Gzip compressed JavaScript file is corrupted

[krisztianb]

Hello, it is me again. :-)

I have a problem with the following .js file:

function fileQueueError(file, errorCode, message) {
    try {
        var errorMessage = "";

        switch (errorCode) {
            case SWFUpload.QUEUE_ERROR.QUEUE_LIMIT_EXCEEDED:
                errorMessage = "[[[Sie wollen zu viele Bilder auf einmal hochladen. Löschen Sie bereits vorhandene Bilder oder wählen Sie weniger Bilder aus!]]]";
                break;
            case SWFUpload.QUEUE_ERROR.ZERO_BYTE_FILE:
                errorMessage = "Die ausgewaehlte Datei " + file + " ist ungueltig.\nWaehlen Sie eine andere Datei aus.";
                break;
            case SWFUpload.QUEUE_ERROR.FILE_EXCEEDS_SIZE_LIMIT:
                errorMessage = "Das ausgewaehlte Bild " + file + " ist zu gross!\nEs sind maximal 5MB pro Bild erlaubt.";
                break;
            case SWFUpload.QUEUE_ERROR.INVALID_FILETYPE:
                errorMessage = "Die ausgewaehlte Datei " + file + " ist ungeueltig.\nNur jpg-Dateien sind erlaubt.";
                break;
            default:
                errorMessage = "code: " + errorCode + " mess: " + message;
                break;
        }

        alert(errorMessage);
    } catch (ex) {
        this.debug(ex);
    }
}

I checked everything I could:

• file is UTF-8 encoded
• i18n.WhiteList contains *.js in the web.config file
• message is put into the correct messages.po file
• message is translated within the messages.po file
• i18n.LocalizedApplication.Current.ContentTypesToLocalize is not set

and still the message is not being translated.

This is the response header:

Accept-Ranges   bytes
Content-Length  10124
Content-Type    application/x-javascript
Date    Thu, 28 Nov 2013 11:15:14 GMT
Etag    "2d7284c128ecce1:0"
Last-Modified   Thu, 28 Nov 2013 10:58:22 GMT
Server  Microsoft-IIS/7.5
X-Powered-By    ASP.NET

Do you have any ideas what could possibly be wrong?

[turquoiseowl]

It is probably the Content-Type again as "application/x-javascript" won't be picked up by default. If so, this should get it working:

        protected void Application_Start()
        {
            ...
            i18n.LocalizedApplication.Current.ContentTypesToLocalize = new System.Text.RegularExpressions.Regex(@"^(?:(?:(?:text|application)/(?:plain|html|xml|javascript|x-javascript|json))(?:\s*;.*)?)$");
        }

I'll add it to the default setting if that is it.

[krisztianb]

Unfortunately this does not solve the problem.

I also tried this:

i18n.LocalizedApplication.Current.ContentTypesToLocalize = new System.Text.RegularExpressions.Regex("^(?:text/html)|(?:application/json)|(?:application/x-javascript)$");

But the translation is not working.

[turquoiseowl]

Have you got other javascript with translations that do work?

[krisztianb]

No, but I have a simple HTML file that is translated correctly.

[turquoiseowl]

I'm suspicious of the fact that your context-type is "x-javascript" because I've not seen that before with ASP.NET: it's normally just "javascript". Any idea why that might be?

There's no charset indicated in the response header. Are you sure it's UTF-8?

[krisztianb]

@ content-type:
The content type "application/x-javascript" is the default content-type used by IIS 7.5 when delivering .js files. I did not specify that nor did I alter any IIS settings. So it should probably be put into the list of content types handled by default.

@ encoding:
The file is UTF-8 encoded including a signature. The encoding itself is not returned by IIS in the response header. But it is the same for HTML files. I don't know why? Maybe this is normal?

Back to the problem: I found the cause but I don't understand why it was happening.

The problem seemed to be IIS. The application was running on IIS, but it was not at all configured within the IIS manager. I don't even understand how it could run because it is not located in the default IIS directory.

I created a new website within IIS and specified the application's root folder. I added a binding to a random local domain name like "local.test.project" and created a static DNS entry in my HOSTS file. Accessing the site with "local.test.project" without modifying anything else; the translation was working.

The domain name used should not be relevant, should it? First I was accessing the web application like this:

http://localhost/admin/index.aspx

And now it looks like this:

http://local.test.project/index.aspx

This should be totally irrelevant, right? So I guess it must have been some strange thing happening within IIS. All .aspx and .ascx files were correctly translated, but .html and .js files not.

To sum it up: It now works correctly. Thank you once again for your help.

[krisztianb]

Update: I changed the project's settings to use the Visual Studio Development Server as web server:

 http://localhost:52716/index.aspx

The translation is still working correctly. So it really must have been some weird IIS problem.

[krisztianb]

I just want to emphasize one totally awesome thing that is working with this i18n module that I jused realized using the translation for JavaScript files.

Translation is normally done like this in your markup:

[[[This is a simple text to translate.]]]

If you need to have placeholders you can use any of these on the server side code:

label.Text = String.Format(Lang.GetText("[[[The file {0} is not valid!]]]"), filename);

or using the nugget parametrized syntax (which is a little harder to read)

label.Text = String.Format(Lang.GetText("[[[The file %0 is not valid!||||{0}]]]"), filename);

But what do you do if you have placeholders on the client side (eg. JavaScript files)? You can do this:

errorMessage = "[[[The file %0 is not valid!||||" + file.name + "]]]";

First you would look at this and think, this can't work. But it does! Why? Because the following happens:

1. The translation is applied by the i18n module and the text is translated to "[[[The file " + file.name + " is not valid!]]]"
2. This is perfectly valid JavaScript which looks exectly like what your normal non localized message was. The whole string between ||| and ]]] including all spaces and apostrophes is inserted into the placeholder and creates valid JavaScript.

Total awesomeness. :-)

[krisztianb]

Soon after the solution above, I was facing another problem.

If I add "application/x-javascript" to the ContentTypesToLocalize then the module is also altering gzip-compressed JavaScript content provided by the ASP.NET AJAX framework.

I get error messages like this: "ASP.NET Ajax client-side framework failed to load." in the JavaScript console.

Adding the following lines into the web.config file solved the problem:

<system.web.extensions>
   <scripting>
      <scriptResourceHandler enableCompression="false" />
    </scripting>
</system.web.extensions>

Is it possible for the module to check whether the content is compressed and only process it when it is not compressed?

[turquoiseowl]

It would help to see the headers for the responses you are working with along with descriptions of the content the response contains.

From: krisztianb [mailto:notifications@github.com]
Sent: 02 December 2013 12:19
To: turquoiseowl/i18n
Cc: Martin Connell
Subject: [SPAM] Re: [i18n] JavaScript file is not translated (#109)

Soon after the solution above, I was facing another problem.

If I add "application/x-javascript" to the ContentTypesToLocalize then the module is also altering gzip-compressed JavaScript content provided by the ASP.NET AJAX framework.

I get error messages like this: "ASP.NET Ajax client-side framework failed to load." in the JavaScript console.

Adding the following lines into the web.config file solved the problem:

<system.web.extensions>



</system.web.extensions>

Is it possible for the module to check whether the content is compressed and only process it when it is not compressed?

—
Reply to this email directly or view it on GitHub #109 (comment) . https://github.com/notifications/beacon/xL73SQRLzck8DJI41v2Mt6aJG7ERa7iXp2-ITaTwmwMFGjvZ1xh61IpmKcmGRveL.gif

[krisztianb]

I removed the web.config settings to see the HTTP request and response data.

This is the response header:

HTTP/1.1 200 OK
Server: ASP.NET Development Server/11.0.0.0
Date: Mon, 02 Dec 2013 12:35:36 GMT
X-AspNet-Version: 4.0.30319
Content-Encoding: gzip
Cache-Control: public
Expires: Tue, 02 Dec 2014 12:35:35 GMT
Last-Modified: Mon, 02 Dec 2013 12:35:35 GMT
Content-Type: application/x-javascript
Content-Length: 28069
Connection: Close

The response can't be displayed as the browser says "content-encoding error". FireBug shows the following text when I try to access the response tab:

Reload the page to get source for: http://localhost:52716/ScriptResource.axd?d=dFe7KUtdT3q9FEDOBGrkIJhxvJoNpHlY2Oj33Y-NGB8wpYiph6SmN8bZUkQiUUbQBG5gGFcSKFvkPYIMQtApCkXYz35_FV9B6HAlIry9MNdXvurNqqOGwQeIeepM6XrPlGos4B8uA7CnCDtJEs8Z9ZE9Vj9vfm8kNHh-8Vofos1aw11O0&t=6119e399

[turquoiseowl]

Please try the commit just pushed into a new branch called "Dev" and let me know any difference.

[krisztianb]

The module crahes at line 121 of LocalizingModule.cs:

string ce = context.Response.Headers.Get("Content-Encoding");

Exception:

Ausnahmedetails: System.PlatformNotSupportedException: Für diesen Vorgang muss der integrierte Pipelinemodus von IIS verwendet werden.

Looks like that this method is only supported using the pipeline mode of IIS.

[turquoiseowl]

Ok, we seem to have quite a few things different in our set ups what with that error and the x-javascript content-type.

If you read the comments above the line that throws you can see the challenge I've had with this. Not sure where to go with this from here. Happy to code any suggestions you may have.

[turquoiseowl]

http://stackoverflow.com/questions/5434858/can-i-detect-if-content-has-been-compressed-in-my-httpmodule

[turquoiseowl]

Latest push implements the second suggestion in the SO article.

[krisztianb]

The module does not crash any more, but we have the same symptom as in the beginning. The ASP.NET AJAX framework can't be loaded. The gzip response seems to get corrupted by the module.

Btw.: I moved the code into our SVN repository and noticed that the "Output path" (Visual Studio, Project preferences..., Build-tab) of the i18n.PostBuild project is not set to "....\bin\lib\net40". So one needs to manually copy it into the bin-folder of the referencing application. The EXE is not part of the NuGet package as it is not found by pack-nuget.bat. Is this intentional?

[krisztianb]

It seems that the integrated Visual Studio web server is using IIS classic mode.

I tried running the application within an application pool that uses IIS pipeline mode. The problem is gone there and gzip compression is detected correctly.

I also tried to reduce the code only to the hack remaining (as the poster says that is works in both modes) and the result is the same.

How can we debug the module? The poster says that it should work, so I am interested in finding out why it doesn't work for us.

[turquoiseowl]

Sorry, I don't understand your message. Posters? Hack? Same?

From: krisztianb [mailto:notifications@github.com]
Sent: 04 December 2013 11:48
To: turquoiseowl/i18n
Cc: Martin Connell
Subject: [SPAM] Re: [i18n] JavaScript file is not translated (#109)

It seems that the integrated Visual Studio web server is using IIS classic mode.

I tried running the application within an application pool that uses IIS pipeline mode. The problem is gone there and gzip compression is detected correctly.

I also tried to reduce the code only to the hack remaining (as the poster says that is works in both modes) and the result is the same.

How can we debug the module? The posters say that it should work, so I am interested in fining out why it doesn't work for us.

—
Reply to this email directly or view it on GitHub #109 (comment) . https://github.com/notifications/beacon/xL73SQRLzck8DJI41v2Mt6aJG7ERa7iXp2-ITaTwmwMFGjvZ1xh61IpmKcmGRveL.gif

[krisztianb]

Sorry. :-/

The poster is the person who posted the hack at stackoverflow. He said that his hack is working in both modes (IIS classic and pipeline). But it does not work for me in classic mode. Which we wanted it to use for, since there are no problems running the application in piple mode. So it would be a good idea to debug the module in order to find out, what exactly is going on. How can I achieve this?

[turquoiseowl]

The easiest way to debug the library -- or at least the way I do it -- is like this:

1. Load your ASP.NET app solution into Visual Studio. An assumption here is that you are referencing from this solution the i18n.dll which you have built from source (in the i18n solution).
2. Open a source file of interest from the i18n solution. In this case it would be, say, C:\DevRoot\DevGit\i18n\src\i18n\LocalizingModule.cs
3. Set a breakpoint in that file. In this case, the first line of the LocalizingModule.OnReleaseRequestState method.
4. Start debugging your solution and wait for the BP to get triggered. All being well all the symbols etc. will get loaded. You may need to ensure you build i18n with Debug configuration for this.

The comments in the above mentioned method should be self-explanatory.

The Debug output should show in the Debug output window of visual studio.

Pleased to hear you are willing to help in this respect.

[krisztianb]

In case someone is reading this and has not basic understanding (like me) of how response filters work, I found this nice little tutorial on creating one: http://www.4guysfromrolla.com/articles/120308-1.aspx

I fiddled around with the IsResponseCompressed method and the IIS pipeline modes. This is what I found out:

• Visual Studio Development web server is running in classic mode.
• IIS Express (also accessible trhough Visual Studio) is running in integrated mode.
• An application configured within IIS can run in an Application Pool that uses classic or integrated mode. (this can be configured at the Application Pool settings in IIS)

There is no problem in integrated mode, as we already found out. So I tried to find a soultion for classic mode and commented all solutions from this thread you found: http://stackoverflow.com/questions/5434858/can-i-detect-if-content-has-been-compressed-in-my-httpmodule

Neither of them work.

• The solution using Reflection does not work, because the Headers collection will be null and thence you can't call .Get("Content-Encoding")
• The solution using the Filter field does not work either, because Filter is a stream. Calling .ToString() returns the name of the class. Reading from the stream is also not possible, because Filter is of type HttpResponseStreamFilterSink which is going to throw an exception when you try to read from it.

It seems to me that we must accept that the filter is not working when the HTTP pipeline is running in classic mode.

Maybe something like:

private bool IsResponseCompressed(HttpResponseBase response)
{
  if (HttpRuntime.UsingIntegratedPipeline)
  {
    ...
  }
  else
  {
    throw new NotSupportedException("HTTP pipelines running in classic mode are not supported");
  }
}

Question 1: However, another idea might be to apply the i18n-module before gzip compression is applied. Do you think this is possible?

Question 2: The problem should only occur, when you have set MessageKeyIsValueInDefaultLanguage to true, since this is the option that makes the module remove nugget brackets from the (gzip) output even when there is no translation found, right?

[turquoiseowl]

You're right, the best solution is to do the response processing before anyone else.

Oddly enough, the way the response filters are chained together, or rather layered together, the last one installed is the first one to get called. So what I've done is try the VERY last pipeline stage that we can install our filter, namely the PostReleaseRequestState. (See remarks section of this page for a full list of all the stages: http://msdn.microsoft.com/en-us/library/System.Web.HttpApplication(v=vs.110).aspx)

I've just pushed this mod to the dev branch for you to try.

[krisztianb]

I've just tested it. The error is still the same.

The link you posted doesn't work. Are you sure about the filter execution order? That really sounds strange. Maybe we could try installing the filter at the beginning of the chain? Just in case the documentation is mistaken. :-)

Edit: I modified the title of this issue

[turquoiseowl]

I can't get the URL to work with the markdown here. To get the link to work, replace the %28 with ( and %29 with ).

[turquoiseowl]

Feel free to try the other stages yourself. Hopefully they will work. You can search-and-replace PostReleaseRequestState with the name of the other event (NOT whole-word).

It still could be that the GZIP filter is being installed in the PostReleaseRequestState event, but by a another handler of that event called after our handler.

[krisztianb]

I'll happily try several stages. But before I start doing so... Could you give me a list of stages that are worth trying? Unfortunately there are so many stages and I have no real clue about which to use. I've never written an HTTP handler or something similar. Which would be the first stage that we can use to register our module?

[turquoiseowl]

We can register at the very first stage, the BeginRequest event (which we already handle but expect no harm in just adding another handler for that).

I don't think it work trying any intermediate ones, but may be wrong.

[krisztianb]

I think I will only be able to try this next year. Other more important tasks are at hand. I'll report back. :-/

[ryan1234]

Not sure if this helps, but I was having the same problem. Example response headers that worked versus headers that did not work:

Good:

HTTP/1.1 200 OK
Content-Type: application/x-javascript
Last-Modified: Tue, 31 Dec 2013 21:40:19 GMT
Accept-Ranges: bytes
ETag: "b1b5d7e6706cf1:0"
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Thu, 02 Jan 2014 00:06:31 GMT
Content-Length: 282525

Bad:

HTTP/1.1 200 OK
Content-Type: application/x-javascript
Content-Encoding: gzip
Last-Modified: Tue, 31 Dec 2013 21:40:19 GMT
Accept-Ranges: bytes
ETag: "80a37be6706cf1:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Date: Thu, 02 Jan 2014 00:06:34 GMT
Content-Length: 148424

I turned off "Enable static content compression" in IIS and that fixed the problem for me.

I tried all sorts of combinations of settings from this thread and wasn't able to fix it outside of the IIS setting. Any idea how this Content-Encoding: gzip sneaks its way into the response header?

[krisztianb]

Hi Ryan. Make sure that your application pool that contains your application is running in integrated mode. The problem described in this thread only occurs in classic mode.

[krisztianb]

I finally had some time to look into the problem. Unfortunately I found no solution for the filter to run before the gzip compression:

• I tried the BeginRequest stage. This resulted in the page endlessly reloading itself until aborted by Firefox.
• I found this thread on stackoverflow: (http://stackoverflow.com/questions/7363717/rewrite-asp-net-response-before-it-gets-cached-with-dynamiccompressionbeforecach) But the solution using the PostRequestHandlerExecute stage does not work. There is no difference.

Last I found this page: (http://blogs.msdn.com/b/tmarq/archive/2007/08/30/iis-7-0-asp-net-pipelines-modules-handlers-and-preconditions.aspx) that describes the difference between integrated and classic pipeline modes and the possible pipeline stages. Maybe this can help you guys?!

I consider this a huge problem. The module must run before gzip compression is applied. Otherwise you can't use compression on content that uses translations/nuggets. :-(

[ryan1234]

I am running in Integrated mode and still have the problem. =(

[krisztianb]

Sorry I was mistaken. You actually have to disable gzip compression to use the module. For some reason the module gets gzip compressed input (which can't be translated) and has to skip it. Thus your compressed pages are not translated.

The only solution I currently see is:

1. Disable gzip compression on the web server
2. Install the i18n module
3. Install a gzip compression module manually in code

[krisztianb]

@ryan1234: I ran into a similar problem on our live server. JavaScript was served with MIME content type "application/x-javascript" and gzip compression was applied. For this reason the i18n module did not translate the file.

Locally IIS serves the JavaScript file with MIME content type "application/javascript" and gzip is not applied. Without any specific configuration from my side. But, I am sure something must be different...

The following entry in the web.config file solved both issues (MIME conent type and gzip) on the live server:

<configuration>
  <system.webServer>
        <staticContent>
            <remove fileExtension=".js" />
            <mimeMap fileExtension=".js" mimeType="application/javascript" />
        </staticContent>
  </system.webServer>
</configuration>

I don't understand why, but with this setting gzip compression is no longer applied to JavaScript files (extension .js). Maybe gzip compression is MIME conent type dependent?

To summarize: I am serving JavaScript files with MIME conent type "application/javascript" and the i18n module is configured to only run for this JavaScript content type. Not for "application/x-javascript" that is being used for several script files auto-generated by ASP.NET WebForms. Why? See ticket #116.

[couraud]

Thanks - this was useful.

[intoccabil]

I've also encountered problems with gzip compressed .js files.
I've got an ASP.NET MVC5 application running on IIS 8.5 in integrated mode.

Here are the response headers for bundled javascript files:

HTTP/1.1 200 OK
Cache-Control: public
Content-Type: text/javascript; charset=utf-8
Content-Encoding: gzip
Expires: Wed, 17 Jun 2015 08:44:06 GMT
Last-Modified: Tue, 17 Jun 2014 08:44:06 GMT
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Tue, 17 Jun 2014 08:44:06 GMT
Content-Length: 38090

and here are the headers for files that are loaded explicitly e.g. @Scripts.Render("~/Scripts/js/test.js")

HTTP/1.1 200 OK
Content-Type: application/javascript
Content-Encoding: gzip
Last-Modified: Fri, 06 Jun 2014 13:52:20 GMT
Accept-Ranges: bytes
ETag: "02f3888e81cf1:0"
Vary: Accept-Encoding
Server: Microsoft-IIS/8.5
X-Powered-By: ASP.NET
Date: Tue, 17 Jun 2014 08:44:06 GMT
Content-Length: 900

I have seemingly resolved the issue with changing the default Regex for ContentTypesToLocalize from

@"^(?:(?:(?:text|application)/(?:plain|html|xml|javascript|x-javascript|json|x-json))(?:\s*;.*)?)$"

to

@"^(?:(?:(?:text|application)/(?:plain|html|xml|x-javascript|json))(?:\s*;.*)?)$"

but this workaround leaves me with .js files not being localized. That's not a big issue for me as I have only a couple strings to localize, but nevertheless it would be nice to have centralized localization.

That's just my two cents that hopefully will provide some additional insights to the issue. Thanks for reading.

[gitsno]

This issue is now resolved in a commit in #193
Note that the fix prevents corruption of content that is compressed before it reaches the i18n module, it does not localize within the compressed content, so if you need to have nuggets in static files localized you should turn off static file compression, but having static file compression turned on will no longer result in corrupt responses.