New deployment issue when using grunt / traceur

[JayBeavers]

My automated deployment scripts for a nodejs/bower/grunt/traceur website started failing unexpectedly. Last successful deployment was Wednesday June 25, first new failed deployment was Monday June 30.

Repro case: https://github.com/JayBeavers/webtemplate deployed via Azure Web Site -> Source Control Deployment -> Github

AFAIK, this isn't an issue with the build scripts. I do not reproduce the error when building/running the website locally on my Windows 8.1 dev box with similar environment.

Any recent changes deployed to Kudu that might explain this? The failure seems to be in finding a file which is pathed using:

var traceurCommandPath = path.resolve(path.join(__dirname, "../node_modules/traceur/src/node/command.js"));

and I've validated via kudu debug console that the pathing looks correct. In kudu debug console, I don't get the funky D:\10.207.110.176... pathing, I see the simpler D:\home\site... syntax.

Running "traceur:all" (traceur) task
transpiling: public/js/app.js <- public/js/app.6.js
>> Error: Command failed: 
>> module.js:340
>>     throw err;
>>           ^
>> Error: Cannot find module 'D:\10.207.110.176\volume-24-default\fa3af64b925e20b3e746\03e0f263c5d842e7bfd2c3d0b44e76d1\site\wwwroot\node_modules\grunt-traceur-simple\node_modules\traceur\src\node\command.js'
>>     at Function.Module._resolveFilename (module.js:338:15)
>>     at Function.Module._load (module.js:280:25)
>>     at Function.Module.runMain (module.js:497:10)
>>     at startup (node.js:119:16)
>>     at node.js:906:3
>> module.js:340
>>     throw err;
>>           ^

[JayBeavers]

Crossposted to rse/grunt-traceur-simple#5.

[davidebbo]

Something similar was just reported here. We need to investigate what's going on.

[JayBeavers]

ETA? This just broke a site that's scheduled to go live on 7/15. Need to know if I should start contingency planning...

[davidebbo]

We are now actively investigating. I will keep this thread updated with status.

[follesoe]

I'll fill in on our findings. Some time between friday 27. and monday 30 we started having issues with our build scripts on Azure. We use gulp for our build scripts, and the problem we started experiencing was that the gulp.src-task would now longer be able to read files.

We resolve our base-path for the build scripts using path.resolve(__dirname + '/../'). We discovered that only certain tasks would fail - things like LESS-compilation and SVG-asset building. Other tasks, like browserify would still run.

So after experimenting a bit, we saw that __dirname was resolved to a UNC-path like:

\\100.84.138.54\volume-1-default\2540d69c3e4e1c5d0a98\1a601593a75d4dde894445905aa24003\site\repository\foobar\gulp/tasks.

After looking at the source for the gulp.src task, we saw that it uses vinly-fs and node-glob, and there are numerous issues, such as isaacs/node-glob#74 describing issues with UNC and node-glob.

So our fix was to rewrite the build scripts to use process.cwd() to get the base path in our build scripts.

But, as I mentioned on Twitter I still wanted to report this, as this is something that must have changed on the Azure side, as our build scripts have been working just fine with __dirname, so this variable must have been a standard path until recently.

[davidebbo]

The problem is now understood, and we have a fix ready, which should be deployed by tomorrow.

[davidebbo]

It looks like this may take a bit longer than anticipated, and we ran into some edge cases and we want to make sure we get a fully correct fix. Apologies for the delay.

[follesoe]

Thanks for keeping us posted. In our case the process.Cwd() changed fixed our problems, so we're at least good for now 👍

[ghost]

+1
Our deployments scripts broke as well for the same reason. Many dependent libraries internally use __dirname so it isn't practical to update to process.cwd()

[davidebbo]

The fix is being deployed now, but it can take a few hours to propagate everywhere, Depending on where your site is, you may see the issue being fixed already.

[ghost]

@davidebbo Just FYI, the issue is still not fixed for our deployment in SE Asia

Our logs still show __dirname resolving to
local installation: \100.77.66.97\volume-20-default\6d184f641ed336caca55\a24e5c9012984854978ff8729fbcf85f\site\repository\node_modules

[davidebbo]

Sorry, SE Asia is in fact the only region that is not complete. Hopefully it will happen later today. In the meantime, feel free to test your code in a different region, to make sure that everything works as you expect.

[ghost]

Thanks for the update. We are in no hurry.

[davidebbo]

Unfortunately, due to an issue during the update, SE Asia (Singapore) is still not updated. Everywhere else is done. Some possible options:

1. wait for it to be done. I don't have a clear ETA, but we'll get there
2. move your site to East Asia (Hong Kong)
3. if your site is in Basic or Standard mode (i.e. had its own VM), we can manually patch just this VM. I'll just need your site name to do this. If you don't want to post it publicly, you can email me at david.ebbo (AT) microsoft.com.

Apologies for the inconvenience.

[ghost]

Thank you for the detailed follow up during this long weekend there in US. We are perfectly willing to wait.

[JayBeavers]

Confirmed fixed in my West US Azure website. Please close when you feel the issue is fully resolved.

Thank you for the speedy response!

[ghost]

Confirmed fix at SE Asia as well. Thanks.

[davidebbo]

Indeed, it is deployed everywhere now, and we can close this for good. Thanks for your patience!