Offer a Meteor 1.4-compatible vagrant-spk platform stack #193

paulproteus · 2016-10-05T23:25:40Z

Context:

If you try to package an app that uses Meteor 1.4.x with vagrant-spk's Meteor platform stack, you will run into mysterious build problems.
This is due to version incompatibilities between the nodejs version bundled with Meteor pre-1.4 and the node version bundled with Meteor 1.4+.
A while ago, @zarvox and I discussed this, and the best solution we could come up with was to have two different platform stacks -- one for Meteor 1.4+ and one for Meteor pre-1.4.

Sub-problems to consider:

vagrant-spk should properly package a Meteor apps that has no .sandstorm/. This may involve looking at .meteor/release and printing an error message, or auto-switching which .sandstorm/ it generates.
vagrant-spk should provide reasonable advice for apps that used to Meteor 1.3 and now are using Meteor 1.4. This may involve looking at .meteor/release and printing an link to docs.

The text was updated successfully, but these errors were encountered:

paulproteus · 2016-10-05T23:29:45Z

FWIW I believe the reason we need to make these two different platform stacks is that different versions of https://github.com/sandstorm-io/meteor-spk either support Meteor through 1.3.x or Meteor 1.4+. I read the commit log message at sandstorm-io/meteor-spk@22ead5a and couldn't see any documented reason why that is, but it sure says it's true.

FWIW we could look at .meteor/.release and dispatch to a different meteor-spk binary. When Drew and I talked about that, it seemed like too much work, but right now I wonder if it really is too much work.

paulproteus · 2016-10-06T00:03:42Z

FWIW here's the output with current meteor-spk & Meteor 1.4 code:

paulproteus@slittingmill:~/projects/meteor-example-clock$ vagrant-spk dev
Calling 'vagrant' 'ssh' '-c' '/opt/app/.sandstorm/build.sh && cd /opt/app/.sandstorm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef' in /home/paulproteus/projects/meteor-example-clock/.sandstorm
npm WARN package.json meteor-dev-bundle@0.0.0 No description                       
npm WARN package.json meteor-dev-bundle@0.0.0 No repository field.
npm WARN package.json meteor-dev-bundle@0.0.0 No README data

> fibers@1.0.13 install /home/vagrant/bundle/programs/server/node_modules/fibers
> node build.js || nodejs build.js

make: Entering directory '/home/vagrant/bundle/programs/server/node_modules/fibers/build'
  CXX(target) Release/obj.target/fibers/src/fibers.o
make: g++: Command not found
fibers.target.mk:91: recipe for target 'Release/obj.target/fibers/src/fibers.o' failed
make: *** [Release/obj.target/fibers/src/fibers.o] Error 127
make: Leaving directory '/home/vagrant/bundle/programs/server/node_modules/fibers/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/home/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:820:12)
gyp ERR! System Linux 3.16.0-4-amd64
gyp ERR! command "node" "/home/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild" "--release"
gyp ERR! cwd /home/vagrant/bundle/programs/server/node_modules/fibers
gyp ERR! node -v v0.10.36
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok 
node-gyp exited with code: 1
Please make sure you are using a supported platform and node version. If you
would like to compile fibers on this machine please make sure you have setup your
build environment--
Windows + OS X instructions here: https://github.com/nodejs/node-gyp
Ubuntu users please run: `sudo apt-get install g++`
Alpine users please run: `sudo apk add python make g++`
sh: 1: nodejs: not found

npm ERR! fibers@1.0.13 install: `node build.js || nodejs build.js`
npm ERR! Exit status 127
npm ERR! 
npm ERR! Failed at the fibers@1.0.13 install script.
npm ERR! This is most likely a problem with the fibers package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node build.js || nodejs build.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls fibers
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.16.0-4-amd64
npm ERR! command "/home/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/node" "/home/vagrant/.meteor/packages/meteor-tool/.1.1.3.4sddkj++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/npm" "install"
npm ERR! cwd /home/vagrant/bundle/programs/server
npm ERR! node -v v0.10.36
npm ERR! npm -v 1.4.28
npm ERR! code ELIFECYCLE
npm ERR! not ok code 0
Connection to 127.0.0.1 closed.
Traceback (most recent call last):
  File "/usr/local/bin/vagrant-spk", line 1028, in <module>
    main()
  File "/usr/local/bin/vagrant-spk", line 1025, in main
    operation(args)
  File "/usr/local/bin/vagrant-spk", line 680, in dev
    "spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef"
  File "/usr/local/bin/vagrant-spk", line 294, in call_vagrant_command
    return subprocess.check_call(command, cwd=sandstorm_dir)
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['vagrant', 'ssh', '-c', '/opt/app/.sandstorm/build.sh && cd /opt/app/.sandstorm && spk dev --pkg-def=/opt/app/.sandstorm/sandstorm-pkgdef.capnp:pkgdef']' returned non-zero exit status 1

paulproteus · 2016-10-06T23:04:28Z

Things I've noticed so far, when trying a Sandstorm app based on Meteor 1.4:

The Meteor platform stack needs a C++ compiler, which is harmless to add for all versions of Meteor.
- Note, however, that this is a change to setup.sh that existing app authors would need to make.
Meteor likes to hang forever at "Updating package catalog" and/or create an invalid sqlite database when you ^C it, which results in a pretty sad state of affairs. This seems to be fixed by the following change in setup.sh:

diff --git a/.sandstorm/setup.sh b/.sandstorm/setup.sh
index 8bdc63e..d6e07e2 100755
--- a/.sandstorm/setup.sh
+++ b/.sandstorm/setup.sh
@@ -42,7 +42,7 @@ cp -a /lib/x86_64-linux-gnu/libtinfo.so.* /opt/meteor-spk/meteor-spk.deps/lib/x8
 # Unfortunately, Meteor does not explicitly make it easy to cache packages, but
 # we know experimentally that the package is mostly directly extractable to a
 # user's $HOME/.meteor directory.
-METEOR_RELEASE=1.1.0.2
+METEOR_RELEASE=1.4.1.2
 METEOR_PLATFORM=os.linux.x86_64
 METEOR_TARBALL_FILENAME="meteor-bootstrap-${METEOR_PLATFORM}.tar.gz"
 METEOR_TARBALL_URL="https://d3sqy0vbqsdhku.cloudfront.net/packages-bootstrap/${METEOR_RELEASE}/${METEOR_TARBALL_FILENAME}"

Further notes:
- It's possible that using the newer Meteor release allows us to overcome a Meteor CDN bug that @kentonv mentioned this morning. This does, however, pin us to a particular version of Meteor. I'd prefer to always download the latest release of Meteor, which we can theoretically get by doing curl https://install.meteor.com/ | grep ^RELEASE= but that's pretty ugly.
- This is also a change that existing app authors probably need to do, to get past the weird CDN problem. It's probably safe to pin this to a newer version for all Meteor-based Sandstorm app authors. It's probably unsafe to make this totally dynamic, since that decreases how "reproducible" the vagrant-spk builds are, so the above is probably a bad idea.
The app works properly if I set PACKAGE=meteor-spk-0.3.0. Interestingly, if I set PACKAGE=meteor-spk-0.1.8 the build "succeeds" but I get this stuff in my grain log:

...** SANDSTORM SUPERVISOR: Starting up grain.
** Starting Mongo...
2016-10-06T23:03:26.831+0000 I STORAGE  Engine custom option: log=(prealloc=false,file_max=200KB)
about to fork child process, waiting until server is ready for connections.
forked process: 6
child process started successfully, parent exiting
** Starting Meteor...

assert.js:93
  throw new assert.AssertionError({
        ^
AssertionError: "undefined" === "function"
    at wrapPathFunction (/programs/server/mini-files.js:77:10)
    at Object.<anonymous> (/programs/server/mini-files.js:108:24)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/programs/server/boot.js:9:13)
    at Module._compile (module.js:456:26)
sandstorm/sandstorm-http-bridge.c++:2236: warning: App isn't listening for TCP connections after 30 seconds. Continuing to attempt to connect; address->toString() = 127.0.0.1:8000

paulproteus · 2016-10-06T23:07:45Z

Thoughts:

It's too bad meteor-spk can't be depended-on via Atmosphere or npm. Can we fix it so it can be...? That might be the best thing.
Why does meteor-spk allow itself to get the node version wrong, i.e. use a node version that isn't the one from the Meteor app? I see this line:
- Should that be meteor node start.js?


    spk init -p 4000 -I.meteor-spk/deps -I.meteor-spk/bundle -A "$@" -- node start.js

paulproteus · 2016-10-06T23:09:00Z

...I see, it's because meteor-spk packs the node version in via its deps/ or bundle/ directory.

So how about it not? :)

jemc · 2017-01-17T22:17:16Z

Any update on this?

ocdtrekkie · 2018-04-30T02:34:16Z

So Rocket.Chat is potentially willing to resume releases, they only stopped pushing them because the build process broke, likely due to the old Debian box disappearing. They are currently on Meteor 1.6 or so, and their existing Sandstorm build used vagrant-spk's Meteor stack. So I'd like to fix this up once meteor-spk supports 1.6.

I wonder if the correct answer is not for vagrant-spk to set up for the latest version as a default scenario. From what I can see, it's common or expected for your .sandstorm files to get customized as part of your packaging work, so vagrant-spk doesn't tamper with them after you initially make them. Therefore, it should probably be pushing the latest version when starting new packaging.

A fair question at this point, two years forward from where this topic started, is... is pre-Meteor 1.4 support still needed? Wekan and Rocket.Chat are both on 1.6 already. Is there a trove of applications out there that don't work on modern Meteor?

ocdtrekkie · 2018-05-22T21:13:47Z

Taking a look at this again, I found it intriguing @paulproteus suggested that "Note, however, that this is a change to setup.sh that existing app authors would need to make." was perhaps a bad thing.

I think the fragility we've seen from Vagrant changes alone suggests that developers are going to need to update their .sandstorm scripts. PHP 5 -> PHP 7 was brutal, the whole MySQL/MariaDB thing, the fact that our pinned box stopped being hosted, etc. Either developers may wish to compare stacks occasionally to see what vagrant-spk has changed, or we need to look at making these scripts more modular so they can be easily replaced and worked around.

It is possibly vagrant-spk's best use for stacks is just that they be a great starting point, and that you either manually fix as necessary or replace and rework occasionally past that.

ocdtrekkie · 2018-05-22T21:23:17Z

Also, I just found CommonGarden/Grow-IoT@3f10818 by @aruntk which looks like a version of vagrant-spk's meteor stack that works after the fibers debacle. I am interested in testing the changes made there as a possible solution.

ocdtrekkie · 2018-05-23T02:50:53Z

@aruntk's solution does indeed work with Meteor 1.6.1.1 on vagrant-spk. I am not totally sold on all the changes though, it looks like a lot of error checking got stripped out that probably wasn't doing any harm. I am going to walk a couple of those back, I think, keep this down to just essential changes to fix the stack.

This was referenced Oct 6, 2016

Support Meteor 1.4 sandstorm-io/meteor-spk#24

Closed

Can meteor-spk stop embedding nodejs? sandstorm-io/meteor-spk#26

Open

dwrensha mentioned this issue Jan 1, 2017

Error during vagrant-spk dev in meteor project. sandstorm-io/sandstorm#2810

Closed

ocdtrekkie mentioned this issue May 23, 2018

Update Meteor stack to 1.6.1.1 #231

Merged

kentonv closed this as completed in #231 Jun 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offer a Meteor 1.4-compatible vagrant-spk platform stack #193

Offer a Meteor 1.4-compatible vagrant-spk platform stack #193

paulproteus commented Oct 5, 2016

paulproteus commented Oct 5, 2016

paulproteus commented Oct 6, 2016

paulproteus commented Oct 6, 2016 •

edited

Loading

paulproteus commented Oct 6, 2016 •

edited

Loading

paulproteus commented Oct 6, 2016

jemc commented Jan 17, 2017

ocdtrekkie commented Apr 30, 2018

ocdtrekkie commented May 22, 2018

ocdtrekkie commented May 22, 2018

ocdtrekkie commented May 23, 2018

Offer a Meteor 1.4-compatible vagrant-spk platform stack #193

Offer a Meteor 1.4-compatible vagrant-spk platform stack #193

Comments

paulproteus commented Oct 5, 2016

paulproteus commented Oct 5, 2016

paulproteus commented Oct 6, 2016

paulproteus commented Oct 6, 2016 • edited Loading

paulproteus commented Oct 6, 2016 • edited Loading

paulproteus commented Oct 6, 2016

jemc commented Jan 17, 2017

ocdtrekkie commented Apr 30, 2018

ocdtrekkie commented May 22, 2018

ocdtrekkie commented May 22, 2018

ocdtrekkie commented May 23, 2018

paulproteus commented Oct 6, 2016 •

edited

Loading

paulproteus commented Oct 6, 2016 •

edited

Loading