Investigate reducing the dependencies to install vips

[ide]

One of the hardest parts about using Sharp is setting it up. Being able to just say "npm install sharp" would make it easier for a lot of people to try it out.

Currently libvips isn't very easy to set up on an Amazon Linux box, and on Mac OS X it requires brew (there are some instances in which one may wish not to install brew). So if there were an easier way to build vips and it were integrated with Sharp's installation that could help a lot.

[lovell]

Instructions in the README for yum-based installation (to include AWS's default AMIs) would definitely help, thanks for the suggestion.

How about a sample Dockerfile? That would help with an Elastic Beanstalk deployment.

Not sure I'd want the maintenance overhead of including and compiling all dependencies from source!

[ide]

Instructions for yum (EC2 in particular) and a dockerfile would certainly be useful for production setups.

I was chatting with a friend earlier today who didn't have brew installed and he found Sharp difficult to set up on Mavericks. I don't think it's reasonable for the Sharp project to precompile vips for major platforms but it would be nice for compilation to be easier.

For example node-fibers is pretty easy to install on all major platforms and it ships with the source of its dependencies. So something similar where Sharp ships with the source of vips could be helpful, even if it's a longer term goal that might require help from the vips team.

On Tue, May 27, 2014 at 11:31 PM, Lovell Fuller notifications@github.com
wrote:

Instructions in the README for yum-based installation (to include AWS's default AMIs) would definitely help, thanks for the suggestion.
How about a sample Dockerfile? That would help with an Elastic Beanstalk deployment.

Not sure I'd want the maintenance overhead of including and compiling all dependencies from source!

Reply to this email directly or view it on GitHub:
#42 (comment)

[lovell]

The libcoro dependency of node-fibers appears a lot simpler than libvips (which then has its own dependencies to worry about too).

What I really like about the node-fibers repo is the provision of build.js, an idea sharp could borrow as a "bootstrap" to verify that dependencies are present, providing more useful feedback/help in the event things are not quite ready. Thanks for the inspiration!

I'm also happy to provide installation instructions for other Mac OS package managers, e.g. Fink, if there's enough call for it.

[jonathanong]

i would love if vips was included so i can deploy it to places like heroku without crazy images and stuff. however, it makes total sense not to as it's super complicated. it could also take forever to build

[lovell]

@jonathanong There's a Heroku Buildpack at https://github.com/alex88/heroku-buildpack-vips which I've not tried but seems only slightly out of date with respect to required version numbers. Perhaps send a PR to upgrade to libvips 7.38.5 and liborc 0.4.19.

[lovell]

I've added a preinstall.sh shell script on the preinstall branch that can be used to install libvips and its dependencies on a wide variety of OS and package manager combos (but not Windows).

If everyone/anyone following this issue is able to test the instructions at https://github.com/lovell/sharp/tree/preinstall#prerequisites - especially on Mac OS as I've not been able to verify this - I would very much appreciate it.

[lovell]

When everyone is happy that preinstall.sh is solid enough, I'd like to either add it as the preinstall script in package.json (completely ignoring the "INSTALL SCRIPTS ARE AN ANTIPATTERN" warning) or somehow get gyp to call it.

[ide]

Here's my output on Yosemite. Seems to be working.

$ ./preinstall.sh 
Found libvips 7.40.11
$ brew remove vips
Uninstalling /usr/local/Cellar/vips/7.40.11...
$ ./preinstall.sh 
Could not find libvips using a PKG_CONFIG_PATH of '/usr/lib/pkgconfig:/usr/local/Library/ENV/pkgconfig/10.10::/usr/local/lib/pkgconfig:/usr/lib/pkgconfig'
Detected Mac OS
Installing libvips via homebrew
==> Downloading http://www.vips.ecs.soton.ac.uk/supported/7.40/vips-7.40.11.tar.g
Already downloaded: /Library/Caches/Homebrew/vips-7.40.11.tar.gz
==> ./configure --prefix=/usr/local/Cellar/vips/7.40.11 --with-magick --with-magi
==> make check
==> make install
==> Caveats
If you need Python to find the installed site-packages:
  mkdir -p ~/Library/Python/2.7/lib/python/site-packages
  echo '/usr/local/lib/python2.7/site-packages' > ~/Library/Python/2.7/lib/python/site-packages/homebrew.pth
==> Summary
🍺  /usr/local/Cellar/vips/7.40.11: 196 files, 13M, built in 65 seconds

[lovell]

Great news, thanks @ide.

"I was chatting with a friend earlier today who didn't have brew installed and he found Sharp difficult to set up on Mavericks."

Is your friend available to help? What, if any, package/dependency management tool(s) do they use?

[toulouse]

That's me.

I use vanilla ./configure with $PREFIX set to a subdirectory of my home directory, and make install things. I'm not a UNIX neckbeard, but it's a very reasonable expectation to expect to be able to make local, non-global installations of compiled software.

Only being able to have a single active version of a dependency on an entire system is literally a singleton pattern and it makes a number of compromises to the system's integrity I'm not really willing to make, not the least of which is making a subdirectory of /usr/local user-owned-and-writable.

Not to mention, new OS upgrades often break - sometimes subtly, sometimes blatantly - changes made by the package managers, and I prefer not to be dependent on upstream fixes to get a system up and running to where I'd like it to be.

[papandreou]

The preinstall script works fine on Ubuntu 14.04 here. This patch makes it work with Debian Wheezy as well: #111

[lovell]

@toulouse Thanks for your feedback - you're definitely not a "neckbeard"! What you say is pretty much why the separation of Linux Containers via tools such as Docker is attracting so much attention at the moment.

Do you rely on pkg-config for dependency management? If so, are you manually modifying/setting the PKG_CONFIG_PATH environment variable? If not, how to you achieve this?

My current thinking is that the preinstall shell script will install libvips and dependencies globally only if run with root/sudo access, otherwise it will attempt to compile at least libvips from source into a relevant directory under node_modules and add the path to PKG_CONFIG_PATH.

@papandreou Thanks Andreas, I've merged your patch.

[papandreou]

My current thinking is that the preinstall shell script will install libvips and dependencies globally only if run with root/sudo access, otherwise it will attempt to compile at least libvips from source into a relevant directory under node_modules and add the path to PKG_CONFIG_PATH.

Very much in favor of that approach. It needs to work when not sudoing npm install, and I also prefer a local install.

Would it be possible to bundle the libvips source code à la https://github.com/bnoordhuis/node-iconv?

[lovell]

The preinstall.sh script, which currently requires root/sudo access, is now in master. The Travis and Snap CI jobs both use this script for their test environments. I'll only make this part of npm install when it can be run as a non-root user.

@papandreou The upstream libvips library changes often so I'd rather have an install script fetch a specific version (one line change) than include it in the repo (1000+ file change).

There's also the small matter of libvips' dependencies (libjpeg, libpng, libwebp etc.) to consider. How much can we assume is already present on a system?

[toulouse]

IMO, pkg-config is definitely an acceptable mechanism for locating libraries - it's well-known and mature tool that has worked well in my experience.

As for including libvips in the repo, I think it would make a lot of sense to include it as a git subtree (for making tracking and merging changes pretty straightforward), and track the libvips branch versions.

One major priority for running code in production is to have well-known combinations between libraries, so if there's a bug that only manifests in version X of libvips when combined with version Y of sharp, it's a lot easier to test for / verify / prevent.

On the other hand, your point about libjpeg, libpng, libwebp, etc. is quite valid.

[papandreou]

@lovell Sure, it's a balance, and I understand why you would like to keep the size of the package itself down. However, on top of @toulouse's points about predictiblity it's also a big plus not to depend on downloading stuff from the Internet at install time.

The source code of the mentioned libraries adds up to about 16MB, which is about the same size as libiconv, and node.js itself bundles about 80MB of dependencies. It's a lot, but it could be done.

[marcbachmann]

FYI: I just created a Dockerfile which installs libvips from source. It's the same installation procedure like in the install script. I created the dockerfile because I failed installing libvips using the provided script (some pkg-config problem).

Here is the file https://registry.hub.docker.com/u/marcbachmann/dockerfile-libvips/dockerfile.
It's based on ubuntu & the image has about 726MB, which is really huge. I have no problem with that size but I think it can get smaller.

I'm sorry for ignoring the comments in here. It's late 😪

[lovell]

Thanks for creating a Dockerfile @marcbachmann, I'll add a link to the prerequisite docs.

"I failed installing libvips using the provided script (some pkg-config problem)."

Feel free to open a new issue if you'd like more help with this specific problem.

[binarykitchen]

Many packages place OS dependant binaries in the local .bin folder inside node_modules. Would that be an option?

[lovell]

#186 provides the a rather compelling solution to this.

[vlapo]

@lovell Why my self share compiled libvips+sharp ZIP on AWS has 6MB and look branche is so big ?

[lovell]

@vlapo See #292 (comment)

[vlapo]

I know about comment. I check it again.

$ du -h --max-depth=1 | sort -hr
22M     ./node_modules
18M     ./vips

This is my libvips compiled in vips and node_modules with sharp ~40MB

And here is look branche without tests

57M     ./node_modules

hmm maybe, I have lite compilation or skip some libs ~17MB. :D
Okey, 17MB are a little bit real.

[lovell]

@vlapo I'd expect shared libs to be a bit larger than static libs. Did you include libMagickCore?

[mahnunchik]

What about binaries for Mac OS X?

npm install lovell/sharp#look fails even if vips is installed by brew.

> sharp@0.12.0 install /Users/User/node_modules/sharp
> node-gyp rebuild

ERROR: Please install libvips on Mac OS by running the following command:
 brew install homebrew/science/vips --with-webp --with-graphicsmagick
gyp: Call to 'LDD_VERSION="" node -e "require('./binding').download_vips()"' returned exit status 1. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/Users/User/.nvm/versions/node/v4.2.2/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:355:16)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.0.0
gyp ERR! command "/Users/User/.nvm/versions/node/v4.2.2/bin/node" "/Users/User/.nvm/versions/node/v4.2.2/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/User/node_modules/sharp
gyp ERR! node -v v4.2.2
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok 
npm ERR! Darwin 15.0.0
npm ERR! argv "/Users/User/.nvm/versions/node/v4.2.2/bin/node" "/Users/User/.nvm/versions/node/v4.2.2/bin/npm" "install" "lovell/sharp#look" "--save"
npm ERR! node v4.2.2
npm ERR! npm  v2.14.7
npm ERR! code ELIFECYCLE

[lovell]

@mahnunchik The look branch requires libvips v8.1.1+ as this is the minimum (recent) version that will work correctly with Windows. Currently only v8.1.0 is available via brew so if you're able to submit a PR to the homebrew-science repo to bump the version to v8.1.1 that would be great.

I am investigating the provision of pre-compiled .dylib files - see https://github.com/lovell/package-libvips-darwin-x64/blob/master/.travis.yml - but this may have to wait until a future release.

Mac users are generally well-served by homebrew. The vast majority of installation-related questions raised here are Windows and Linux related.

[mahnunchik]

I'm going to find out how to update vips in the homebrew-science repo https://github.com/Homebrew/homebrew-science/issues/2977

Thank you for Linux version. I have tested it locally and inside the docker. It works like a charm!

[lovell]

@mahnunchik 👍

[mahnunchik]

@lovell vips has been updated in the brew repo to 8.1.1 but sharp#look could not be installed.
The error is the same as above.

[lovell]

@mahnunchik Thanks for the brew update - I've been able to reproduce this now too.

I know what the problem is so will fix and improve the OS X error messaging (e.g. "you need to install ... " vs "you need to upgrade ... ") over the weekend.

[lovell]

The look branch has been updated and is ready for further testing.

The Windows and Linux CI environments are both healthy.

Windows support will be 64-bit only from v0.12.0.

@vlapo The shared libs for Linux are now a mere 6MB when compressed, about 19MB when uncompressed. I'd forgotten to strip them of debug symbols.

@mahnunchik This should now work with brew-installed vips again on OS X.

[lovell]

Any more comments, suggestions or test results before the look branch merges into master?

[mahnunchik]

I've tested on:

• Ubuntu Linux 14.04 - works
• node:4.2 Docker image - works
• OS X 10.11.1 - works, but I've faced with one problem related to vips package from brew.

vips package does not pass build time checks. So I've been able to install it only by the following comand:

$ brew install vips --with-webp --with-graphicsmagick --without-check

And it works as expected.

[lovell]

@mahnunchik Thanks for testing/confirming. If it makes you feel any better, I'm hearing about all sorts of problems with El Capitan upgrades breaking homebrew, xcode etc.

[papandreou]

On Debian Wheezy I get:

$ npm install git+https://github.com/lovell/sharp#look
-
> sharp@0.12.0 install /home/andreas/express-processimage/node_modules/sharp
> node-gyp rebuild

ERROR: libc version 2.13 requires manual installation
gyp: Call to 'LDD_VERSION="ldd (Debian EGLIBC 2.13-38+deb7u8) 2.13
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper." node -e "require('./binding').download_vips()"' returned exit status 1. while trying to load binding.gyp
gyp ERR! configure error 
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onCpExit (/home/andreas/.nvm/v0.10.40/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:343:16)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:820:12)
gyp ERR! System Linux 3.14.36-20150326-1131-6606ede
gyp ERR! command "node" "/home/andreas/.nvm/v0.10.40/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/andreas/express-processimage/node_modules/sharp
gyp ERR! node -v v0.10.40
gyp ERR! node-gyp -v v1.0.1
gyp ERR! not ok 
npm WARN optional dep failed, continuing sharp@0.12.0

[papandreou]

Oh, I should look at https://github.com/lovell/sharp/blob/look/docs/install.md instead, of course, and it does say Debian 8. Bummer.

[lovell]

@papandreou Yes, sorry, I had to draw a line somewhere. Debian Wheezy's version of libc is over 4 years' old. Is this likely to be a problem?

[papandreou]

I'm stuck with Wheezy in my production environment for the time being, so for me personally, yes :(

Do you have any idea how much work it'd be?

[lovell]

@papandreou It might be as simple as changing the base OS in the Dockerfile used to generate the binaries.

If you'd like to experiment, you can run ./packaging/build.sh to generate the packaging/libvips-8.1.1-lin.tar.gz file. If present, this file is used instead of downloading from bintray at npm install time.

[papandreou]

@lovell That worked, yay! PR here: #307

[lovell]

I've managed to locally reproduce the heap corruption problem affecting Node v4 and v5 on the Windows CI environment. I think it's something to do with thread switching but I haven't figured it out yet. Here's the stacktrace should anyone else have any ideas:

ntdll!RtlpNtMakeTemporaryKey+0x4000
ntdll!memset+0x160da
msvcrt!free+0x1c
node!node::Start+0x53e
node!v8::Testing::DeoptimizeAll+0x12bdf3
node!v8::Testing::DeoptimizeAll+0x12c5f5
node!v8::Testing::DeoptimizeAll+0xa5ca3
node!v8::Testing::DeoptimizeAll+0xa4e81
node!v8::Testing::DeoptimizeAll+0x14d4fe
node!v8::Unlocker::~Unlocker+0x1429ce

[lovell]

Windows problems finally fixed with commit 3dbedf1 - there was a mismatched malloc/free - see https://ci.appveyor.com/project/lovell/sharp/build/269

[lovell]

The look branch has now been merged to the master branch pending release.

[lovell]

The forthcoming v0.12.0 release will provide pre-compiled binaries for 64-bit Windows and Linux. I've moved the work to investigate the best way to support OS X (and ARM) to #312.

Thank you very much to everyone on this thread for all your suggestions, help and testing during the last year and a half!

I'm closing+locking this issue now - please open a new issue if you're having any problems with the pre-compiled binaries or have further thoughts.

[lovell]

v0.12.0 now available via npm - tell all your friends!