FreeBSD support

[tsgan]

Hi,

We have created necessary changes to support nvm in FreeBSD.
Basically it uses FreeBSD pkg system to install node and npm versions from packages to its necessary places.

The change is at:

master...ateamsystems:master

Please take a look at only nvm.sh related changes and let us know how it looks like.
If it is generally OK then we can clean it up and make pull request.
Please let us know if you have any questions.

thanks a lot,
Ganbold

[ljharb]

Absolutely I do not want anything installing node/npm from anywhere but the official site - distro packages are unofficial, and shouldn't be used to install node. I'm open to things that improve FreeBSD support, but using pkg isn't the right path.

Separately, why change ∞ to ????

[ateam-adam]

ljharb we use pkg because node does not compile cleanly without patches under FreeBSD, especially for past versions, last i had checked which is part of why we use pkg. I feel like the amount of work to back port every major.minor version is not going to happen for FreeBSD .. or it would have already.

[ljharb]

That's quite possibly true. I think that the direction for FreeBSD users here is:

1. get node to start supporting FreeBSD from any point onwards, and nvm will support that too
2. explore a way to hook into nvm and provide the patches for older versions

If node doesn't support FreeBSD, most of the ecosystem likely won't either.

[ateam-adam]

NPM and it's modules for the most part work out of the box with no problem. So the issue is really just node itself. Getting node to support FreeBSD better is a task which is ... something else entirely.

We debated long and hard how to best treat this. We wanted to give you a chance to review this before we simply forked this entirely from NVM and made a "FreeBSD NVM".

Is there anything we could do to avoid this (UTF conversions aside :P) ?

[tsgan]

∞ was changed to ??? due to editor I was using, I can fix that.

But overall there are node and npm versions of official FreeBSD packages exists already so it is easier to install them using pkg system.

[ljharb]

@ateam-adam I think perhaps a better approach is actually hosting archives that are FreeBSD-compatible on a site that mirrors https://nodejs.org/dist - and then setting the NVM_NODEJS_ORG_MIRROR environment variable to point to your mirror. Then you won't require any changes in nvm at all.

[ateam-adam]

Not sure if this is what you mean but with FreeBSD the way it's handled either for a port (compiled by user locally) or a pkg (built by compiling the port on a port building machine) is the port grabs the dist file, from nodejs.org then applies the FreeBSD patches. So there is no "free standing" archive of the resulting code -- except as a package after it's compiled. The port itself just references a software's download and contains patches to apply (I'm being simplistic here).

So if your asking if there is a node-that-works-on-freebsd.tar.gz somewhere there isn't. One could be made but this brings it 100% out of line with all the existing pipelines. We wanted to capitalize on the effort already made by the FreeBSD node portmaintainers. Which, I agree ... there shouldn't need to be 91 lines of patches to make, for example 0.12, compile properly. But again, there it is.

When there is a security or other update since this is tied to ports/pkgs it is addressed as part of the pipeline, we're not having to cut a tar.gz file or 'keep up'. The flip side is, you don't have NVM's range on Linux which is "every version ever". This was the trade off. For our use, the massive (and continuous effort) to support that is not worth it.

Let me know if that makes sense.

[ljharb]

@ateam-adam another possibility, however, is that after #1204 lands, you could, on-the-fly, generate that .tar.gz, put it in the right place, and then initiate the nvm commands. Would that work?

[ljharb]

Actually now that I think about it, that wouldn't work, because the checksums would still not match :-/

[ljharb]

I'm very open to exploring how to make this work - but putting pkg inside nvm.sh doesn't seem to be the right solution to me.

Could you use pkg yourself to install the desired version X of node to nvm_version_path "$(nvm_version X)", and then let nvm take over after that? If that works, I'd be happy to add that as a top-level API (instead of using nvm internals)

[ateam-adam]

I was thinking that might be cool but I think there would be other issues we'd run into. The local user would need the entire ports tree on their machine and it would need to be kept up to date -- something that pkg has thankfully eliminated unless you absolutely want to deal with it. It's not a show stopper but there would be additional complexity in 'cooking' that stuff on the fly, and that's just off the top of my head.

There is also dependencies which pkg handles, which again not a show stopper, but just a potential pandora's box.

And yeah, checksums would no longer match ..

[ateam-adam]

That is what this does unless I mistake your meaning. At no point does node get installed "on the system", all the pkg commands are targeted at /tmp (@tsgan can provide more detail here) then moved into place in nvm_version_path basically. You end up having to duplicate some deps but it's clean and it works, and it is not fragile (really the big thing for me).

And those duplicated deps are duplicated inside the nvm_version_path for a specific version. So it maintains FreeBSD's dep tree for each version.

[ljharb]

Right - I guess I'm asking if you can pass that path directly into pkg outside of nvm - and then you could just override nvm install (potentially with a PR that makes the command configurable)

[ateam-adam]

I will leave this up to @tsgan to answer :)

P.S. Very much appreciate you working to try to find a solution here 👍

[ljharb]

for example - i could make nvm install pass the fully qualified version, and the version path, into a specific internal command - and that command could be overrideable by an env var. Then, you could make an nvm-freebsd wrapper package (that was NOT a fork) that provided a shell function/binary, and set that env var - and then you'd have a robust hook into nvm install.

[tsgan]

@ljharb How do you use nvm install?

nvm install node_version_from_some_path?

Can you explain it in detail?

thanks

[ljharb]

@tsgan Users use it by nvm install <versionish> where "versionish" is described in nvm --help as:

Note: <version> refers to any version-like string nvm understands. This includes:
  - full or partial version numbers, starting with an optional "v" (0.10, v0.1.2, v1)
  - default (built-in) aliases: node, stable, unstable, iojs, system
  - custom aliases you define with `nvm alias foo`

Once it's fully resolved and validated the version number, ensured that it's not already installed, and also resolved the "--reinstall-packages-from" version (and figured out if it should try to install a binary first, before falling back to source install), it calls into one of two internal functions: one for source, one for binaries. My thinking is, one or both of those could be overrideable via an env var - that way, you could provide your own installation function or binary, and nvm could pass to it the destination directory, the fully resolved version number, etc.

[tsgan]

I understand current behaviour of nvm install command.

What I don't understand or don't know is the env vars and the way to use them for installing in FreeBSD case.

How to override them from where? From nvm.sh ? from shell? or from something else?

If I understand your meaning correctly:

1. Install node, npm versions from packages to some place
2. Use nvm install to install those node and npm to right place

But this isn't possible, again as @ateam-adam said checksums etc won't allow.

Please correct me if I'm wrong here.

thanks,

[ljharb]

This hook does not yet exist. If I added it, checksum verification would be contained within the internal method that the hook would overwrite, since checksums are tied to the "nodejs.org/dist" installation source.

Sorry for the confusion - I'm no longer suggesting those two steps you restated from #1207 (comment). I'm now suggesting a totally different alternative from #1207 (comment).

[tsgan]

Then it still needs some changes to nvm.sh.
How do you bypass checking checksum etc in case of tarballs or packages for FreeBSD?
I think it still needs changes in nvm.sh
Please correct me if I'm wrong here.

[ljharb]

You are correct! This proposal would still require changes to nvm itself - however, those changes wouldn't be FreeBSD-specific, so I'd be happy to make them.

Essentially, in this hypothetical future world, you'd do something like:
NVM_INTERNAL_INSTALL_COMMAND=install_from_pkg nvm install 4 and then install_from_pkg v4.5.0 path/to/node/4/directory would be invoked - and install_from_pkg is your own custom function or binary that knows how to install from pkg.

[jbergstroem]

@ateam-adam Our intention (nodejs core devs) is to have freebsd build nodejs out of the box, with or without shared libraries. I just emailed Bradley about helping out with upstreaming the v8 patches.

[ljharb]

@jbergstroem once that was set up, how far back could tarballs be generated?

[jbergstroem]

@ljharb hard to give you a reliable version. I've upstreamed a few freebsd things to v8 across both io.js and node.js. The floating patches right now all looks arm-related; not sure how much testing nvm has done on that even on linux?

[ateam-adam]

@jbergstroem I know this is a lot to ask but I am just trying to gauge: Would this commitment to having FreeBSD compile out of the box remain effectively indefinite in the future? I feel like is a lot to ask, but I do not know how close it is and what those port's patches are for (if they're for ARM, that doesn't really impact me).

My concern is that down the road, motivation for this wanes, an update is rolled out, this breaks and we're stuck. Meanwhile the FreeBSD port maintainer continues to ensure it builds, but that doesn't do us any good -- and we could have just used pkg and it would work. OTOH if this is close to compiling like it does on Linux ... do you have specific versions we should try? I am 99% sure we tried this first when we started this project (internally) at the start of the year and it just exploded which is why we went down the pkg route originally.

In terms of retroactive, just for our own use, as long as the latest versions (and here after) of 0.12, 4.x and 6.x work we'd be happy. Our clients just want to switch between major versions (and have deployments use specific major versions), we don't need to jump back in time per sey.

[ljharb]

For nvm's sake, as long as FreeBSD support is on a consistent version range, without any gaps, then it's easy for me to make an "is FreeBSD supported" function.

[jbergstroem]

@ateam-adam: We're getting a bit off topic here, but FreeBSD is part of our build cluster meaning we compile and test pretty much everything against it. We don't have a FreeBSD/ARM combo as of yet, but if we get more rpi hardware donated towards that cause we would. FreeBSD support isn't necessarily tied to an interest, rather a collaborative effort to ensure quality.

I recommend FreeBSD users to stick with the ports/pkg version if they just need one version. It's in excellent shape, kept up to date and builds against shared libraries.

@ljharb at least 6.x+ should be safe for source builds.

[ateam-adam]

So just having 6.x work doesn't fit our needs, nor I think many others. The main use case for NVM for us is to let our devs switch between 0.12.x, 4.x and 6.x and also be able to run different apps with different versions without needing separate machines.

@jbergstroem I agree we are very off topic. Should node.js support FreeBSD better? Yes please! :) But right now it does not. I just downloaded and played with the source to 6.4.0 and it fails to build. 0.12.15 has different issues, but also fails. Oddly 4.5.0 does compile. For all versions though I did have to fix some stuff: "/usr/bin/env python" in configure won't work on FreeBSD since it references python by specific versions, ie; "python2.7" not "python". This is minor. To be clear 6.4.0 and 0.12.15 had actual compile errors after doing this. I will, for the sake of staying on topic, not include what other failures occurred here for 6.4.0 and 0.12.15.

If we could get that stuff fixed and integrated into node.js quickly using our proposed method of pkg might not be needed and FreeBSD could use NVM with minimal changes. Quickly getting 6.x fixed aside, would node even accept patches for 0.12.x at this point? We would be interested in assisting if this seems possible.

Otherwise I think our proposed NVM patch which uses pkg, externalized like @ljharb suggested, is the solution.

Thoughts everyone?

[jbergstroem]

@ateam-adam if you encounter build failures you should file them at the node repo. They should be against latest versions of either LTS or $head. Please cc me.

[ateam-adam]

@jbergstroem I can devote resources to this, but would that offer would also apply to 0.12.x? If that's not the case unfortunately it does not help our situation unless all 3 versions can be fixed and compile directly. We don't need to go back, just the latest version of each.

Basically unless we can see the latest 0.12.x, 4.x and 6.x compiling without fuss -- such that @ljharb can easily add FreeBSD to NVM -- then our effort is better spent with what we're doing with NVM/pkg since directly compiling won't work for all versions we need.

Let me know if that makes sense.

[jbergstroem]

@ateam-adam please ping me on IRC (jbergstroem@freenode) or email (bugs at bergstroem dot nu) if you want to talk more about building node.js on FreeBSD -- I feel like we're out of line wrt this issue.

[ateam-adam]

Sorry for the delay in updating but @jbergstroem and I did talk last week over IRC. The long and short of it is that:

• 0.12 does not compile out of the box under FreeBSD, which we'd need. There is no path to fixing this since it's so close to EoL. It's a v8 error so its not even node.js directly. Upstream patching is closed basically.
• 4.x and 6.x do compile with minimal environment tweaks I think easily added by NVM (exporting some variables when FreeBSD is detected basically) and the project is very committed to building under FreeBSD.

So with this in mind, we'd still like to continue with our pkg-based version, and pursue the path originally discussed about externalizing the functions as @ljharb originally described before this sidetracked into the "will it compile" discussion. 0.12 support aside, I believe using pkg it is still a nice option to have since compiling node from scratch can be fairly intensive and takes a while. For lower end VMs and other applications this can become more of an issue. This also would extend NVM to other platforms Node.JS might not be testing on (ie; FreeBSD Pi/ARM/toaster)

The background is that I think NVM could "easily" support 4.x and 6.x directly as well, I am more than happy to open a ticket/assist with that once we have the above sorted, but that is a separate issue in my view, and unfortunately doesn't full address our needs today because we need 0.12, too.

Let me know if this makes sense and what the next steps are. We're prepared to create any patches and work with any hooks you can give us to avoid "hacking the core" of NVM or making a fork of it just for this. I liked your idea of hooking this in from an external FreeBSD script if I understood correctly.

@ljharb sorry to make you re-type but it might be good to re-cap what you had originally proposed to get this back on track.

Thank you everyone!

[ljharb]

Thanks for everyone's diligence!

My proposal is this:

1. i create an internal nvm method that accepts, for example, "destination dir", "version", "binary" or "source"
2. i listen to an environment variable: if empty/unset, use the previously referred-to method. else, use the function/binary specified.
3. FreeBSD users can use a package/bash script that simply sets the relevant environment variable to point to its own function/binary. this can then override the internal install behavior to use pkg, for example, in a way that's likely compatible with nvm. That avoids the need for a fork, or to package nvm separately. nvm would commit to maintaining that env var contract, and thus, it could be reliably depended on.

(Obviously step 1 is optional for me, but that's an implementation detail and not relevant to explaining the concept)

If this will work for everybody, then I can begin implementing it. I'll want to wait until #1204 lands, but it should be trivial after that.

[ljharb]

(This is also a duplicate of #130)

[tsgan]

Any updates on this?
Please let us know.

thanks a lot,

[ljharb]

@tsgan will my proposal work for you?

[tsgan]

Yes, it would work for us and we were waiting for your implementation.
Please let us know. Thanks.

[ljharb]

The next release of nvm will contain support for an environment variable $NVM_INSTALL_THIRD_PARTY_HOOK - put a binary, or a shell function name, and it will be invoked with the following arguments:

1. full nvm-style version string
2. "node" or "iojs"
3. "std" (in the future, perhaps "rc", "nightly", etc)
4. "binary" or "source"
5. path to the directory in which to install the version

If it exits nonzero, or if it doesn't install the version properly, the overarching nvm install will fail. If the env var is unset, or the empty string, it will install normally.

This will remain undocumented - this github comment, and one line in the changelog, is all I'll write about it, in an effort to minimize people relying on it. However, I've committed to not breaking it in the future, so you can safely rely on it for FreeBSD :-)

Please let me know, post release, if you have any trouble!

[tsgan]

Sorry for late response.

I think we also need to hook in Uninstall path and in ls-remote path.
Can you make changes accordingly?

thanks a lot,

[ljharb]

@tsgan can you elaborate on why? You can set $NVM_NODEJS_ORG_MIRROR and nvm ls-remote will list from your own custom location, and nvm uninstall just deletes the folder - the intention is that the nvm version dir should be the only place files are created.

[tsgan]

Because we will use binary packages via pkg in FreeBSD and according to http://pkg.freebsd.org the correct packages will be used based on ABI (FreeBSD version and platform):

This server's package sets:

FreeBSD:8:i386 (no longer updated)
FreeBSD:8:amd64 (no longer updated)
FreeBSD:9:i386
FreeBSD:9:amd64
FreeBSD:10:i386
FreeBSD:10:amd64
FreeBSD:11:i386
FreeBSD:11:amd64
FreeBSD:11:aarch64
FreeBSD:12:i386
FreeBSD:12:amd64
Tier-2 support package sets:

FreeBSD:11:armv6
FreeBSD:11:mips
FreeBSD:11:mips64
FreeBSD:12:armv6
FreeBSD:12:mips
FreeBSD:12:mips64

So we need hook in terms of ls-remote.

Uninstall path needs it also, since pkg will install node packages in different place than nvm and then copy necessary files to nvm dir.

[ljharb]

hmm, ok - that's adding a bit more complexity than I'd intended when I originally asked if this would work. I'm leaning towards reverting that hook entirely if this is the only way to solve it.

However, for ls-remote, I'd prefer you point users at your own mirror such that it only lists the available versions there. As for uninstall, in the install hook, after copying the files to the nvm dir, you should be deleting the pkg files - thus needing no cleanup later.

[tsgan]

hmm, ok - that's adding a bit more complexity than I'd intended when I originally asked if this would work. I'm leaning towards reverting that hook entirely if this is the only way to solve it.

However, for ls-remote, I'd prefer you point users at your own mirror such that it only lists the available versions there.

Ok, I will advise it with colleague.

As for uninstall, in the install hook, after copying the files to the nvm dir, you should be deleting the pkg files - thus needing no cleanup later.

Well, maybe I didn't explain you well. Each node version (package) has own dependencies (libraries, other packages etc.) and those dependencies are installed in separate directory. That is why we need to remove those dependencies in uninstall path.

[ljharb]

Right - but if you're using nvm, 100% of the needed files should live only inside $NVM_DIR. Those own dependencies should not be installed in a separate directory.

[tsgan]

I will think and try a bit current code. I will let you know.

thanks a lot,

[tsgan]

Quick question, is node v0.10.x or v0.12.x should be installed in nvm home directory (not in versions/node/ directory) ? Which one should be installed in nvm home directory?

[ljharb]

The install hook is called with the desired version directory - number 5 in #1207 (comment). That way, the source of truth for that path can be left with nvm.

[tsgan]

Another questions, in order to use $NVM_NODEJS_ORG_MIRROR should we need to create index.tab file like in https://nodejs.org/dist/index.tab ? Is it created manually or is there any tool to create it?

[ljharb]

Yes, you'd need to create it. node core has a build process to create it, but you'll probably want to create it manually.

[tsgan]

Ok, it seems just version column works, but is there anywhere to tell to FreeBSD user that iojs versions won't work for FreeBSD?

[tsgan]

Ah, we can set that to empty also :)