Build fails when fetching from tag

[g5pw]

I'm in the process of writing a port for julia to install it with MacPorts. I recently ran into a problem. I saw you tagged some releases, but when someone downloads the .tgz from a tag, it won't include the .git directory, so all git commands fail (specifically the submodule init for Rmath, nginx, openlibm and libuv, and thus the build process).

Also, on a different note, please please use a configure script. Writing an extra variable for each build dependency is... unfortunate.

[ViralBShah]

I guess the way to do this is that if the git stuff fails, the fallback should be to download a tarball of each submodule.

Also, one quick way to ease the pain about the various USE_SYSTEM_X variables is to have a collection of config files: Make.user.macports, Make.user.brew, Make.user.ubuntu, etc.

[pao]

Do the tarballs include the submodule contents? If so, the submodule updates could perhaps be made nonfatal or skipped if there's no .git directory

[StefanKarpinski]

We don't support building from tag tarballs, but there's no way to turn them off in GitHub. Maybe the best thing to do is check if .git exists and throw a nice informative warning if it doesn't.

[g5pw]

Hmm.. is there any reason for not supporting them? We at MacPorts prefer them as they can be directly downloaded and users are not forced to install git.

On 11/mar/2013, at 16:38, Stefan Karpinski notifications@github.com wrote:

We don't support building from tag tarballs, but there's no way to turn them off in GitHub. Maybe the best thing to do is check if .git exists and throw a nice informative warning if it doesn't.

—
Reply to this email directly or view it on GitHub.

Aljaž Srebrnič a.k.a g5pw
My public key: http://bit.ly/g5pw_pubkey

[StefanKarpinski]

Git is required to use Julia packages and to determine the correct version
label for Julia, so it doesn't make much sense to try to support systems
without git installed. Our binaries ship with their own bundled gits.

On Mon, Mar 11, 2013 at 6:50 PM, Aljaž Srebrnič notifications@github.comwrote:

Hmm.. is there any reason for not supporting them? We at MacPorts prefer
them as they can be directly downloaded and users are not forced to install
git.

On 11/mar/2013, at 16:38, Stefan Karpinski notifications@github.com
wrote:

We don't support building from tag tarballs, but there's no way to turn
them off in GitHub. Maybe the best thing to do is check if .git exists and
throw a nice informative warning if it doesn't.

—
Reply to this email directly or view it on GitHub.

Aljaž Srebrnič a.k.a g5pw
My public key: http://bit.ly/g5pw_pubkey

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2539#issuecomment-14748203
.

[g5pw]

@ViralBShah fallback to tarballs is a good idea, even better if you let me (the user) specify library paths if they are already installed.

The idea is to let MacPorts install all dependencies (libraries, etc.) and then build and install julia. We already have everything except FemtoLisp, libuv, OpenLibm, DSFMT, OpenBLAS, AMOS, double-conversion and Rmath.

I can make ports for the missing libraries, too.

[vtjnash]

g5pw, I'm interested in having a MacPorts installation option. I put together a port file last year for that purpose (I can put it in a gist if you are interested), and created the file contrib/macports.make as a template makefile to show the most useful environment variables to set (although both may be slightly out-of-date now)

Can I ask why git is not generally preferred? I noticed that several ports are already using this method. Also, since Xcode includes git, it doesn't appear to require any additional software installation on Mac.

libuv is an internal dependency that requires a few patches to work with julia. Similarly, FemtoLisp is treated as an internal library, although it could probably be separated out again (it also has a more limited dependency on libuv).

OpenLibm can be replaced with USE_SYSTEM_LIBM (-lm)

[g5pw]

Aha, yes, please put it in a gist, thank you :)
I actually didnt't notice the file in contrib/ I'll take a look.

Git fetch is not preferred mainly because we can mirror tarballs on our servers. And, not all Xcode versions ship with git AFAIK.
The other reason is, we try to separate the build in stages: there's a fetch stage that should download everything, so configure and build can be made offline.

As far as the libraries, I can fetch them as additional tarballs and put them in the julia build dir at build time, if that's preferable.

Aljaž Srebrnič a.k.a g5pw
My public key: http://bit.ly/g5pw_pubkey
Sent from my tablet

On 12/mar/2013, at 03:48, Jameson Nash notifications@github.com wrote:

g5pw, I'm interested in having a MacPorts installation option. I put together a port file last year for that purpose (I can put it in a gist if you are interested), and created the file contrib/macports.make as a template makefile to show the most useful environment variables to set (although both may be slightly out-of-date now)

Can I ask why git is not generally preferred? I noticed that several ports are already using this method. Also, since Xcode includes git, it doesn't appear to require any additional software installation on Mac.

libuv is an internal dependency that requires a few patches to work with julia. Similarly, FemtoLisp is treated as an internal library, although it could probably be separated out again (it also has a more limited dependency on libuv).

OpenLibm can be replaced with USE_SYSTEM_LIBM (-lm)

—
Reply to this email directly or view it on GitHub.

[ViralBShah]

For now, I would prefer that we document that git is required to build. At some point, we need to support src tarballs.

[g5pw]

Roger that! Feel free to ping me when you add tarball support and I'll gladly add julia to macports! ;)

[ViralBShah]

@vtjnash Is your macports formula current?

@g5pw Julia needs a dependency on git anyways for the package manager to function. Given that is the case, would it not be a good idea to use git to checkout julia in macports too?

[g5pw]

@ViralBShah well, as stated above, using git is slower and we can't mirror it, so it's not preferable. I nearly got it to work, but I'd like to be able to separate fetch and build phases at least.

Is there any target I can invoke to download all dependencies?

[vtjnash]

make -C deps getall will download everything or make -C deps get-X will download X. the second is probably better since you wouldn't be building most of the dependencies

git submodule init && git submodule update will fetch all of the git-based dependencies (I think that the .git folder can then be safely deleted, for the purpose of creating tarballs)

here's a gist of my previous attempt: https://gist.github.com/vtjnash/5206861
I would be happy to merge pull requests for modifications to the contrib/macports.make file

[StefanKarpinski]

You could use git
bundlehttps://www.kernel.org/pub/software/scm/git/docs/git-bundle.htmlto
distribute a stand-alone file that can reproduce a full git repo by
itself. For example, you can create a bundle file like this:

git bundle create julia.gitbundle master

Later you can reconstitute this into a git repo like this:

mkdir julia && (cd julia && git init && git pull ../julia.gitbundle master)

It should be noted, however, that this is a poor solution for deploying
Julia since you still need to compile the whole thing, which includes
downloading and compiling external dependencies from source – and it's
quite possible for those downloads to not be available when you're
deploying.

A much more reliable way to deploy Julia is to pre-build it and then make a
tarball of that which can simply be untarred and used immediately. Our
build scripts will do this for you.

On Wed, Mar 20, 2013 at 1:59 PM, Jameson Nash notifications@github.comwrote:

make -C deps getall will download everything or make -C deps get-X will
download X. the second is probably better since you wouldn't be building
most of the dependencies

git submodule init && git submodule update will fetch all of the
git-based dependencies (I think that the .git folder can then be safely
deleted, for the purpose of creating tarballs)

here's a gist of my previous attempt:
https://gist.github.com/vtjnash/5206861
I would be happy to merge pull requests for modifications to the
contrib/macports.make file

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2539#issuecomment-15192615
.

[g5pw]

Thanks for the gist @vtjnash I'll take a look.

@StefanKarpinski well, the way that MacPorts work is, we mirror source tarball and then we have buildbots that compile and archive binaries. That is so we can support as many architectures and environments as possible.

[ViralBShah]

I thought that with macports, one always has to build from source. Can I directly install binaries from macports now?

[g5pw]

If it is permitted by license, yes.

Aljaž Srebrnič a.k.a g5pw
My public key: http://bit.ly/g5pw_pubkey
Sent from my tablet

On 21/mar/2013, at 04:37, "Viral B. Shah" notifications@github.com wrote:

I thought that with macports, one always has to build from source. Can I directly install binaries from macports now?

—
Reply to this email directly or view it on GitHub.

[StefanKarpinski]

How about we work on a MacPort where Julia gets built from source first. We can provide source tarballs that contain all the source that's required, including dependencies and we can work on making things compile from that without requiring git.

[g5pw]

that would be ok, but it's simpler if we do a port for each dependency and find a way to include them when building julia. So, for example I'm gonna write the port for:

• libuv
• FemtoLisp
• libuv
• DSFMT
• AMOS
• D3
• double-conversion
• Rmath

And then, in the Julia portfile, I'd add a dependency on them. So most difficult thing would be instructing the build process to use libraries in predefined (standard) directories.

[StefanKarpinski]

That's wonderful if you can create ports for all of those – I'm sure even people not using Julia will appreciate it.

[Keno]

Note that our version of libuv is a modified version of joyent's libuv and that we're statically linking it.

[staticfloat]

I think we've got a pretty solution for this now, with our USE_SYSTEM_xyz=1 make flags, and NO_GIT=1 make flags, etc... Please open a new issue if you have a specific request, and we'll get right on it.

[tkelman]

These days doesn't github allow you to upload an alternate source tarball for a tag/release? We should replace the automatic tarballs github creates (which won't work because they're missing submodules) with a tarball generated by make source-dist

[StefanKarpinski]

Oh, that's an excellent idea. We really need something that will do all of the stuff required for a release, including this.

[tkelman]

Add it to the list in #7588

[tkelman]

Since this was my idea, I figured I should go and test it. Good news, building from https://github.com/JuliaLang/julia/releases/download/v0.3.0-rc2/julia-0.3.0-rc2_56e443d08c.tar.gz worked nicely on Ubuntu. I don't get a sha in the banner since it's not a git repository, I suppose that's expected. But all the dependencies are there, submodules and tarballs alike, I don't think it had to download anything (would have tested with network disabled, but I was ssh'ed into the machine).

[staticfloat]

@tkelman The git information is specifically packed into base/version_git.jl, so if you build with NO_GIT=1, the information should make it through the build process. NO_GIT should be getting set automatically, but perhaps something funky is going on.

[tkelman]

base/version_git.jl looks good, I was getting the Warning: git information unavailable; versioning information limited message during build though.

[staticfloat]

Yes, that's purposeful. What does your banner look like when you start up?

On Wed, Aug 6, 2014 at 12:41 PM, Tony Kelman notifications@github.com
wrote:

base/version_git.jl looks good, I was getting the Warning: git
information unavailable; versioning information limited message during
build though.

—
Reply to this email directly or view it on GitHub
#2539 (comment).

[tkelman]

  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc2 (2014-08-06 02:15 UTC)
 _/ |\__'_|_|_|\__'_|  |
|__/                   |  x86_64-linux-gnu

probably good enough, just not sure whether the blank second line is expected

[staticfloat]

Nope, that means you're missing all versioning information. Phooey. Can you try unpacking from the tarball again and manually adding NO_GIT=1 to the build process?

[ivarne]

I think that is expected when you build from a tagged release.

See: https://github.com/JuliaLang/julia/blob/master/base/version.jl#L204

[staticfloat]

Ah, that's true. @tkelman what does GIT_VERSION show? Just for verification, this is what I get from the tarball:

$ ./julia 
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc2 (2014-08-06 02:15 UTC)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |  x86_64-linux-gnu

julia> Base.GIT_VERSION_INFO
GitVersionInfo("56e443d08cd0dbab9e39070588d82064e5319be3","56e443d","master",0,"2014-08-06 02:15 UTC",true,0,1.407291352e9)

[tkelman]

@staticfloat same, so I think we're good. Just didn't realize that second line was customizable via TAGGED_RELEASE_BANNER for tagged builds. Can't remember if I ever read DISTRIBUTING.md where this is mentioned.

[staticfloat]

Yeah, I forgot about it as well. I should have picked up on the date after your 0.3.0-rc2; that's the sign that we actually do have the git information encoded after all.