Rebuild with custom userimg.jl?

[mlhetland]

If one adds a userimg.jl file with require statements in the base subdirectory of the install and rebuilds Julia, one can get cached, precompiled versions of modules one might use often. Highly useful. And … if one already uses Homebrew to build Julia, it would be very useful if one could somehow manage to do this with the requisite Homebrew formula…

Is there any way you could add something like that? Perhaps some environment variable indicating a userimg.jl file to be copied in, or some extra switch to the build command or something? (I'm a Homebrew n00b, so I don't know if there's a standard way of doing something like this…)

[staticfloat]

There..... is not an easy way to do this via Homebrew. Ideally, we should support this in Julia itself. I have a PR open over here that (among a few other things) would allow you to rebuild your system image from within Julia. I think the best thing for you to do here is to just comment on that PR that you'd like the ability to rebuild your system image.

[staticfloat]

I'm going to close this now, as this isn't something that we can do on the Homebrew side, sorry.

[mlhetland]

Looks like a nice PR :-) As for the Homebrew angle: I don't mean rebuilding just the sysimg – I get that that's tricky. I'm thinking more along the lines of

$ brew install julia --userimg=$HOME/userimg.jl

I.e., just copying the given file into base before the rest of the build (or reinstall). I'd certainly be willing to wait for the full rebuild. The alternative would be building manually, I guess, which seems unnecessary. If you'd rather not add something like this, that's fine, of course. I guess I could do it myself, if I can't find some other usable solution. (All I'm missing is really finding the location of base from inside julia.rb.)

[staticfloat]

Hmmmm. That is definitely more doable.

Since the current directory in a homebrew formula is the build directory, you can just say base. So if you were to get the value of --userimg, you could cp userimg, "base", then tack an include on to the end of sysimg.jl, and you'd be good to go, I think.

[mlhetland]

Ah, nice. As for the include, that's not needed. This is the final line of sysimg.jl:

Base.isfile("userimg.jl") && Base.include("userimg.jl")

So this is an “official but undocumented” hack. Which is used as part of the build process. And that's why I thought it would be useful for julia.rb to support it, so one wouldn't have to resort to manual building to use it :-)

[staticfloat]

Fair enough! Submit a PR and I'll test + merge it. :)

[mlhetland]

Done.

[staticfloat]

Thanks again for this.

[mlhetland]

No prob. Thanks for merging :-)

[sglyon]

How is this supposed to work?

I ran the command:

brew install  --userimg=$HOME/dotfiles/Julia/userimg.jl julia

but I didn't see any userimg.jl in /usr/local/Cellar/julia/0.3.2/share/julia/base/.

How do I use this option and then trigger the full rebuild of sysimg?

[mlhetland]

Well, you need to refer to an actual file; if you have a custom userimg.jl in the directory $HOME/dotfiles/Julia/, that part should be OK. If not, make sure you have such a file (with the proper includes/requires) and supply the correct path.

Second, you might need to use reinstall to have anything change, if you've already got the most recent version installed. This is just a switch to the actuall full Julia build via the brew formula – so it doesn't rebuild the sysimg; it just includes your userimg.jl as part of the sysimg during installation.

So, yeah, either use reinstall or uninstall Julia before using the install command above.

[sglyon]

@mlhetland yes of course, the file $HOME/dotfiles/Julia/userimg.jl does exist.
Here is what I have tried:

~|⇒ cat $HOME/dotfiles/Julia/userimg.jl
require("DataFrames")
require("Optim")
require("Distributions")
~|⇒ brew reinstall --userimg=$HOME/dotfiles/Julia/userimg.jl julia
==> Reinstalling julia
==> Downloading https://juliabottles.s3.amazonaws.com/julia-0.3.2.mavericks.bottle.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/julia-0.3.2.mavericks.bottle.1.tar.gz
==> Pouring julia-0.3.2.mavericks.bottle.1.tar.gz
==> Caveats
Documentation and Examples have been installed into:
/usr/local/Cellar/julia/0.3.2/share/julia

Test suite has been installed into:
/usr/local/Cellar/julia/0.3.2/share/julia/test

To perform a quick sanity check, run the command:
brew test -v julia

To crunch through the full test suite, run the command:
/usr/local/Cellar/julia/0.3.2/bin/julia -e "Base.runtests()"
==> Summary
🍺  /usr/local/Cellar/julia/0.3.2: 480 files, 80M

~|⇒ ls /usr/local/Cellar/julia/0.3.2/share/julia/base/u*
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf16.jl        /usr/local/Cellar/julia/0.3.2/share/julia/base/utf8proc.jl
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf32.jl        /usr/local/Cellar/julia/0.3.2/share/julia/base/util.jl
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf8.jl         /usr/local/Cellar/julia/0.3.2/share/julia/base/uv_constants.jl

~|⇒ brew uninstall julia
Uninstalling /usr/local/Cellar/julia/0.3.2...

~|⇒ brew install --userimg=$HOME/dotfiles/Julia/userimg.jl julia
==> Installing julia from staticfloat/homebrew-julia
==> Downloading https://juliabottles.s3.amazonaws.com/julia-0.3.2.mavericks.bottle.1.tar.gz
Already downloaded: /Library/Caches/Homebrew/julia-0.3.2.mavericks.bottle.1.tar.gz
==> Pouring julia-0.3.2.mavericks.bottle.1.tar.gz
==> Caveats
Documentation and Examples have been installed into:
/usr/local/Cellar/julia/0.3.2/share/julia

Test suite has been installed into:
/usr/local/Cellar/julia/0.3.2/share/julia/test

To perform a quick sanity check, run the command:
brew test -v julia

To crunch through the full test suite, run the command:
/usr/local/Cellar/julia/0.3.2/bin/julia -e "Base.runtests()"
==> Summary
🍺  /usr/local/Cellar/julia/0.3.2: 480 files, 80M

~|⇒ ls /usr/local/Cellar/julia/0.3.2/share/julia/base/u*
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf16.jl        /usr/local/Cellar/julia/0.3.2/share/julia/base/utf8proc.jl
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf32.jl        /usr/local/Cellar/julia/0.3.2/share/julia/base/util.jl
/usr/local/Cellar/julia/0.3.2/share/julia/base/utf8.jl         /usr/local/Cellar/julia/0.3.2/share/julia/base/uv_constants.jl

Notice that under each of the two methods (reinstall or uninstall, then install I don't see my userimg file...

Any other suggestions.

Also, even when I do get the userimg to work, how do I get that into the sysimg?

Thank you

[mlhetland]

Are you sure you've updated the formula? The one you're running doesn't seem to be executing the proper cp command. But if you manage to get the userimg.jl to work (i.e., to have it copied into base during installation, before compilation), it will automatically be included into sysimg, so it should just work from there on.

[mlhetland]

That is, I guess my suggestion boils down to running brew update before trying the reinstall again.

[mlhetland]

Sorry for making these really basic suggestions, BTW. (“Is it plugged in?” <wink>) Actually, I can't get it to work myself, now, so … I guess something is broken. I'll have a look.

[mlhetland]

The problem is that the formula is installing a Homebrew bottle (binary), rather than compiling it. Try to add --build-from-source, and see if that helps.

[mlhetland]

Meh. Using --build-from-source crashed on my end. @staticfloat – any thoughts on why this is? Shouldn't it be possible to compile it, even if the default is to use bottles? If you don't mind using the bleeding-edge version, a workaround for now would be to use --HEAD.

[mlhetland]

The build-from-source issue was the following, BTW:

ld: file not found: /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libgfortran.3.dylib for architecture x86_64

[mlhetland]

Actually, I got the same error trying to build --HEAD, so… I guess that may be something on my end?

@staticfloat: (1) If this isn't just a snafu in my current brew state, it should perhaps be added as an issue? And (2) if --build-from-source works in general, should I update README.md to include this (as the switch won't work when installing a bottle)?

[mlhetland]

I added a PR (#128) for the build instructions. Not sure how to fix the compile issues, though.

[mlhetland]

Aaand … I've broken my Julia. Even running brew reinstall, installing with a bottle, I now have trouble with the Fortran library:

Warning: error initializing module LinAlg:
ErrorException("error compiling __init__: error compiling check_blas: error compiling blas_vendor: could not load module libopenblas: dlopen(libopenblas.dylib, 1): Library not loaded: /usr/local/lib/gcc/x86_64-apple-darwin13.3.0/4.9.1/libgfortran.3.dylib
  Referenced from: /usr/local/opt/openblas-julia/lib/libopenblas.dylib
  Reason: image not found")

My Xcode is up to date. Not sure if this is related to the state of my OS X or my Homebrew…

[staticfloat]

Hahaha, I love waking up to long threads like these. :)

First off; the fact that passing --userimg does not disable bottle building and instead just ignores the option is a bug. Pure and simple. I will fix that soon and force a build from source if --userimg is passed.

Secondly, the reason you're running into these libgfortran issues is because your openblas-julia is too old. You need to:

$ brew update
$ brew rm openblas-julia arpack-julia suite-sparse-julia
$ brew install openblas-julia arpack-julia suite-sparse-julia

[mlhetland]

Right—that'll teach me to actually read the README ;-) Just curious, though: Isn't this kind of updating of versions exactly what a package manager like Homebrew is for? I mean, doing it automatically?-) I guess there's some technical reason why you can't just add some version requirement in the formula?

Anyway, after this fix, running with --build-from-source (which I guess will be unnecessary in the future, once the Homebrew bug is fixed) seems to do the trick.

@spencerlyon2: The thing to look for in your build log is the line

==> cp /path/to/userimg.jl base/userimg.jl

[staticfloat]

Yes, this is exactly what Homebrew is for. Unfortunately, there are some oddities with regards to gfortran that are difficult to deal with on the Homebrew side. You can read about our efforts here.

[mlhetland]

OK, thanks!

[mlhetland]

OK, revisiting this again.

• Currently can't get the --userimg switch to work
• Julia now ships with an in-julia mechanism sort of for rebuilding the sysimg.
• There are issues with using that with the homebrew-installed Julia

The first point: I haven't worked too hard on figuring out what's going on there, as the whole switch is probably a lot less useful now, anyway, with the more standard (albeit probably stop-gap) mechanism. The issue with build_sysimg seems to be just that it can't overwrite the existing image at the default location. Once I did

$ chmod u+w /usr/local/Cellar/julia/0.3.7/lib/julia/sys.*

it works. I.e., I can run the following to rebuild with my userimg:

include(joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "build_sysimg.jl"))
build_sysimg(default_sysimg_path, "native", usrimg; force=true)

It also works if I install the sysimg elsewhere, and load it manually with the -J switch. One might think this is the Right Thing™ to do, but chmod-ing the existing sys.* files yields the following conclusion to the rebuild:

INFO: Linking sys.dylib
INFO: System image successfully built at /usr/local/Cellar/julia/0.3.7/lib/julia/sys.dylib
INFO: Julia will automatically load this system image at next startup

I.e., it seems this is a scenario that should be possible. (I even seem to think it worked earlier?)

So:

• Perhaps the formula should adjust the write permissions (i.e., u+w for the existing sys.* files)?
• Perhaps we should point to build_sysimg rather than maintaining/fixing the --userimg switch? (Or we could just leave that; I guess I may just be having compile trouble in general.)

For now, though, I can always just add a chmod in my rebuild script. Adding something about that in the README might also be an option, perhaps.

[staticfloat]

I agree with you; we should chmod the sys.* files. It looks like homebrew is changing the permissions behind our backs, I imagine it's a "best practices" type of thing, as you typically wouldn't change your files in lib/. Once we do that, I also think we should remove the --userimg switch, since it does seem like the build_sysimg.jl script should do everything we need it to.