Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4 #17993

hanzer · 2016-08-12T16:10:10Z

Today:

sudo freebsd-update fetch
sudo freebsd-update install
sudo pkg upgrade
git config --global core.autocrlf input
git clone https://github.com/JuliaLang/julia.git
cd julia/
git checkout release-0.5
gmake MARCH=native FC=gfortran47

Ends with error:

[snip]
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE

The text was updated successfully, but these errors were encountered:

hanzer · 2016-08-12T16:12:26Z

Just in case it is relevant in some way, this is on an x86/32-bit machine.

tkelman · 2016-08-12T16:54:24Z

probably related to JuliaLang/libuv#31? cc @kpamnany

hanzer · 2016-08-12T17:57:48Z

After this change to Make.inc (I lack sufficient understanding of the issues, factors, and ramifications to know if this is a reasonable thing to do, but just as a superficial experiment):

USE_SYSTEM_LIBUV:=1

and the build process moves forward a bit before the next problem:

 /usr/bin/install -c -m 644 include/libunwind-common.h '/usr/home/hanzer/build/julia/usr-staging/libunwind-1.1-julia2/usr/home/hanzer/build/julia/usr/include'
src/s_cosl.c:44:2: error: #error "Unsupported long double format"
 #error "Unsupported long double format"
  ^
gmake[2]: *** [Make.inc:66: src/s_cosl.c.o] Error 1
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/openlibm.mk:11: build/openlibm-71e79eb6f74d3f04ce724195b8ef5846a70d281e/libopenlibm.so] Error 2
gmake: *** [Makefile:81: julia-deps] Error 2

Following the same pattern, changing Make.inc with:

`USE_SYSTEM_LIBUNWIND:=1`

results in:

src/s_cosl.c:44:2: error: #error "Unsupported long double format"
 #error "Unsupported long double format"
  ^
gmake[2]: *** [Make.inc:66: src/s_cosl.c.o] Error 1
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/openlibm.mk:11: build/openlibm-71e79eb6f74d3f04ce724195b8ef5846a70d281e/libopenlibm.so] Error 2
gmake: *** [Makefile:81: julia-deps] Error 2

so changing Make.inc again, with the additional:

USE_SYSTEM_LIBM:=1

and the build moves forward. To speed things up, trying with dependencies supplied from the FreeBSD package system (e.g., pkg search pcre if found then sudo pkg install pcre):

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=1
USE_SYSTEM_LIBUV:=1
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

which results in:

/usr/home/hanzer/build/julia/deps/SuiteSparse_wrapper.c:2:21: fatal error: cholmod.h: No such file or directory
 #include <cholmod.h>
                     ^
compilation terminated.
gmake[1]: *** [/usr/home/hanzer/build/julia/deps/suitesparse.mk:108: /usr/home/hanzer/build/julia/usr/lib/libsuitesparse_wrapper.so] Error 1
gmake: *** [Makefile:81: julia-deps] Error 2

so:
USE_SYSTEM_SUITESPARSE:=0

compiling then:

 gmake[1]: llvm-config: Command not found

pkg info | grep llvm
llvm37-3.7.1_2                 LLVM and Clang
llvm38-3.8.0_3                 LLVM and Clang

lvm-config is installed as llvm-config37 and llvm-config38, as are the other llvm- programs. I don't suppose there is a simple top-level configuration for such things?

tkelman · 2016-08-12T18:07:37Z

You should be able to set an LLVM_CONFIG variable either on the gmake command line or in a Make.user file (create it at the top level and the makefiles will include it if present, useful to avoid having to patch the existing files, you can just override certain variables instead).

hanzer · 2016-08-12T18:16:31Z

Trying gmake MARCH=native FC=gfortran47 LLVM_CONFIG=llvm-config38

with Make.inc modified like:

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=0
USE_SYSTEM_LIBUV:=1
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

results in:

gmake[2]: *** No rule to make target '/usr/home/hanzer/build/julia/src/support/hashing.o', needed by '/usr/home/hanzer/build/julia/src/support/libsupport.a'. Stop.
gmake[1]: *** [Makefile:158: support/libsupport.a] Error 2
gmake: *** [Makefile:55: julia_flisp.boot.inc.phony] Error 2

tkelman · 2016-08-12T18:45:53Z

That might be the fault of USE_SYSTEM_LIBUV which isn't likely to work right now. Maybe try modifying deps/libuv.version to find an older commit of https://github.com/JuliaLang/libuv that builds on freebsd? We used to have a few contributors more actively testing on that platform and submitting patches to keep things working.

hanzer · 2016-08-12T21:05:46Z

I am reluctant to mention this, but a statement like:

Currently Supported Platforms

Linux
Darwin/OS X
FreeBSD
Windows

might lead one to believe that reasonable quality assurance procedures [for each of those platforms] is part of the release engineering process (e.g., the software builds and passes it's own tests).

And I am very reluctant to use a term like "professional ethics" but it seems like more explicit, matter-of-fact statements about the reality of the current situation is a better way to attract and appeal to highly technical people (the science and engineering communities).

johnmyleswhite · 2016-08-12T21:20:39Z

Perhaps the easiest thing to do is to submit a PR that removes the line that troubles you? I suspect no one would be not reluctant to remove official claims of support for a platform that they agree is no longer actively supported.

tkelman · 2016-08-12T21:27:49Z

Sorry things aren't working. We don't currently have buildbots or CI for FreeBSD, or an active maintainer for it. The last time that line of the readme was touched was in 2012 (https://github.com/JuliaLang/julia/blame/5838a12ba037f40b31affe5b408edcf9ce4bed28/README.md#L53), before Julia had buildbots or CI for any platform. You're correct that we should do a better job about keeping the readme up to date. It tends to get updated when a platform initially starts working (e.g. recent arm and power support), but not so much when support for a platform regresses, if no one actively notices and reports issues like this one.

kpamnany · 2016-08-12T23:51:50Z

Julia's libuv is required to build Julia since upstream libuv hasn't merged libuv/libuv#597 yet (adds uv_thread_detach() and uv_thread_setaffinity()).

FreeBSD appears to have thread affinity support (http://www.freebsd.org/cgi/man.cgi?manpath=FreeBSD%208.2-RELEASE&query=pthread_setaffinity_np), but CPU set management appears to be different. Simplest answer right now is to patch libuv to exclude affinity setting on FreeBSD also. Add FreeBSD at src/uv-common.c:423:

#if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)

It might be better to invert the #if to only select those OSes we know support affinity setting rather than excluding the ones that don't...

I can't raise a PR for this anytime soon, sorry. Also, can't speak to the other issues.

Also, this whole problem should go away in the next threading runtime because I'm using hwloc there.

hanzer · 2016-08-13T03:26:25Z

There currently is no src/uv-common.c in my git checkout release-0.5

But find . -type f -name "uv-common.c" finds this:
./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c

A gmake cleanall followed by a gmake distcleanall
then a gmake MARCH=native FC=gfortran47 LLVM_CONFIG=llvm-config38
with this Make.inc modification:

USE_SYSTEM_LLVM:=1
USE_SYSTEM_LIBUNWIND:=1
USE_SYSTEM_PCRE:=1
USE_SYSTEM_LIBM:=1
USE_SYSTEM_OPENLIBM:=0
UNTRUSTED_SYSTEM_LIBM:=0
USE_SYSTEM_OPENSPECFUN:=0
USE_SYSTEM_DSFMT:=0
USE_SYSTEM_BLAS:=1
USE_SYSTEM_LAPACK:=1
USE_SYSTEM_FFTW:=1
USE_SYSTEM_GMP:=1
USE_SYSTEM_MPFR:=1
USE_SYSTEM_ARPACK:=1
USE_SYSTEM_SUITESPARSE:=0
USE_SYSTEM_LIBUV:=0
USE_SYSTEM_UTF8PROC:=1
USE_SYSTEM_MBEDTLS:=1
USE_SYSTEM_LIBSSH2:=1
USE_SYSTEM_LIBGIT2:=1
USE_SYSTEM_PATCHELF:=1

fails with:

config.status: executing depfiles commands
config.status: executing libtool commands
  CC       src/libuv_la-fs-poll.lo
  CC       src/libuv_la-inet.lo
  CC       src/libuv_la-threadpool.lo
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE

(and there is no src/uv-common.c file)

yuyichao · 2016-08-13T03:33:49Z

./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c this is the src/uv-common.c

hanzer · 2016-08-13T03:45:31Z

Changed ./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c
to this:

    422 #else
    423 #if defined(__APPLE__) || defined(_AIX) || || defined(__FreeBSD__)
    424   return -ENOTSUP;
    425 #else
    426   int i;
    427   cpu_set_t cpuset;

The error is a bit different:

  CC       src/libuv_la-threadpool.lo
  CC       src/libuv_la-uv-common.lo
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c: In function 'uv_thread_setaf
finity':
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:378:3: error: unknown type na
me 'cpu_set_t'
   cpu_set_t cpuset;
   ^
In file included from /usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:26:0:
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/include/uv.h:1474:33: error: 'CPU_SETSIZE' un
declared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c:380:23: note: in expansion of
 macro 'UV_CPU_SETSIZE'
   assert(mask_size >= UV_CPU_SETSIZE);
                       ^

hanzer · 2016-08-13T04:34:34Z

err, correction: changed ./deps/srccache/libuv-28743d6091531340cfe316de2b2d385fe1778ff5/src/uv-common.c like this:

    374 #if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)
    423 #if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)

and the build gets to:

    CC src/jl_uv.o
    CC src/simplevector.o
    CC src/APInt-C.o
g++: error: unrecognized command line option '-Wcovered-switch-default'
g++: error: unrecognized command line option '-fcolor-diagnostics'
gmake[1]: *** [Makefile:121: APInt-C.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

g++ -v

Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc48/gcc/i386-portbld-freebsd10.1/4.8.5/lto-wrapper
Target: i386-portbld-freebsd10.1
Configured with: /wrkdirs/usr/ports/lang/gcc/work/gcc-4.8.5/configure --disable-multilib --disable-bootstrap --disable-nls --enable-gnu-indirect-function --libdir=/usr/local/lib/gcc48 --libexecdir=/usr/local/libexec/gcc48 --program-suffix=48 --with-as=/usr/local/bin/as --with-gmp=/usr/local --with-gxx-include-dir=/usr/local/lib/gcc48/include/c++/ --with-ld=/usr/local/bin/ld --with-pkgversion='FreeBSD Ports Collection' --with-system-zlib --with-ecj-jar=/usr/local/share/java/ecj-4.5.jar --enable-languages=c,c++,objc,fortran,java --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/info/gcc48 --build=i386-portbld-freebsd10.1
Thread model: posix
gcc version 4.8.5 (FreeBSD Ports Collection)

vtjnash · 2016-08-13T04:57:15Z

It seems that the compiler used to build llvm supported those flags, but the compiler you are trying to use to build Julia does not. I think the userspace of FreeBSD may be entirely built with clang now?

hanzer · 2016-08-13T05:04:26Z

Yes, I believe so.

NOTE: Clang/LLVM is the system compiler on several platforms in FreeBSD 10.0 and later, and GCC is not installed by default.

https://wiki.freebsd.org/BuildingFreeBSDWithClang

yuyichao · 2016-08-13T06:31:42Z

In another word, since llvm-config dumps the compiler flags it is compiled with, you should compile julia with a compiler that is compatible with the compiler llvm is compiled with.

hanzer · 2016-08-13T16:59:48Z

To move forward, last night I set USE_SYSTEM_LLVM:=0 in Make.inc and launched the build. The results of that are:

In file included from /usr/home/hanzer/build/julia/src/support/ios.h:7:0,
                 from /usr/home/hanzer/build/julia/src/support/libsupport.h:14,
                 from /usr/home/hanzer/build/julia/src/julia.h:12,
                 from /usr/home/hanzer/build/julia/src/threading.c:22:
/usr/home/hanzer/build/julia/src/threading.c: In function 'jl_start_threads':
/usr/home/hanzer/build/julia/usr/include/uv.h:1474:33: error: 'CPU_SETSIZE' undeclared (first use in this function)
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/src/threading.c:567:20: note: in expansion of macro 'UV_CPU_SETSIZE'
     char *cp, mask[UV_CPU_SETSIZE];
                    ^
/usr/home/hanzer/build/julia/usr/include/uv.h:1474:33: note: each undeclared identifier is reported only once for each function it appears in
 #  define UV_CPU_SETSIZE        CPU_SETSIZE
                                 ^
/usr/home/hanzer/build/julia/src/threading.c:567:20: note: in expansion of macro 'UV_CPU_SETSIZE'
     char *cp, mask[UV_CPU_SETSIZE];
                    ^
/usr/home/hanzer/build/julia/src/threading.c:567:15: warning: unused variable 'mask' [-Wunused-variable]
     char *cp, mask[UV_CPU_SETSIZE];
               ^
gmake[1]: *** [Makefile:117: threading.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

I guess an #if defined(__FreeBSD__) is needed in usr/include/uv.h?

I am trying (blindly (I don't know what this means)):

   1471 # if defined(__APPLE__) || defined(_AIX) || defined(__FreeBSD__)
   1472 #  define UV_CPU_SETSIZE        8
   1473 # else
   1474 #  define UV_CPU_SETSIZE        CPU_SETSIZE
   1475 # endif
   1476 #endif

and compilation continues.

hanzer · 2016-08-13T17:04:27Z

Now the build ends with:

In file included from /usr/home/hanzer/build/julia/src/intrinsics.cpp:7:0,
                 from /usr/home/hanzer/build/julia/src/codegen.cpp:1504:
/usr/home/hanzer/build/julia/src/ccall.cpp: In function 'bool runtime_sym_gvs(const char*, const char*, MT&&, llvm::GlobalVariable*&, llvm::GlobalVariable*&, void**)':
/usr/home/hanzer/build/julia/src/ccall.cpp:112:17: error: 'to_string' is not a member of 'std'
         name += std::to_string(globalUnique++);
                 ^
gmake[1]: *** [Makefile:121: codegen.o] Error 1
gmake: *** [Makefile:90: julia-src-release] Error 2

hanzer · 2016-08-13T17:59:48Z

In another word, since llvm-config dumps the compiler flags it is compiled with, you should compile julia with a compiler that is compatible with the compiler llvm is compiled with.

Can you give an example of how this might be accomplished?

tkelman · 2016-08-13T18:27:59Z

Possibly need to set different combinations of USECLANG=1 and/or USEGCC=0 if those aren't already getting set by default for freebsd?

hanzer · 2016-08-15T05:36:58Z

The compile times are too long for [systematic] guessing, I'm giving up on this for now. The Julia RPM (Version 0.4.6) installed easily on CentOS-7 (x86_64-unknown-linux-gnu), I can explore it there.

I'll probably check periodically for any progress on the [potential(?)] FreeBSD port.

...Scientific analysis software without rigorous quality assurance and engineered/maintained releases; how many actual scientists and engineers rely on that? Should they?

ararslan · 2016-08-15T07:17:16Z

Scientific analysis software without rigorous quality assurance and engineered/maintained releases; how many actual scientists and engineers rely on that?

I think that's an unfair accusation to make. Julia has well maintained releases (note all of the backport tags in PRs) and excellent QA. Plus, despite being a fairly young language, Julia is used for a lot of serious science and engineering. For major platforms on which continuous integration, testing, and building is regularly run, it's known to work fantastically well. As was said previously, there are no active maintainers on FreeBSD; the only way we'll know whether there are issues on that or any platform other than Windows, OS X, and Linux is through issues such as this. So thank you very much for reporting the issue! Now it's known and can be tracked.

Should they?

If anything, I'd encourage more scientists and engineers to rely on Julia. 😉

hanzer · 2016-08-19T19:25:07Z

...has well maintained releases [snip] and excellent QA. [snip] If anything, I'd encourage more scientists and engineers to rely on Julia.

It is probably important not to let sentiment and enthusiasm become a substitute for traction with reality. The systems engineering of high reliability tools requires sobriety and rigor. This might be a lot to ask from the mainstream computer programming technologies where a pervading goal seems to be the creation of cult-like fanatics (battology in typical software documentation is so severe one can imagine what the ancients might have meant by "idolatry").

I suspect significant advancement could be made with more sophisticated socio-technical systems supporting the measurement, analysis, and regulation of Limbic System Class Information Processing within the software development communities.

tkelman · 2016-08-19T19:34:57Z

Uh, what?

Try latest master, this was probably fixed by #18063.

JeffBezanson · 2016-08-19T19:58:04Z

more sophisticated socio-technical systems

In other words, adding freebsd to our CI testing matrix.

ararslan · 2017-06-16T02:56:48Z

Julia now builds out of the box on FreeBSD 11.0-RELEASE.

tkelman added the system:freebsd Affects only FreeBSD label Aug 12, 2016

ararslan closed this as completed Jun 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4 #17993

Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4 #17993

hanzer commented Aug 12, 2016 •

edited by yuyichao

Loading

hanzer commented Aug 12, 2016

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016 •

edited by yuyichao

Loading

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016

Currently Supported Platforms

johnmyleswhite commented Aug 12, 2016

tkelman commented Aug 12, 2016

kpamnany commented Aug 12, 2016

hanzer commented Aug 13, 2016

yuyichao commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

vtjnash commented Aug 13, 2016

hanzer commented Aug 13, 2016

yuyichao commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

tkelman commented Aug 13, 2016

hanzer commented Aug 15, 2016

ararslan commented Aug 15, 2016

hanzer commented Aug 19, 2016 •

edited

Loading

tkelman commented Aug 19, 2016

JeffBezanson commented Aug 19, 2016

ararslan commented Jun 16, 2017

Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4 #17993

Julia release-0.5 fails to build on FreeBSD 10.3-RELEASE-p4 #17993

Comments

hanzer commented Aug 12, 2016 • edited by yuyichao Loading

hanzer commented Aug 12, 2016

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016 • edited by yuyichao Loading

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016

tkelman commented Aug 12, 2016

hanzer commented Aug 12, 2016

Currently Supported Platforms

johnmyleswhite commented Aug 12, 2016

tkelman commented Aug 12, 2016

kpamnany commented Aug 12, 2016

hanzer commented Aug 13, 2016

yuyichao commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

vtjnash commented Aug 13, 2016

hanzer commented Aug 13, 2016

yuyichao commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

hanzer commented Aug 13, 2016

tkelman commented Aug 13, 2016

hanzer commented Aug 15, 2016

ararslan commented Aug 15, 2016

hanzer commented Aug 19, 2016 • edited Loading

tkelman commented Aug 19, 2016

JeffBezanson commented Aug 19, 2016

ararslan commented Jun 16, 2017

hanzer commented Aug 12, 2016 •

edited by yuyichao

Loading

hanzer commented Aug 12, 2016 •

edited by yuyichao

Loading

hanzer commented Aug 19, 2016 •

edited

Loading