Provide a Caffe package in Debian #2601

cdluminate · 2015-06-14T10:24:16Z

Status

Caffe packages are available for Debian/unstable.
Caffe packages are failing to build for Ubuntu-devel and need to be patched.

Last update: Dec.20 2016

Draft guide

Deploy Caffe with merely one command.

Brief Guide for Debian/unstable users

Only experienced linux users are recommended to try Debian/unstable (Sid).
To install caffe, first make sure you have something like the follows in file /etc/apt/sources.list:
(Uncomment the second line if you want to re-compile caffe locally.)

deb http://ftp.cn.debian.org/debian sid main contrib non-free
#deb-src http://ftp.cn.debian.org/debian sid main contrib non-free

Then update apt cache and install it. Note, you cannot install both the cpu version and the cuda version.

# apt update
# apt install [ caffe-cpu | caffe-cuda ]
# caffe

It should work out of box. I hope this work is helpful since there are many people struggling at the Caffe compiling process.

Here are some notes:

Please re-compile OpenBLAS locally with optimization flags for sake of performance. This is highly recommended if you are writing a paper. The way to re-compile OpenBLAS from Debian source is very similar with the next subsection.
If you are going to install caffe-cuda, it will automatically pull the CUDA package and the nvidia driver packages. The installation procress may fail if any part of the caffe dependency chain gets into trouble. That is to say, please take care if you have manually installed or significantly modified nvidia driver or CUDA toolkit or protobuf or any other related stuff.
if you encountered any problem when installing caffe-cuda on a clean Debian system, report bug to me (via Debian's bug tracking system) please.
If you encountered any problem when installing caffe-cpu, please report bug to me via Debian's bug tracking system.
Both of caffe-cpu and caffe-cuda contain a manpage (man caffe) and a bash complementation script (caffe <TAB><TAB>, caffe train <TAB><TAB>). Both of them are still not merged into caffe master.
The python interface is Python3 version: python3-caffe-{cpu,cuda}. No plan to support python2.

Compiling your custom caffe package on Debian/unstable

There is no promise for the content in this subsection. If you just want to compile again from the source without any change, the following should work as expected. If you want to compile it with e.g. CUDNN support, you should at least be able to read and hack the file debian/rules under the source tree (It's a Makefile).

First make sure you have a correct deb-src line in your apt source list file. Then we compile caffe with several simple commands.

# apt update
# apt install build-essential debhelper devscripts    # These are standard package building tools
# apt build-dep [ caffe-cpu | caffe-cuda ]    # the most elegant way to pull caffe build dependencies
# apt source [ caffe-cpu | caffe-cuda ]    # download the source tarball
# cd caffe-XXXX    # now we enter into the source tree
[ ... optional, make your custom changes at your own risk ... ]
# debuild -B -j4    # build caffe with 4 parallel jobs (similar to make -j4)
[ ... building ...]
# debc    # optional, if you want to check the package contents
# debi    # install the generated packages

FAQ

where is caffe-cudnn?
Due to legal reason the cudnn library cannot be redistributed. I'll be happy to make this package when CUDNN becomes re-distributable. The workaround is to install cudnn by yourself, and hack at least the debian/rules file if you really want the caffe *.deb packages with CUDNN support.
how to report bug via Debian bug tracking system?
See https://www.debian.org/Bugs/ .
I installed the CPU version, what should I do if I want to switch to CUDA verison?
sudo apt install caffe-cuda, apt's dependency resolver is smart enough for this.
Where is the examples, the models and other documentation stuff?
sudo apt install caffe-doc; dpkg -L caffe-doc

The text was updated successfully, but these errors were encountered:

bhack · 2015-06-14T10:31:36Z

Probably #2523 could help you.

cdluminate · 2015-06-14T10:46:50Z

@bhack Thank you for this info.
I'm now struggling with caffe's Makefile and Debian packaging rules.
Maybe I will have to look into compiler issues/etc someday in the future.

bhack · 2015-06-14T10:59:32Z

Actually caffe maintain two build systems: Makefile and Cmake.

flx42 · 2015-06-14T23:57:39Z

Is there really a need to provide both Atlas and OpenBLAS versions? I don't think it makes a big difference performance-wise so I won't be upset if you were to settle on one BLAS library and only provide caffe-cpu and caffe-gpu.
Of course that's only my personal opinion.

cdluminate · 2015-06-16T10:41:41Z

@bhack Yes, and I prefer to use Makefile :-)

cdluminate · 2015-06-16T11:11:48Z

@flx42 Initially I don't want to force users to use one of the BLASes.
However thank to your reminder I realized that, maybe it's really not necessary to provide both atlas version and openblas version, because those who really need extraordinary performance will build caffe by themselves, linking with some proprietary software/libraries (cuDNN, MKL).

These blogs are found in order to make a choice between atlas and openblas:
this one , this one,
this one,
and this one

Well, seems that I should take openblas ?

talda · 2015-06-16T11:18:33Z

I completely agree with @flx42 's comment.
Those who want a specific configuration will compile for themselves anyway, and for those who just want to try it out on a reasonable configuration it wouldn't matter.

Sometimes too much choice is not that good.

I think that OpenBLAS is a better choice

cdluminate · 2015-06-16T11:26:29Z

@talda OK, thank you for feedback :-)

bhack · 2015-06-16T11:43:47Z

See also https://wiki.debian.org/DebianScience/LinearAlgebraLibraries

cdluminate · 2015-06-16T12:34:31Z

@bhack Gotcha.
This is inspiring:

The optimal way of using these two packages is to recompile them locally on the machines on which they are to be used.

and README.Debian of openblas says:

 34 Compile the package:
 35 
 36 $ cd openblas-<VERSION>
 37 $ fakeroot debian/rules custom
 38 
 39 It should produce a package that you can install with:
 40 
 41 $ sudo dpkg -i ../libopenblas-base_<DEBVERSION>.deb

Then IMHO

it seems to be of less and less need to provide many versions of compiled caffe
it could be better if I provide similar custom target like atlas/openblas does

cdluminate · 2015-06-16T12:52:37Z

By the way, the alternative link libblas.so.3 will not work for caffe, because e.g. the reault of readelf -d against openblas libcaffe.so is

 0x0000000000000001 (NEEDED)             Shared library: [libopenblas.so.0]
[...]
 0x000000000000000e (SONAME)             Library soname: [libcaffe.so.0]

and the result of readelf -d caffe.bin is

 0x0000000000000001 (NEEDED)             Shared library: [libcaffe.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglog.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgflags.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

We see libcaffe.so is directly linked to openblas, so alternating libblas.so makes no difference.

Note that, I added SONAME and fixed RPATH issue for distro, so the result on your local caffe may differ from mine.

cdluminate · 2015-07-09T03:01:42Z

The packaging work is available at the temp repo:
http://anonscm.debian.org/cgit/users/cdluminate-guest/caffe.git
it passes my local build and the core component works, but is still under heavy development.

These packages are provided:

Package: caffe
Package: libcaffe
Package: libcaffe-dev
Package: caffe-cuda
Package: libcaffe-cuda
Package: libcaffe-cuda-dev
Package: caffe-doc

And maybe python-caffe packages will be added in the future.

cdluminate · 2015-07-09T05:40:19Z

The repo http://anonscm.debian.org/cgit/users/cdluminate-guest/caffe.git/
has trouble to be cloned.

So I just made a mirror for it:
https://github.com/CDLuminate/caffe.git

cdluminate · 2015-08-07T08:50:52Z

Packaging work was moved to Debian-Science:
http://anonscm.debian.org/cgit/debian-science/packages/caffe.git

cdluminate · 2015-08-07T08:52:12Z

Current summary:
https://lists.debian.org/debian-science/2015/08/msg00030.html

shelhamer · 2015-08-07T16:25:34Z

Thank you for working on packaging. This has been requested a lot.
On Fri, Aug 7, 2015 at 01:52 Lumin Zhou notifications@github.com wrote:

Current summary:

https://lists.debian.org/debian-science/2015/08/msg00030.html

—
Reply to this email directly or view it on GitHub
#2601 (comment).

cdluminate · 2015-09-02T08:51:06Z

Good news, the Debian Packaging of Caffe is done.
http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

Current packaging scheme provided these Debian package:
Suite 1: caffe-cpu:

caffe-cpu
libcaffe-cpu0
libcaffe-cpu-dev
python3-caffe-cpu

Suite 2: caffe-cuda

caffe-cuda
libcaffe-cuda0
libcaffe-cuda-dev
python3-caffe-cuda

For pyhton2 caffe package please have a look at debian/README.Debian in the packaging repo.

Currently I'm running the last build test on my machine,
and will soon find sponsorship to upload it to debian/experimental.

If you can't wait for Debian Official Archive to accept this package,
you can build package by yourself.
(you can do simply debuild or do custom build following
instructions from debian/README.Debian)

I'll provide a new installation guide once the caffe was uploaded to
debian/experimental, for Debian and possibly Ubuntu users.
(well, installing Debian packages on Ubuntu is BAD IDEA but I hope it works when there is e.g. urgent need.)

P.S. Did you Caffe Devs plan any release date for caffe-1.0 ?
Please tell me if there is, so I can schedule my working time for importing caffe-1.0 into Debian Packaging repo.

bhack · 2015-09-02T09:25:18Z

Do you plan to re-organize the modularization after #2523?

cdluminate · 2015-09-02T11:42:21Z

Well, I'm not going to change the modularization after that merger.
And I think providing all IO compatibility is a good choice for Debian package.
:-)
@bhack

cdluminate · 2015-09-02T16:03:35Z

Oops, seems that there are some troubles with python3-* packages.
By the way which version of python is recommended? 2 or 3?

shelhamer · 2015-09-02T18:27:31Z

@cdluminate thanks for your packaging work! For pycaffe I think Python 2 is the most common choice but some use Python 3 and both are included in our continuous integration.

cdluminate · 2015-09-03T06:24:07Z

wishlist: more cmake flags -

+ USE_PYTHON2
+ USE_PYTHON3
- python_version

which means, if I turn USE_PYTHON2 and USE_PYTHON3 on, py2caffe and py3caffe will both be built. Then I can provide both python2-caffe* and python3-caffe* packages at the same time, without hacking cmake system too much, or compiling caffe 4 times.

currently there's no python3 version of google.protobuf in Debian, so I‘d better to bump caffe's python interface version to python2.

cdluminate · 2015-11-10T04:02:02Z

This is a packaging status ping, as said in #3311

Caffe itself currently doesn't block packaging at all (it was completed). The real trouble and the real blocker is CUDA from Debian Unstable/Experimental, which is quite out of date (6.5.14 currently) so that it refuses to work with GCC-5. And that is what I'm looking into recently.

micahcc · 2015-11-28T04:06:34Z

Hmm. So will the package show up in experimental now that there is a package in? http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

cdluminate · 2015-11-28T06:02:59Z

@MicahChambers I plan to upload Caffe onto experimental once Debian's CUDA version is working with GCC-5. This may take several more months.

http://anonscm.debian.org/cgit/debian-science/packages/caffe.git/

is my working repo. You may see there is a release version of caffe for experimental in debian/changelog but actually it was blocked by cuda and hence not uploaded.

micahcc · 2015-11-28T07:06:07Z

Thanks for the reply. Thats kind of a bummer. Debian packages are a pain in the ass.

lukeyeager · 2016-07-05T17:45:14Z

And it seems that this change fixes all the failures above: #3731

Glad to hear it. Did you change the source tarball or include those commits as patches? Just curious what the preferred Debian workflow is.

cdluminate · 2016-07-06T00:39:10Z

Did you change the source tarball or include those commits as patches?

Will cherry-pick the patches for the next upload 1.0.0~rc3-4 for CPU version and 1.0.0~rc3-2 for cuda version.

Just curious what the preferred Debian workflow is.

The release experimental can be used for debugging. I build package on amd64 locally, and then upload it to experimetal, which triggers the Debian automatic build machine to build the uploaded source, so I can see on which architecture it fails to build. Now have the result and fix is needed.

CUDA version was uploaded into experimental.

cdluminate · 2016-07-06T00:58:03Z

@shelhamer @lukeyeager

Updated the post on the top of this page with a brief and informal guide. However only Debian/unstable users are recommended to have a try (Do production research machines use Debian/Sid? I guess no way. The administrator must be very cool to tame that beast).

cdluminate · 2016-07-28T11:49:22Z

@lukeyeager @shelhamer

I'd like to bump the caffe debian package version to the current master branch, since some fixes like

lmdb map size
float comparison [1]
are introduced after the release of rc3, which solves build failures on many architectures. How do you like?

[1] f1a8470

shelhamer · 2016-07-30T00:54:19Z

@cdluminate sure, I think the current master is a reasonable point to make a package. Fixing these failures, which do show up in the wild, would be nice. Thanks!

cdluminate · 2016-10-03T00:48:12Z

Uploaded caffe (cpu version) 1.0.0~rc3+20160930-ga7f950b-1 and caffe (cuda version)1.0.0~rc3+20160930-ga7f950b-1 onto Debian/experimental.

cdluminate · 2016-11-17T10:42:52Z

@shelhamer I'm trying to get caffe-cpu package into the next Debian release (its freeze date is comming soon -- Jan 5 2017). Do you have any release plan within this period?

As mentioned before, the next Debian release is not possible to ship caffe-cuda since CUDA8.0 doesn't support its compiler (GCC-6).

shelhamer · 2016-11-21T20:11:42Z

@cdluminate Thanks for the heads-up about packaging. It has been quite a while since a release at this point so we'll be meeting this week to make a plan for the next release. No promises, but with any luck there'll be a release for the end of the year.

cdluminate · 2016-11-22T02:12:48Z

@shelhamer Thanks. For me the only blocker is python3-protobuf
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844608

That means I can go ahead uploading caffe to unstable (then it will automatically enter the next release) as long as some Debian Developer is willing to have the protobuf package fixed.

cdluminate · 2016-11-25T06:43:53Z

Python3-protobuf (protobuf_3.0.0-8) is pending in Debian's upload queue.
https://tracker.debian.org/pkg/protobuf

cdluminate · 2016-11-28T04:09:20Z

I'm preparing the experimental -> unstable migration for both caffe-cpu and caffe-cuda.
The cuda version is considered because I finally compiled caffe-cuda with CUDA-8.0 and Clang-3.8, which means I have a good chance to close this issue before the end of this year.

If any Debian 9 Stretch user complains about the GCC-6's failure working with CUDA-8.0 when compiling caffe in the future, you can suggest him/her try Clang/LLVM.

$ sudo apt install clang-3.8
$ cd caffe-master
$ mkdir build; cd build
$ cmake .. -DCMAKE_CXX_COMPILER=clang++-3.8 -DCMAKE_C_COMPILER=clang-3.8 <YOUR_FLAGS>

Edit: Maybe I'm wrong on Stretch's default clang version.

cdluminate · 2016-12-03T15:08:52Z

Caffe cpu version[1] and cuda version[2] both available for Debian/unstable.
These sources were pulled into Ubuntu devel by bot, see [3][4].

It remains to be seen whether caffe will be shipped in the next release of Debian.
Besides, Ubuntu-devel is failing to build caffe because their protobuf version
is still staying at 3.0.0-7 instead of >=3.0.0-8 which yields python3-protobuf.

Well... What I can do now is just waiting.

[1] https://tracker.debian.org/pkg/caffe
[2] https://tracker.debian.org/pkg/caffe-contrib
[3] https://launchpad.net/ubuntu/+source/caffe
[4] https://launchpad.net/ubuntu/+source/caffe-contrib

cdluminate · 2016-12-03T16:58:13Z

Replaced the first post of this issue with a draft guide. FAQ is open to anyone.

cdluminate · 2016-12-20T14:47:45Z

@shelhamer @lukeyeager

I'm thinking of closing this bug, since the Debian packages are working well.
There is a draft guide for these packages on the top of this page.

Currently the caffe packages are only available for Debian/Sid, but if you
would like to merge the guide as a part of your document (e.g. extending install_apt.md),
I'll keep updating that guide by PR's. If not, maybe I'll keep updating the top post
here.

BTW, I think install_apt.md should be renamed to install_apt_ubuntu.md, so
it will not conflict with the Debian guide install_apt_debian.md.

https://tracker.debian.org/pkg/caffe
https://tracker.debian.org/pkg/caffe-contrib

cdluminate · 2016-12-20T14:56:50Z

P.S. Today I verified that caffe-cuda package is working with TitanX Pascal card.

naibaf7 · 2016-12-20T19:03:12Z

@cdluminate Would there be a possibility or interest to add OpenCL-Caffe packages?

shelhamer · 2016-12-21T01:28:25Z

@cdluminate If you close the issue because the packages are working I'd vote for including a debian install guide. Otherwise your guide is unlikely to be easily discoverable. If you could make a PR to the documentation when you close we can merge.

BTW, I think install_apt.md should be renamed to install_apt_ubuntu.md, so
it will not conflict with the Debian guide install_apt_debian.md.

I hear you, but i'd rather not change the URL of the existing guide. Could you just add the new guide instead?

cdluminate · 2016-12-21T02:10:27Z

@naibaf7 Providing caffe-opencl is possible, as long as it has a number of users, so that the packaging work will be valuable. Of course, I'd be happy to do that if OpenCL is obviously faster than CUDA in some cases. However https://github.com/soumith/convnet-benchmarks .

I'm also working on Torch and Tensorflow packaging.

cdluminate · 2016-12-21T02:16:55Z

@shelhamer Please look forward to the PR.

cdluminate · 2017-01-05T02:52:09Z

https://tracker.debian.org/pkg/caffe
https://tracker.debian.org/pkg/caffe-contrib

Both the caffe-cpu and the caffe-cuda package has entered Debian/testing. Although caffe-cuda will be removed from testing soon however I cannot find a reason to keep this issue open.

Marking this bug as done. LOL

cdluminate · 2017-04-14T01:56:07Z

Update:

While Stretch/Sid is still shipping Caffe-{cpu,gpu}_rc4 , the rc5 version is available in experimental.

Ubuntu 17.04 has been released several days ago, and this is the first time that a Ubuntu release ships Caffe. Ubuntu 17.04 ships the rc4 version (automatically pulled from Debian), however with a hacky patch applied by Ubuntu guys.

http://launchpadlibrarian.net/305712477/caffe_1.0.0~rc4-1_1.0.0~rc4-1ubuntu1.diff.gz (canceling unit test)

This ubuntu1 patch above is exactly why I don't want to mention the caffe package existence in Ubuntu. It disabled unit test to bypass the unit test failure so that the package can be built successfully. In this case I cannot make sure whether the resulting package is working at all. However, eventually some users will install and use this package ...

The bypassed unit test failure looks like the follows, and it is a problem of Ubuntu itself:

[----------] 5 tests from ImageDataLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] ImageDataLayerTest/0.TestResize
*** Aborted at 1485019567 (unix time) try "date -d @1485019567" if you are using GNU date ***
PC: @     0x7facc82c00ba (unknown)
*** SIGSEGV (@0xfffffffffffffff7) received by PID 20719 (TID 0x7facb38bf700) from PID 18446744073709551607; stack trace: ***
    @     0x7facd0d42620 (unknown)
    @     0x7facc82c00ba (unknown)
    @     0x7facc82c018b (unknown)
    @     0x7facc82c1ce8 (unknown)
    @     0x7facc82c0692 (unknown)
    @     0x7facc82bb020 (unknown)
    @     0x7facc82b9165 tbb::internal::allocate_root_with_context_proxy::allocate()
    @     0x7faccf7fae22 cv::parallel_for_()
    @     0x7faccfbbeb2a (unknown)
    @     0x7faccfbbbbb7 cv::resize()
    @     0x7facd18240f7 caffe::ReadImageToCVMat()
    @     0x7facd173b82b caffe::ImageDataLayer<>::load_batch()
    @     0x7facd16c9d46 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
    @     0x7facd16a58eb boost::detail::thread_data<>::run()
    @     0x7facd1179576 (unknown)
    @     0x7facd0d386ca start_thread
    @     0x7facceb100ff clone
    @                0x0 (unknown)
Segmentation fault (core dumped)

Full build log (unit test not canceled) https://launchpadlibrarian.net/303350189/buildlog_ubuntu-zesty-amd64.caffe_1.0.0~rc4-1_BUILDING.txt.gz

Ubuntu (>= 16.10) users are suffering from the same problem, see e.g. #5077

@shelhamer @lukeyeager

lukeyeager · 2017-04-14T17:26:45Z

I have thoughts (probably an OpenCV issue rather than "a problem of Ubuntu itself"), but neither this bug nor #5077 seems appropriate for the discussion. Will you make a new issue for this test error?

cdluminate · 2017-04-15T01:31:17Z

@lukeyeager #5540

cdluminate · 2017-04-17T03:17:59Z

Uploaded caffe-cpu 1.0 to experimental. CUDA version will be updated soon.

cdluminate · 2017-04-17T03:18:46Z

I sensed an odor of new release since you guys are ridiculously active these days. hhh

longjon changed the title ~~Intent to package caffe for Debian~~ Provide a Caffe package in Debian Jun 19, 2015

shelhamer added the compatibility label Jun 19, 2015

lukeyeager mentioned this issue Sep 29, 2015

Install libs as non-executable files #3127

Merged

bhack mentioned this issue Nov 18, 2015

Request - switch to CMake as the only build system #3351

Open

cdluminate mentioned this issue Dec 5, 2016

Install on Ubuntu 14.04 uses the default OpenBlas library -> Low Performance torch/distro#121

Open

cdluminate mentioned this issue Dec 21, 2016

docs: add debian installation guide #5111

Merged

cdluminate closed this as completed Jan 5, 2017

shelhamer mentioned this issue Apr 13, 2017

Why is caffes Python binding not a package (hosted on PyPI)? #2440

Open

cdluminate mentioned this issue Apr 15, 2017

Common test failure on Ubuntu (> 16.04), segfault in an OpenCV call according to backtrace #5540

Closed

Provide a Caffe package in Debian #2601

Provide a Caffe package in Debian #2601

Comments

cdluminate commented Jun 14, 2015 • edited Loading

Status

Draft guide

Brief Guide for Debian/unstable users

Compiling your custom caffe package on Debian/unstable

FAQ

bhack commented Jun 14, 2015

cdluminate commented Jun 14, 2015

bhack commented Jun 14, 2015

flx42 commented Jun 14, 2015

cdluminate commented Jun 16, 2015

cdluminate commented Jun 16, 2015

talda commented Jun 16, 2015

cdluminate commented Jun 16, 2015

bhack commented Jun 16, 2015

cdluminate commented Jun 16, 2015

cdluminate commented Jun 16, 2015

cdluminate commented Jul 9, 2015

cdluminate commented Jul 9, 2015

cdluminate commented Aug 7, 2015

cdluminate commented Aug 7, 2015

shelhamer commented Aug 7, 2015

cdluminate commented Sep 2, 2015

bhack commented Sep 2, 2015

cdluminate commented Sep 2, 2015

cdluminate commented Sep 2, 2015

shelhamer commented Sep 2, 2015

cdluminate commented Sep 3, 2015

cdluminate commented Nov 10, 2015

micahcc commented Nov 28, 2015

cdluminate commented Nov 28, 2015

micahcc commented Nov 28, 2015

lukeyeager commented Jul 5, 2016

cdluminate commented Jul 6, 2016 • edited Loading

cdluminate commented Jul 6, 2016

cdluminate commented Jul 28, 2016

shelhamer commented Jul 30, 2016

cdluminate commented Oct 3, 2016

cdluminate commented Nov 17, 2016

shelhamer commented Nov 21, 2016

cdluminate commented Nov 22, 2016

cdluminate commented Nov 25, 2016

cdluminate commented Nov 28, 2016 • edited Loading

cdluminate commented Dec 3, 2016 • edited Loading

cdluminate commented Dec 3, 2016 • edited Loading

cdluminate commented Dec 20, 2016 • edited Loading

cdluminate commented Dec 20, 2016

naibaf7 commented Dec 20, 2016

shelhamer commented Dec 21, 2016

cdluminate commented Dec 21, 2016 • edited Loading

cdluminate commented Dec 21, 2016

cdluminate commented Jan 5, 2017

cdluminate commented Apr 14, 2017

lukeyeager commented Apr 14, 2017

cdluminate commented Apr 15, 2017

cdluminate commented Apr 17, 2017

cdluminate commented Apr 17, 2017

cdluminate commented Jun 14, 2015 •

edited

Loading

cdluminate commented Jul 6, 2016 •

edited

Loading

cdluminate commented Nov 28, 2016 •

edited

Loading

cdluminate commented Dec 3, 2016 •

edited

Loading

cdluminate commented Dec 3, 2016 •

edited

Loading

cdluminate commented Dec 20, 2016 •

edited

Loading

cdluminate commented Dec 21, 2016 •

edited

Loading