-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
551 lines (506 loc) · 18.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<!-- 2021-07-19 Mon 11:46 -->
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>LS4GAN Containers</title>
<meta name="generator" content="Org mode" />
<meta name="author" content="BV" />
<style type="text/css">
<!--/*--><![CDATA[/*><!--*/
.title { text-align: center;
margin-bottom: .2em; }
.subtitle { text-align: center;
font-size: medium;
font-weight: bold;
margin-top:0; }
.todo { font-family: monospace; color: red; }
.done { font-family: monospace; color: green; }
.priority { font-family: monospace; color: orange; }
.tag { background-color: #eee; font-family: monospace;
padding: 2px; font-size: 80%; font-weight: normal; }
.timestamp { color: #bebebe; }
.timestamp-kwd { color: #5f9ea0; }
.org-right { margin-left: auto; margin-right: 0px; text-align: right; }
.org-left { margin-left: 0px; margin-right: auto; text-align: left; }
.org-center { margin-left: auto; margin-right: auto; text-align: center; }
.underline { text-decoration: underline; }
#postamble p, #preamble p { font-size: 90%; margin: .2em; }
p.verse { margin-left: 3%; }
pre {
border: 1px solid #ccc;
box-shadow: 3px 3px 3px #eee;
padding: 8pt;
font-family: monospace;
overflow: auto;
margin: 1.2em;
}
pre.src {
position: relative;
overflow: auto;
padding-top: 1.2em;
}
pre.src:before {
display: none;
position: absolute;
background-color: white;
top: -10px;
right: 10px;
padding: 3px;
border: 1px solid black;
}
pre.src:hover:before { display: inline; margin-top: 14px;}
/* Languages per Org manual */
pre.src-asymptote:before { content: 'Asymptote'; }
pre.src-awk:before { content: 'Awk'; }
pre.src-C:before { content: 'C'; }
/* pre.src-C++ doesn't work in CSS */
pre.src-clojure:before { content: 'Clojure'; }
pre.src-css:before { content: 'CSS'; }
pre.src-D:before { content: 'D'; }
pre.src-ditaa:before { content: 'ditaa'; }
pre.src-dot:before { content: 'Graphviz'; }
pre.src-calc:before { content: 'Emacs Calc'; }
pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
pre.src-fortran:before { content: 'Fortran'; }
pre.src-gnuplot:before { content: 'gnuplot'; }
pre.src-haskell:before { content: 'Haskell'; }
pre.src-hledger:before { content: 'hledger'; }
pre.src-java:before { content: 'Java'; }
pre.src-js:before { content: 'Javascript'; }
pre.src-latex:before { content: 'LaTeX'; }
pre.src-ledger:before { content: 'Ledger'; }
pre.src-lisp:before { content: 'Lisp'; }
pre.src-lilypond:before { content: 'Lilypond'; }
pre.src-lua:before { content: 'Lua'; }
pre.src-matlab:before { content: 'MATLAB'; }
pre.src-mscgen:before { content: 'Mscgen'; }
pre.src-ocaml:before { content: 'Objective Caml'; }
pre.src-octave:before { content: 'Octave'; }
pre.src-org:before { content: 'Org mode'; }
pre.src-oz:before { content: 'OZ'; }
pre.src-plantuml:before { content: 'Plantuml'; }
pre.src-processing:before { content: 'Processing.js'; }
pre.src-python:before { content: 'Python'; }
pre.src-R:before { content: 'R'; }
pre.src-ruby:before { content: 'Ruby'; }
pre.src-sass:before { content: 'Sass'; }
pre.src-scheme:before { content: 'Scheme'; }
pre.src-screen:before { content: 'Gnu Screen'; }
pre.src-sed:before { content: 'Sed'; }
pre.src-sh:before { content: 'shell'; }
pre.src-sql:before { content: 'SQL'; }
pre.src-sqlite:before { content: 'SQLite'; }
/* additional languages in org.el's org-babel-load-languages alist */
pre.src-forth:before { content: 'Forth'; }
pre.src-io:before { content: 'IO'; }
pre.src-J:before { content: 'J'; }
pre.src-makefile:before { content: 'Makefile'; }
pre.src-maxima:before { content: 'Maxima'; }
pre.src-perl:before { content: 'Perl'; }
pre.src-picolisp:before { content: 'Pico Lisp'; }
pre.src-scala:before { content: 'Scala'; }
pre.src-shell:before { content: 'Shell Script'; }
pre.src-ebnf2ps:before { content: 'ebfn2ps'; }
/* additional language identifiers per "defun org-babel-execute"
in ob-*.el */
pre.src-cpp:before { content: 'C++'; }
pre.src-abc:before { content: 'ABC'; }
pre.src-coq:before { content: 'Coq'; }
pre.src-groovy:before { content: 'Groovy'; }
/* additional language identifiers from org-babel-shell-names in
ob-shell.el: ob-shell is the only babel language using a lambda to put
the execution function name together. */
pre.src-bash:before { content: 'bash'; }
pre.src-csh:before { content: 'csh'; }
pre.src-ash:before { content: 'ash'; }
pre.src-dash:before { content: 'dash'; }
pre.src-ksh:before { content: 'ksh'; }
pre.src-mksh:before { content: 'mksh'; }
pre.src-posh:before { content: 'posh'; }
/* Additional Emacs modes also supported by the LaTeX listings package */
pre.src-ada:before { content: 'Ada'; }
pre.src-asm:before { content: 'Assembler'; }
pre.src-caml:before { content: 'Caml'; }
pre.src-delphi:before { content: 'Delphi'; }
pre.src-html:before { content: 'HTML'; }
pre.src-idl:before { content: 'IDL'; }
pre.src-mercury:before { content: 'Mercury'; }
pre.src-metapost:before { content: 'MetaPost'; }
pre.src-modula-2:before { content: 'Modula-2'; }
pre.src-pascal:before { content: 'Pascal'; }
pre.src-ps:before { content: 'PostScript'; }
pre.src-prolog:before { content: 'Prolog'; }
pre.src-simula:before { content: 'Simula'; }
pre.src-tcl:before { content: 'tcl'; }
pre.src-tex:before { content: 'TeX'; }
pre.src-plain-tex:before { content: 'Plain TeX'; }
pre.src-verilog:before { content: 'Verilog'; }
pre.src-vhdl:before { content: 'VHDL'; }
pre.src-xml:before { content: 'XML'; }
pre.src-nxml:before { content: 'XML'; }
/* add a generic configuration mode; LaTeX export needs an additional
(add-to-list 'org-latex-listings-langs '(conf " ")) in .emacs */
pre.src-conf:before { content: 'Configuration File'; }
table { border-collapse:collapse; }
caption.t-above { caption-side: top; }
caption.t-bottom { caption-side: bottom; }
td, th { vertical-align:top; }
th.org-right { text-align: center; }
th.org-left { text-align: center; }
th.org-center { text-align: center; }
td.org-right { text-align: right; }
td.org-left { text-align: left; }
td.org-center { text-align: center; }
dt { font-weight: bold; }
.footpara { display: inline; }
.footdef { margin-bottom: 1em; }
.figure { padding: 1em; }
.figure p { text-align: center; }
.equation-container {
display: table;
text-align: center;
width: 100%;
}
.equation {
vertical-align: middle;
}
.equation-label {
display: table-cell;
text-align: right;
vertical-align: middle;
}
.inlinetask {
padding: 10px;
border: 2px solid gray;
margin: 10px;
background: #ffffcc;
}
#org-div-home-and-up
{ text-align: right; font-size: 70%; white-space: nowrap; }
textarea { overflow-x: auto; }
.linenr { font-size: smaller }
.code-highlighted { background-color: #ffff00; }
.org-info-js_info-navigation { border-style: none; }
#org-info-js_console-label
{ font-size: 10px; font-weight: bold; white-space: nowrap; }
.org-info-js_search-highlight
{ background-color: #ffff00; color: #000000; font-weight: bold; }
.org-svg { width: 90%; }
/*]]>*/-->
</style>
<link rel="stylesheet" type="text/css" href="https://ls4gan.github.io/other/darksun/css/htmlize.css"/>
<link rel="stylesheet" type="text/css" href="https://ls4gan.github.io/other/darksun/css/darksun.css"/>
<script type="text/javascript">
// @license magnet:?xt=urn:btih:e95b018ef3580986a04669f1b5879592219e2a7a&dn=public-domain.txt Public Domain
<!--/*--><![CDATA[/*><!--*/
function CodeHighlightOn(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.classList.add("code-highlighted");
target.classList.add("code-highlighted");
}
}
function CodeHighlightOff(elem, id)
{
var target = document.getElementById(id);
if(null != target) {
elem.classList.remove("code-highlighted");
target.classList.remove("code-highlighted");
}
}
/*]]>*///-->
// @license-end
</script>
</head>
<body>
<div id="content">
<h1 class="title">LS4GAN Containers</h1>
<div id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#info">1. General information</a>
<ul>
<li><a href="#build">1.1. Build</a></li>
<li><a href="#run">1.2. Run a container</a></li>
<li><a href="#sing">1.3. Singularity</a></li>
<li><a href="#dockerhub">1.4. Docker Hub</a></li>
</ul>
</li>
<li><a href="#avilable">2. Container Images</a>
<ul>
<li><a href="#wirecell-image">2.1. wirecell</a></li>
<li><a href="#runner-image">2.2. <span class="todo TODO">TODO</span> runner</a></li>
<li><a href="#notebook-image">2.3. <span class="todo TODO">TODO</span> notebook</a></li>
</ul>
</li>
<li><a href="#portus">3. BNL/SDCC Registry</a>
<ul>
<li><a href="#remote">3.1. Remote access</a></li>
<li><a href="#port-sing-prob">3.2. Singularity</a></li>
</ul>
</li>
</ul>
</div>
</div>
<p>
This package collects recipes to build container images that may be
used for LS4GAN related development and processing.
</p>
<div id="outline-container-info" class="outline-2">
<h2 id="info"><span class="section-number-2">1</span> General information</h2>
<div class="outline-text-2" id="text-info">
<p>
Images are defined via a <code>Dockerfile</code> and some additional files in a
sub directory. From a Docker image a Singularity image may be derived
and images are pushed to Docker Hub and/or SDCC's registry.
</p>
<p>
In the remainder of this section, basic and generic guidance is
documented. In the next section, descriptions of available images are
listed. Finally, some guidance in dealing with SDCC's registry is
provided.
</p>
</div>
<div id="outline-container-build" class="outline-3">
<h3 id="build"><span class="section-number-3">1.1</span> Build</h3>
<div class="outline-text-3" id="text-build">
<p>
A container image may be built locally with:
</p>
<pre class="example" id="orgc99ad71">
$ mkdir ls4gan
$ git clone https://github.com/LS4GAN/containers.git ls4gan/containers
$ cd ls4gan/containers/docker/<name>
$ docker build -t ls4gan/<name> .
</pre>
<p>
Note, some images build on top of others that are also built from
here. The naming convention illustrated above should be kept in mind
when looking at a <code>FROM</code> line in a <code>Dockerfile</code>.
</p>
</div>
</div>
<div id="outline-container-run" class="outline-3">
<h3 id="run"><span class="section-number-3">1.2</span> Run a container</h3>
<div class="outline-text-3" id="text-run">
<p>
Each container has its own default command (<code>CMD</code>) of <code>bash</code> which is
run as the argument to the entry point (<code>ENTRYPOINT</code>) of <code>bash -c</code>.
Thus to get an interactive shell:
</p>
<pre class="example" id="orgd6d331b">
$ docker run -ti ls4gan/wirecell
#
</pre>
<p>
Or to run a command provided by the image:
</p>
<pre class="example" id="orgc5b019f">
$ docker run -ti ls4gan/wirecell "wire-cell --help"
[...help message...]
</pre>
</div>
</div>
<div id="outline-container-sing" class="outline-3">
<h3 id="sing"><span class="section-number-3">1.3</span> Singularity</h3>
<div class="outline-text-3" id="text-sing">
<p>
To derive a Singularity container image from a Docker image:
</p>
<pre class="example" id="orga9ab8db">
$ singularity build ls4gan-<name>-latest.sif $IMAGE_URL
</pre>
<p>
The <code>$IMAGE_URL</code> can be in one of several forms
</p>
<dl class="org-dl">
<dt>local</dt><dd><code>docker-daemon://ls4gan/<name>:latest</code></dd>
<dt>docker hub</dt><dd><code>docker://ls4gan/<name>:latest</code></dd>
<dt>SDCC registry</dt><dd></dd>
</dl>
<p>
It recommended to follow the illustrated naming convention so that
some provenance is kept when sharing the resulting image file.
</p>
<p>
To run a Singularity container
</p>
<pre class="example" id="orged05555">
$ singularity exec ls4gan-<name>-latest.sif "wire-cell --help"
</pre>
<p>
It is recommended to name the Singularity image file following the
convention so that when sharing these files their origin is hinted.
</p>
<p>
To run the default shell or a program in the container
</p>
<pre class="example" id="org0b367f8">
$ singularity run /srv/tmp/ls4gan-wirecell-latest.sif "wire-cell --help"
[ ...help message...]
$ singularity run /srv/tmp/ls4gan-wirecell-latest.sif
Singularity> which wire-cell
/usr/local/bin/wire-cell
</pre>
</div>
</div>
<div id="outline-container-dockerhub" class="outline-3">
<h3 id="dockerhub"><span class="section-number-3">1.4</span> Docker Hub</h3>
<div class="outline-text-3" id="text-dockerhub">
<p>
The <a href="https://hub.docker.com/u/ls4gan.">ls4gan area on Docker Hub</a> holds some of the images produced
here. In the examples we will use the <a href="https://hub.docker.com/repository/docker/ls4gan/wirecell">ls4gan/wirecell</a> image.
To get this image into your local docker, run:
</p>
<pre class="example" id="org7ffebcf">
$ docker pull ls4gan/wirecell:latest
</pre>
<p>
Or, if you are building images and a <code>docker login</code> they may be
uploaded with, eg:
</p>
<pre class="example" id="orga06cdb3">
$ docker push ls4gan/wirecell:latest
</pre>
</div>
</div>
</div>
<div id="outline-container-avilable" class="outline-2">
<h2 id="avilable"><span class="section-number-2">2</span> Container Images</h2>
<div class="outline-text-2" id="text-avilable">
</div>
<div id="outline-container-wirecell-image" class="outline-3">
<h3 id="wirecell-image"><span class="section-number-3">2.1</span> wirecell</h3>
<div class="outline-text-3" id="text-wirecell-image">
<p>
This image provides the Wire-Cell Toolkit C++ and Python and their
externals. It is built on a minimal Debian with WCT and additional
software installed under <code>/usr/local/</code>
</p>
<p>
This environment can be used to run any "stand-alone" <code>wire-cell</code> job
or any of the <code>wirecell-*</code> Python CLIs. It also provides lots of
Python goodies including Numpy, Matplotlib, ipython, JupyterLab
(needing special <code>docker run</code> to see its ports).
</p>
<p>
It also provides <code>snakemake</code> so can be used to exercise the
<a href="https://ls4gan.github.io/toyzero/">toyzero</a> data generator. Using
the derived Singularity image to enjoy easy access to native home
directory files:
</p>
<pre class="example" id="org82f99b4">
$ git clone https://github.com/LS4GAN/toyzero.git
$ cd toyzero
$ singularity run /srv/tmp/ls4gan-wirecell-latest.sif "snakemake -jall -p just_images"
$ tree data
[...generated data files...]
</pre>
</div>
</div>
<div id="outline-container-runner-image" class="outline-3">
<h3 id="runner-image"><span class="section-number-3">2.2</span> <span class="todo TODO">TODO</span> runner</h3>
<div class="outline-text-3" id="text-runner-image">
<p>
This container provides (will provide) support for running a toyzero
pipeline as a single, ready-to-run job. It rides on top of the
wirecell container.
</p>
</div>
</div>
<div id="outline-container-notebook-image" class="outline-3">
<h3 id="notebook-image"><span class="section-number-3">2.3</span> <span class="todo TODO">TODO</span> notebook</h3>
<div class="outline-text-3" id="text-notebook-image">
<p>
This container provides (will provide) support for JupyterLab
notebooks. It includes ls4gan-python, toytools and other Python
</p>
</div>
</div>
</div>
<div id="outline-container-portus" class="outline-2">
<h2 id="portus"><span class="section-number-2">3</span> BNL/SDCC Registry</h2>
<div class="outline-text-2" id="text-portus">
<p>
BNL/SDCC provides a container registry called "Portus" at the internal
nost <code>registry.sdcc.bnl.gov</code>. It is used much like Docker Hub but one
must give the hostname explicitly:
</p>
<pre class="example" id="orgc853e9f">
❯ docker pull registry.sdcc.bnl.gov/toyzero/wirecell
</pre>
</div>
<div id="outline-container-remote" class="outline-3">
<h3 id="remote"><span class="section-number-3">3.1</span> Remote access</h3>
<div class="outline-text-3" id="text-remote">
<p>
Outside the BNL network it is possible to access the registry by
forwarding a local port via SSH to an internal HTTPS proxy.
Basically, follow <a href="https://docs.docker.com/config/daemon/systemd/#httphttps-proxy">docker guidance on setting HTTP/HTTPS proxy</a>. For
example
</p>
<pre class="example" id="org0e1ba40">
# cat <<EOF > /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:3128"
Environment="HTTPS_PROXY=http://127.0.0.1:3128"
Environment="NO_PROXY=localhost,127.0.0.1,.home,haiku"
EOF
# systemctl daemon-reload
# systemctl restart docker
</pre>
<p>
It is also helpful to add the internal IP address to <code>/etc/hosts</code>.
</p>
<p>
It should now be possible to login
</p>
<pre class="example" id="orga303389">
❯ docker login registry.sdcc.bnl.gov
</pre>
<p>
And the two steps to register and upload an image:
</p>
<pre class="example" id="orgbce6c60">
❯ docker tag ls4gan/wirecell registry.sdcc.bnl.gov/toyzero/wirecell
❯ docker push registry.sdcc.bnl.gov/toyzero/wirecell
</pre>
<p>
Or, <code>docker pull</code> as above.
</p>
</div>
</div>
<div id="outline-container-port-sing-prob" class="outline-3">
<h3 id="port-sing-prob"><span class="section-number-3">3.2</span> Singularity</h3>
<div class="outline-text-3" id="text-port-sing-prob">
<p>
Currently building a Singularity image from a Docker image in Portus
does not work. Expect an error like:
</p>
<pre class="example" id="org7774c6e">
❯ singularity pull --docker-login docker://registry.sdcc.bnl.gov/ls4gan/toyzero/wirecell
Enter Docker Username: bvlbne
Enter Docker Password:
FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://registry.sdcc.bnl.gov/ls4gan/toyzero/wirecell: error pinging docker registry registry.sdcc.bnl.gov: Get "https://registry.sdcc.bnl.gov/v2/": dial tcp 130.199.148.226:443: i/o timeout
</pre>
<p>
This may be due to offsite access restriction. In any case, it needs
more checking.
</p>
</div>
</div>
</div>
</div>
<div id="postamble" class="status">
<p class="author">Author: BV</p>
<p class="date">Created: 2021-07-19 Mon 11:46</p>
<p class="validation"><a href="https://validator.w3.org/check?uri=referer">Validate</a></p>
</div>
</body>
</html>