Skip to content
This repository has been archived by the owner on Jan 30, 2024. It is now read-only.

Commit

Permalink
Adding documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@brinick
brinick committed Apr 29, 2021
1 parent 873814c commit f943bc0
Show file tree
Hide file tree
Showing 37 changed files with 653 additions and 253 deletions.
24 changes: 18 additions & 6 deletions docs/html/_modules/pilot/api/data.html
View file

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion docs/html/_modules/pilot/common/errorcodes.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,7 @@ <h1>Source code for pilot.common.errorcodes</h1><div class="highlight"><pre>
<span class="n">REMOTEFILECOULDNOTBEOPENED</span> <span class="o">=</span> <span class="mi">1361</span>
<span class="n">XRDCPERROR</span> <span class="o">=</span> <span class="mi">1362</span>
<span class="n">KILLPAYLOAD</span> <span class="o">=</span> <span class="mi">1363</span> <span class="c1"># note, not a failure but a kill instruction from Raythena</span>
<span class="n">MISSINGCREDENTIALS</span> <span class="o">=</span> <span class="mi">1364</span>

<span class="n">_error_messages</span> <span class="o">=</span> <span class="p">{</span>
<span class="n">GENERALERROR</span><span class="p">:</span> <span class="s2">&quot;General pilot error, consult batch log&quot;</span><span class="p">,</span>
Expand Down Expand Up @@ -303,7 +304,8 @@ <h1>Source code for pilot.common.errorcodes</h1><div class="highlight"><pre>
<span class="n">IMAGENOTFOUND</span><span class="p">:</span> <span class="s2">&quot;Image not found&quot;</span><span class="p">,</span>
<span class="n">REMOTEFILECOULDNOTBEOPENED</span><span class="p">:</span> <span class="s2">&quot;Remote file could not be opened&quot;</span><span class="p">,</span>
<span class="n">XRDCPERROR</span><span class="p">:</span> <span class="s2">&quot;Xrdcp was unable to open file&quot;</span><span class="p">,</span>
<span class="n">KILLPAYLOAD</span><span class="p">:</span> <span class="s2">&quot;Raythena has decided to kill payload&quot;</span>
<span class="n">KILLPAYLOAD</span><span class="p">:</span> <span class="s2">&quot;Raythena has decided to kill payload&quot;</span><span class="p">,</span>
<span class="n">MISSINGCREDENTIALS</span><span class="p">:</span> <span class="s2">&quot;Unable to locate credentials for S3 transfer&quot;</span>
<span class="p">}</span>

<span class="n">put_error_codes</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1135</span><span class="p">,</span> <span class="mi">1136</span><span class="p">,</span> <span class="mi">1137</span><span class="p">,</span> <span class="mi">1141</span><span class="p">,</span> <span class="mi">1152</span><span class="p">,</span> <span class="mi">1181</span><span class="p">]</span>
Expand Down
139 changes: 85 additions & 54 deletions docs/html/_modules/pilot/control/data.html
View file

Large diffs are not rendered by default.

139 changes: 100 additions & 39 deletions docs/html/_modules/pilot/control/job.html
View file

Large diffs are not rendered by default.

5 changes: 4 additions & 1 deletion docs/html/_modules/pilot/control/payload.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -228,10 +228,13 @@ <h1>Source code for pilot.control.payload</h1><div class="highlight"><pre>
<span class="k">try</span><span class="p">:</span>
<span class="n">job</span> <span class="o">=</span> <span class="n">queues</span><span class="o">.</span><span class="n">validated_payloads</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">block</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>

<span class="c1">#q_snapshot = list(queues.finished_data_in.queue) if queues.finished_data_in else []</span>
<span class="c1">#peek = [s_job for s_job in q_snapshot if job.jobid == s_job.jobid]</span>
<span class="c1">#if job.jobid not in q_snapshot:</span>

<span class="n">q_snapshot</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">queues</span><span class="o">.</span><span class="n">finished_data_in</span><span class="o">.</span><span class="n">queue</span><span class="p">)</span>
<span class="n">peek</span> <span class="o">=</span> <span class="p">[</span><span class="n">s_job</span> <span class="k">for</span> <span class="n">s_job</span> <span class="ow">in</span> <span class="n">q_snapshot</span> <span class="k">if</span> <span class="n">job</span><span class="o">.</span><span class="n">jobid</span> <span class="o">==</span> <span class="n">s_job</span><span class="o">.</span><span class="n">jobid</span><span class="p">]</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">peek</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
<span class="c1">#queues.validated_payloads.put(job)</span>
<span class="n">put_in_queue</span><span class="p">(</span><span class="n">job</span><span class="p">,</span> <span class="n">queues</span><span class="o">.</span><span class="n">validated_payloads</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span> <span class="c1"># Python 3</span>
<span class="k">if</span> <span class="n">args</span><span class="o">.</span><span class="n">graceful_stop</span><span class="o">.</span><span class="n">is_set</span><span class="p">():</span>
Expand Down
18 changes: 7 additions & 11 deletions docs/html/_modules/pilot/control/payloads/eventservice.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ <h1>Source code for pilot.control.payloads.eventservice</h1><div class="highligh
<span class="c1">#</span>
<span class="c1"># Authors:</span>
<span class="c1"># - Wen Guan, wen.guan@cern.ch, 2017-2018</span>
<span class="c1"># - Paul Nilsson, paul.nilsson@cern.ch, 2020</span>
<span class="c1"># - Paul Nilsson, paul.nilsson@cern.ch, 2021</span>


<span class="kn">import</span> <span class="nn">os</span>
Expand All @@ -48,7 +48,6 @@ <h1>Source code for pilot.control.payloads.eventservice</h1><div class="highligh
<span class="kn">from</span> <span class="nn">pilot.common</span> <span class="kn">import</span> <span class="n">exception</span>
<span class="kn">from</span> <span class="nn">pilot.control.payloads</span> <span class="kn">import</span> <span class="n">generic</span>
<span class="kn">from</span> <span class="nn">pilot.eventservice.workexecutor.workexecutor</span> <span class="kn">import</span> <span class="n">WorkExecutor</span>
<span class="kn">from</span> <span class="nn">pilot.util.config</span> <span class="kn">import</span> <span class="n">config</span>

<span class="kn">import</span> <span class="nn">logging</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="vm">__name__</span><span class="p">)</span>
Expand Down Expand Up @@ -94,7 +93,7 @@ <h1>Source code for pilot.control.payloads.eventservice</h1><div class="highligh
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">&quot;payload: </span><span class="si">%s</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="n">payload</span><span class="p">)</span>

<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">&quot;Starting EventService WorkExecutor&quot;</span><span class="p">)</span>
<span class="n">executor_type</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_executor_type</span><span class="p">(</span><span class="n">job</span><span class="p">)</span>
<span class="n">executor_type</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_executor_type</span><span class="p">()</span>
<span class="n">executor</span> <span class="o">=</span> <span class="n">WorkExecutor</span><span class="p">(</span><span class="n">args</span><span class="o">=</span><span class="n">executor_type</span><span class="p">)</span>
<span class="n">executor</span><span class="o">.</span><span class="n">set_payload</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
<span class="n">executor</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
Expand All @@ -112,28 +111,25 @@ <h1>Source code for pilot.control.payloads.eventservice</h1><div class="highligh

<span class="k">return</span> <span class="n">executor</span></div>

<div class="viewcode-block" id="Executor.get_executor_type"><a class="viewcode-back" href="../../../../components/control/payloads/eventservice.html#pilot.control.payloads.eventservice.Executor.get_executor_type">[docs]</a> <span class="k">def</span> <span class="nf">get_executor_type</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">job</span><span class="p">):</span>
<div class="viewcode-block" id="Executor.get_executor_type"><a class="viewcode-back" href="../../../../components/control/payloads/eventservice.html#pilot.control.payloads.eventservice.Executor.get_executor_type">[docs]</a> <span class="k">def</span> <span class="nf">get_executor_type</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Get the executor type.</span>
<span class="sd"> This is usually the &#39;generic&#39; type, which means normal event service. It can also be &#39;raythena&#39; if specified</span>
<span class="sd"> in the pilot config file, and can also be dynamically decided using the job object (in the case of interceptor</span>
<span class="sd"> job).</span>
<span class="sd"> in the Pilot options.</span>

<span class="sd"> :param job: job object.</span>
<span class="sd"> :return: executor type dictionary.</span>
<span class="sd"> &quot;&quot;&quot;</span>

<span class="c1"># executor_type = &#39;hpo&#39; if job.is_hpo else config.Payload.executor_type</span>
<span class="c1"># executor_type = &#39;hpo&#39; if job.is_hpo else os.environ.get(&#39;PILOT_ES_EXECUTOR_TYPE&#39;, &#39;generic&#39;)</span>
<span class="c1"># return {&#39;executor_type&#39;: executor_type}</span>
<span class="k">return</span> <span class="p">{</span><span class="s1">&#39;executor_type&#39;</span><span class="p">:</span> <span class="n">config</span><span class="o">.</span><span class="n">Payload</span><span class="o">.</span><span class="n">executor_type</span><span class="p">}</span></div>
<span class="k">return</span> <span class="p">{</span><span class="s1">&#39;executor_type&#39;</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;PILOT_ES_EXECUTOR_TYPE&#39;</span><span class="p">,</span> <span class="s1">&#39;generic&#39;</span><span class="p">)}</span></div>

<div class="viewcode-block" id="Executor.wait_graceful"><a class="viewcode-back" href="../../../../components/control/payloads/eventservice.html#pilot.control.payloads.eventservice.Executor.wait_graceful">[docs]</a> <span class="k">def</span> <span class="nf">wait_graceful</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">proc</span><span class="p">,</span> <span class="n">job</span><span class="p">):</span>
<div class="viewcode-block" id="Executor.wait_graceful"><a class="viewcode-back" href="../../../../components/control/payloads/eventservice.html#pilot.control.payloads.eventservice.Executor.wait_graceful">[docs]</a> <span class="k">def</span> <span class="nf">wait_graceful</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">proc</span><span class="p">):</span>
<span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> (add description)</span>

<span class="sd"> :param args:</span>
<span class="sd"> :param proc:</span>
<span class="sd"> :param job:</span>
<span class="sd"> :return:</span>
<span class="sd"> &quot;&quot;&quot;</span>

Expand Down
Loading

0 comments on commit f943bc0

Please sign in to comment.