Plugin Docs: Python Tools & Matplotlib #2726

codingS3b · 2018-10-15T10:54:03Z

Adds sections on how to develop and use the new python tools.

~~It already includes documentation for the jupyter_widgets as well (see #2691)~~

update with changes from Shorter python imports for postprocessing tools #2727
update with changes from Visualizers: axes passed on startup #2728
update with changes from Python: base class for data readers #2730
move all widget additions to Python visualizers as jupyter widgets #2691

ax3l

Thank you very much!
For the PR, I suggest little open improvements and have a few follow-up PR tasks we should tackle next.

ax3l · 2018-10-15T11:13:07Z

docs/source/dev/py_postprocessing.rst

+Python postprocessing tool structure
+====================================
+
+For each plugin there are (or have to be implemented) three different classes.


Each plugin should implement at least one of the following Python classes.

docs/source/dev/py_postprocessing.rst

+2. A visualizer class that outputs a matplotlib plot
+3. A jupyter-widget class that exposes the parameters of the matplotlib visualizer to the user
+   via other widgets
+


ax3l · 2018-10-15T11:15:08Z

docs/source/dev/py_postprocessing.rst

+
+Data reader
+~~~~~~~~~~~
+The data readers should reside in the ``lib/python/picongpu/plugins/`` directory.


This is a follow-up to this PR:
they should actually get their own sub-directory soon for clean-up :)
E.g., lib/python/picongpu/plugins/data

ax3l · 2018-10-15T11:17:41Z

docs/source/dev/py_postprocessing.rst

+   def __init__(self, run_directory)
+
+   def get_data_path(self, your_specific_arguments)
+       # has to return the path to the underlying data file


Can we write the usual

""" Comments as now... Returns ------- str ... """ ...

format?

ax3l · 2018-10-15T11:19:03Z

docs/source/dev/py_postprocessing.rst

+The visualizers should reside in the ``lib/python/picongpu/plugins/plot_mpl/`` directory.
+The module names should end on ``_visualizer.py`` and the class name should only be ``Visualizer``.
+
+There is a base class for visualization found in ``base_visualizer.py`` which already handles the plotting logic. It uses the


As in latex, just start a newline when starting a new sentence. No limit on line length in prose text to break for.

This will make diffs more readable.

ax3l · 2018-10-15T11:38:25Z

docs/source/usage/plugins.rst

+Python postprocessing
+=====================
+
+In order to further work with the data produced by a plugin during a simulation run, PIConGPU provides


please keep one sentence per line for smaller diffs.

ax3l · 2018-10-15T11:39:00Z

docs/source/usage/plugins.rst

+
+
+They can be found under ``lib/python/picongpu/``.
+It is our goal to provide three modules for each plugin to make postprocessing as convenient as possible:


provide at least

the thing is: every plugin can have multiple data readers and multiple data visualization methods. For example, the energy histogram can analyze

a histogram

an integrated charge per opening angle in a detector for a range in the histogram

cut-off energies

...

ax3l · 2018-10-15T11:42:51Z

docs/source/usage/plugins/png.rst

@@ -6,7 +6,7 @@ PNG
 This plugin generates **images in the png format** for slices through the simulated volume.
 It allows to draw a **species density** together with electric, magnetic and/or current field values.
 The exact field values, their coloring and their normalization can be set using ``*.param`` files.
-It is a very rudimentary and useful tool to get a first impression on what happens in the simulation and to verify that the parameter set chosen leads to  the desired physics. 
+It is a very rudimentary and useful tool to get a first impression on what happens in the simulation and to verify that the parameter set chosen leads to  the desired physics.


Just spotted a double whitespace between "to" and "the desired physics".

ax3l · 2018-10-15T11:43:13Z

docs/source/usage/plugins/png.rst


 .. note::

   For a 2D simulation, even a 2D image can be a quite heavy output.
   Make sure to reduce the preview size!

-It  is possible to draw the borders between the GPUs used as white lines. 
+It  is possible to draw the borders between the GPUs used as white lines.


Just spotted a double whitespace between "It" and "is possible to ...".

ax3l · 2018-10-15T11:45:22Z

docs/source/dev/py_postprocessing.rst

+
+Each reader class needs to implement the following interface functions:
+
+.. code:: python


Did we forget to write a base class for this? If so, we should build one in a follow-up PR.

Another follow-up to this PR:
On Friday, we realized we forgot to add a meta-data method to return which units (and unit dimensions) the read data have. We should add a get_meta or similar method. We should make this similar or identical to unitSI and unitDimension in openPMD and/or can rely on pint-strings.

Yes, so far we don't have a base class for the readers. Would you want it to contain all the interface functions (and raising NotImplementedError or should this class already implement some logic?
I think only the phase_space.py file contains some metadata class. Is it something like this you had in mind?

Implemented in #2730

added now with last commit to avoid updating the interface in the manual and the source (now source only :) )

codingS3b · 2018-10-16T08:59:57Z

I adressed most of your comments locally but after rebasing on the current dev, I am not able to generate my docs anymore, the make html fails with Exception occurred: File "/usr/local/lib/python2.7/dist-packages/sphinx/domains/cpp.py", line 4060, in _add_symbols assert len(withDecl) <= 1 AssertionError .
Did I do something wrong?

ax3l · 2018-10-16T10:02:14Z

That's a Sphinx 1.8 bug. You need to downgrade to sphinx 1.7: #2725
It's already fixed in Sphinx master and maybe an upcoming sphinx 1.8.2 or 1.9 release will ship the fix.

codingS3b · 2018-10-16T13:55:50Z

Actually I'm using Sphinx 1.7.0. Anyway, it is working now...don't know what the issue was.

codingS3b · 2018-10-16T14:17:42Z

What do I have to change so that the py_postprocessing.rst also shows up in the navigation on the left of the page in the development section?

Allow Numpy-style Python autodocs for classes, modules and members with Napoleon.

ax3l · 2018-10-17T00:04:19Z

Don't be confused: I moved all widget docs to a follow-up in #2691.
The rationale behind this is that the doc PR here shall still go into 0.4.0, but the widgets go into 0.5.0/1.0.0 since they are still in an early draft.

Add section on postproc tools (PNG, Energy Histogram, PS). Devel: Architecture of postprocessing tools.

ax3l · 2018-10-17T08:56:06Z

@codingS3b are you ok with my modifications and can I merge this? :)

codingS3b · 2018-10-17T09:00:31Z

Yes, thanks for restructuring!

ax3l self-requested a review October 15, 2018 11:11

ax3l self-assigned this Oct 15, 2018

ax3l added component: tools scripts, python libs and CMake documentation regarding documentation or wiki discussions labels Oct 15, 2018

ax3l reviewed Oct 15, 2018

View reviewed changes

ax3l added this to the 0.5.0 / 1.0.0: Next Stable milestone Oct 15, 2018

codingS3b force-pushed the documentation_python_vis branch from 91918eb to 3be055c Compare October 16, 2018 14:12

ax3l mentioned this pull request Oct 16, 2018

Release 0.4.0-rc4 #2717

Merged

21 tasks

Sphinx: Napoleon Autodocs

ff5e837

Allow Numpy-style Python autodocs for classes, modules and members with Napoleon.

ax3l force-pushed the documentation_python_vis branch from f830e53 to 3c159ba Compare October 17, 2018 00:00

codingS3b mentioned this pull request Oct 17, 2018

Python visualizers as jupyter widgets #2691

Merged

2 tasks

ax3l modified the milestones: 0.5.0 / 1.0.0: Next Stable, 0.4.0: CPU Support, Filter & Merging Oct 17, 2018

ax3l force-pushed the documentation_python_vis branch from 3c159ba to c1e2cd2 Compare October 17, 2018 00:12

ax3l changed the title ~~Documentation of python visualizers~~ Plugin Docs: Python Tools & Matplotlib Oct 17, 2018

ax3l added the component: plugin in PIConGPU plugin label Oct 17, 2018

ax3l force-pushed the documentation_python_vis branch from c1e2cd2 to 16d9a6a Compare October 17, 2018 00:20

ax3l previously approved these changes Oct 17, 2018

View reviewed changes

Plugin Docs: Python Tools & Matplotlib

3832df5

Add section on postproc tools (PNG, Energy Histogram, PS). Devel: Architecture of postprocessing tools.

ax3l dismissed their stale review via 3832df5 October 17, 2018 00:25

ax3l force-pushed the documentation_python_vis branch from 16d9a6a to 3832df5 Compare October 17, 2018 00:25

ax3l approved these changes Oct 17, 2018

View reviewed changes

ax3l merged commit 400e8d3 into ComputationalRadiationPhysics:dev Oct 17, 2018

ax3l mentioned this pull request Oct 17, 2018

Docs: openPMD Update #2733

Merged

ax3l mentioned this pull request Oct 17, 2018

Python Docs: Fix Paths #2738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin Docs: Python Tools & Matplotlib #2726

Plugin Docs: Python Tools & Matplotlib #2726

codingS3b commented Oct 15, 2018 •

edited by ax3l

Loading

ax3l left a comment

ax3l Oct 15, 2018

This comment was marked as resolved.

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

ax3l Oct 15, 2018

codingS3b Oct 16, 2018 •

edited

Loading

ax3l Oct 16, 2018

ax3l Oct 16, 2018

codingS3b commented Oct 16, 2018 •

edited

Loading

ax3l commented Oct 16, 2018

codingS3b commented Oct 16, 2018 •

edited

Loading

codingS3b commented Oct 16, 2018 •

edited

Loading

ax3l commented Oct 17, 2018

ax3l commented Oct 17, 2018

codingS3b commented Oct 17, 2018



		They can be found under ``lib/python/picongpu/``.
		It is our goal to provide three modules for each plugin to make postprocessing as convenient as possible:


		Each reader class needs to implement the following interface functions:

		.. code:: python

Plugin Docs: Python Tools & Matplotlib #2726

Plugin Docs: Python Tools & Matplotlib #2726

Conversation

codingS3b commented Oct 15, 2018 • edited by ax3l Loading

ax3l left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codingS3b Oct 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codingS3b commented Oct 16, 2018 • edited Loading

ax3l commented Oct 16, 2018

codingS3b commented Oct 16, 2018 • edited Loading

codingS3b commented Oct 16, 2018 • edited Loading

ax3l commented Oct 17, 2018

ax3l commented Oct 17, 2018

codingS3b commented Oct 17, 2018

codingS3b commented Oct 15, 2018 •

edited by ax3l

Loading

codingS3b Oct 16, 2018 •

edited

Loading

codingS3b commented Oct 16, 2018 •

edited

Loading

codingS3b commented Oct 16, 2018 •

edited

Loading

codingS3b commented Oct 16, 2018 •

edited

Loading