Create Markdown Operator Docs

[ansMHanmer]

• Add script .ci/generate_operators_doc.py and template doc/source/operators_doc/operator_doc_template.md
• Script makes requests to DPF local server for available operators, and generates the same documentation found on https://dpf.docs.pyansys.com/version/stable/operator_reference.html in Markdown format
• Background in: https://github.com/ansys-internal/empowerment/issues/615

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.71%. Comparing base (68f0755) to head (2c03c97).
Report is 4 commits behind head on master.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2019      +/-   ##
==========================================
- Coverage   84.02%   83.71%   -0.32%     
==========================================
  Files          90       90              
  Lines       10303    10303              
==========================================
- Hits         8657     8625      -32     
- Misses       1646     1678      +32     

[PProfizi]

Hi @ansMHanmer, just a few suggestions.

Also, do you know if we could set some of the properties as Markdown metadata?
I think this is what we will use to allow for tagging and filtering of operators in the final documentation.
Not sure if we want to use MultiMarkdown Metadata or YAML Front Matter or something else.

[ansMHanmer]

Hi @ansMHanmer, just a few suggestions.

Also, do you know if we could set some of the properties as Markdown metadata? I think this is what we will use to allow for tagging and filtering of operators in the final documentation. Not sure if we want to use MultiMarkdown Metadata or YAML Front Matter or something else.

Thanks for the suggestions, working away at them. Yes, we can set some of the properties as Markdown metadata, and it's probably best to use YAML Front Matter based on some research, so I'll go with that. If we find we need to use MultiMarkdown I can easily switch.

@PProfizi - which properties would you want to see in the metadata? I was about to go implement, but figured it would be best to ask what the desired outcome was first 😄

[PProfizi]

Hi @ansMHanmer, just a few suggestions.
Also, do you know if we could set some of the properties as Markdown metadata? I think this is what we will use to allow for tagging and filtering of operators in the final documentation. Not sure if we want to use MultiMarkdown Metadata or YAML Front Matter or something else.

Thanks for the suggestions, working away at them. Yes, we can set some of the properties as Markdown metadata, and it's probably best to use YAML Front Matter based on some research, so I'll go with that. If we find we need to use MultiMarkdown I can easily switch.

@PProfizi - which properties would you want to see in the metadata? I was about to go implement, but figured it would be best to ask what the desired outcome was first 😄

So we are working on actually adding some properties we eventually will want to see as filterable/searchable metadata, but for a first PoC we can use plugin to demonstrate it is handled. Probably the license property of the operator should be metadata we can filter with. Also, the category is used in the current doc to organize operators, so I think it should be included in the metadata of the operator to allow filters/classifiers to have access to that.

The end-goal would be for example to allow a user to look for operators with license equal to None, in category invariant, for plugin core.

[ansMHanmer]

@PProfizi i've added in the metadata as requested.

On another note - I've noticed that quite a few operators were failing to retrieve their info properly because of some apparently missing data.

When trying to retrieve spec.config_specification for many operators, I am met with:

Ansys Path: C:\users\mhanmer\dpf\ansys\dpf\server_2025_1_pre0\
Server Info: {'server_ip': '', 'server_port': None, 'server_process_id': 27724, 'server_version': '9.0', 'os': 'nt', 'path': 'C:\\users\\mhanmer\\dpf\\ansys\\dpf\\server_2025_1_pre0\\'}
Server Context: Server Context of type LicensingContextType.premium with no xml path
Server Config: Server configuration: protocol=InProcess
Server version: 9.0
Traceback (most recent call last):
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 160, in <module>
    main()
    ~~~~^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 156, in main
    generate_operator_doc(server, operator_name, args.include_private)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 120, in generate_operator_doc
    operator_info = fetch_doc_info(server, operator_name)
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 60, in fetch_doc_info
    for configuration_key in spec.config_specification:
                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.venv\Lib\site-packages\ansys\dpf\core\operator_specification.py", line 452, in config_specification
    num_options = self._api.operator_specification_get_num_config_options(self)
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.venv\Lib\site-packages\ansys\dpf\gate\generated\operator_specification_capi.py", line 228, in operator_specification_get_num_config_options
    raise errors.DPFServerException(sError.value)
ansys.dpf.gate.errors.DPFServerException: C-layer: Required type is incorrect

I have added some handling around it so the docs generation can continue, but we're left with very incomplete docs in the cases where this error comes up. Perhaps if there is a certain amount of missing information we log an error and skip the doc for that particular operator.

For you to see, it's probably easiest just to try the latest version of the script on your end to see if you can reproduce and provide comment.

[PProfizi]

@PProfizi i've added in the metadata as requested.

On another note - I've noticed that quite a few operators were failing to retrieve their info properly because of some apparently missing data.

When trying to retrieve spec.config_specification for many operators, I am met with:

Ansys Path: C:\users\mhanmer\dpf\ansys\dpf\server_2025_1_pre0\
Server Info: {'server_ip': '', 'server_port': None, 'server_process_id': 27724, 'server_version': '9.0', 'os': 'nt', 'path': 'C:\\users\\mhanmer\\dpf\\ansys\\dpf\\server_2025_1_pre0\\'}
Server Context: Server Context of type LicensingContextType.premium with no xml path
Server Config: Server configuration: protocol=InProcess
Server version: 9.0
Traceback (most recent call last):
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 160, in <module>
    main()
    ~~~~^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 156, in main
    generate_operator_doc(server, operator_name, args.include_private)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 120, in generate_operator_doc
    operator_info = fetch_doc_info(server, operator_name)
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.ci\generate_operators_doc.py", line 60, in fetch_doc_info
    for configuration_key in spec.config_specification:
                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.venv\Lib\site-packages\ansys\dpf\core\operator_specification.py", line 452, in config_specification
    num_options = self._api.operator_specification_get_num_config_options(self)
  File "C:\Users\mhanmer\Repositories\GitHub\ansys\pydpf-core\.venv\Lib\site-packages\ansys\dpf\gate\generated\operator_specification_capi.py", line 228, in operator_specification_get_num_config_options
    raise errors.DPFServerException(sError.value)
ansys.dpf.gate.errors.DPFServerException: C-layer: Required type is incorrect

I have added some handling around it so the docs generation can continue, but we're left with very incomplete docs in the cases where this error comes up. Perhaps if there is a certain amount of missing information we log an error and skip the doc for that particular operator.

For you to see, it's probably easiest just to try the latest version of the script on your end to see if you can reproduce and provide comment.

Thanks @ansMHanmer I'll have a look at the issue.

[PProfizi]

I've made a MRE which lists failing operators, I'll investigate the error and try to fix it.

import ansys.dpf.core as dpf
from ansys.dpf.gate.errors import DPFServerException


failing = []
for operator_name in dpf.available_operator_names():
    op = dpf.dpf_operator.Operator(operator_name)  # "mapdl::cms_deprecated::EEL"
    spec = op.specification
    try:
        _ = spec.config_specification
    except DPFServerException as e:
        failing.append(operator_name)
print(sorted(failing))
print(len(failing))
assert len(failing) == 0

[PProfizi]

Hi @ansMHanmer, turns out the issue is that these operators do not have a specification to start with.
Until now we have returned an empty Specification for those, but then any query would fail, and this particular one does not check if the spec is empty. I can fix that with one line, but the question is whether we should not rather throw already when requesting the spec for an operator which does not have one, instead of returning an empty Specification. What do you think?

[ansMHanmer]

Hi @ansMHanmer, turns out the issue is that these operators do not have a specification to start with. Until now we have returned an empty Specification for those, but then any query would fail, and this particular one does not check if the spec is empty. I can fix that with one line, but the question is whether we should not rather throw already when requesting the spec for an operator which does not have one, instead of returning an empty Specification. What do you think?

Thanks for taking a look @PProfizi - and the change you made in #2107 seems to work well on my side!