Rewrite KFP code generation

[ptitzler]

This PR:

• rewrites code generation for Kubeflow Pipelines v1 (v2 is still not supported)
• adds a new (previously removed) export option to the pipeline editor, enabling output of Python DSL in addition to the already supported YAML output (YAML remains the pre-selected option)
  
• adds a new optional format parameter to the elyra-pipeline export CLI command:
  • if the parameter is not specified, the export format defaults to YAML for Kubeflow Pipelines and PY for Apache Airflow
  • if the parameter is specified, it must be one of YAML / PY (Kubeflow Pipelines) or PY (Airflow)
  • The value is processed in a case insensitive manner, e.g. YAML = yAML = yaml

  $ elyra-pipeline export one-custom-node.pipeline --runtime-config cloning1 --format py
  

• Fixes an existing pipeline export bug (image pull secret information is not exported, rendering the pipeline unusable)

Closes #2986

Follow-up for tests: #3002

What changes were proposed in this pull request?

The Kubeflow Pipelines processor now always generates Python DSL code as intermediary output when a pipeline is submitted or exported:

• Submit: Internal Elyra pipeline representation -> Python DSL (generated by processor) -> YAML (generated by KFP argo/tekton compiler)
• Export: Internal Elyra pipeline representation -> Python DSL (generated by processor)
• Export: Internal Elyra pipeline representation -> Python DSL (generated by processor) -> YAML (generated by KFP argo/tekton compiler)
• The pipeline documentation was updated to list the new output format.

How was this pull request tested?

• Updated existing CLI tests
• Added new server tests that validate that code generation yields the expected results. There are now dedicated tests for all code generation aspects:
  • generates expected code for the configured workflow engine
  • generates expected code for CRIO environments
  • generates expected code for a plain generic component
  • generates expected code for a plain generic component that utilizes a runtime image, which is protected by a pull secret
  • generates expected code for a generic component configured with all supported elyra-owned properties
  • generates expected code for a plain custom component
  • generates expected code for data exchange between generic components
• Reviewed the output of make docs

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

[elyra-bot]

Thanks for making a pull request to Elyra!

To try out this branch on binder, follow this link:

[akchinSTC]

small nits, CRIO options look right and exported configs (volumes and modified bootstrapper options) look good.

[ptitzler]

Based on a discussion we've had during today's dev meeting, I've updated code generation to "unload" the generated Python DSL module after it was compiled. This change should address the potential concern that over time module artifacts might accumulate, which could lead to increased memory consumption over long periods of time.

[kiersten-stokes]

Wow this is awesome! Tested a ton of scenarios and all are turning out as expected 🎉

[ptitzler]

To address offline review feedback I've updated the Python DSL template to render a comment that identifies the node name:

    # Task for node 'Download File'
    task_8ee5ee17_1222_434e_bcf4_fd12f43e2510 = factory_8e4384f422a088e4814024df7955e952c1488bd091fa0d4873d5f611d741ceb4(
        url="https://raw.gith...",
    )

This is similar to what is done for Apache Airflow