New module: Gitlab integration #2129

drzraf · 2021-03-29T17:43:23Z

SUMMARY

New module: Gitlab integration services

Fixes ansible/ansible#40053

Reroll of #667

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

community.general.gitlab_integration

ADDITIONAL INFORMATION

This module provides a way to setup Gitlab integration services using Ansible and supports all services defined by Gitlab.com API (and their corresponding parameters) as of ~~2020/07/09~~ 2022/02/17.

Known bugs:

Most Gitlab integration services rely upon secret parameter (password, token, api_key) which are write-only (ie: not exposed on GET request). That currently keeps module from being idempotent when such a suboption is used.
The active state of a service could not be set using the API. Use state=absent to completely remove the service configuration instead of deactivating it. But some work happened recently on GitLab side, it may be fixed by now.

EXTENDED DEFINITION OF SERVICE

Their validation is done by doing a redundant initialization of the AnsibleModule() and modifying the argument spec meanwhile.

~~- The extended definition of services and their suboptions (name, required or not, type) is generated from Gitlab upstream file the 2021/03/29.~~

The extended definition of services and their suboptions (name, required or not, type) is generated from Gitlab upstream file the 2022/02/17.

plugins/modules/source_control/gitlab/gitlab_service.py

ansibullbot · 2021-03-30T06:08:24Z

cc @Lunik @Shaps @dj-wasabi @marwatk @scodeman @waheedi @zanssa
click here for bot help

plugins/modules/source_control/gitlab/gitlab_service.py

felixfontein · 2021-04-02T11:32:49Z

plugins/modules/source_control/gitlab/gitlab_service.py

+        base_spec['params'] = dict(required=False, type='dict', options=sub_arg_specs)
+        if '_events' in definitions[service]:
+            base_spec['events'] = dict(required=True, type='list', elements='str', choices=definitions[service]['_events'])
+        module = AnsibleModule(argument_spec=base_spec, **constraints)


I don't think AnsibleModule is meant to be used like this.

You mean re-instanciating an AnsibleModule a second time in order to further validate parameters based on a first pass (because a complex dict parameter depends upon the value of another)... Then yes, I admit it's indeed exotic (and this is the first plugin to rely on such a construct).

That being said we already discussed the question of parameters (and the benefits of validation) (and so far I couldn't think of any negative side effect for the double AnsibleModule instanciation)

I really don't think it is a good idea to do this. I also asked on #ansible-devel, and one of the core team members confirmed that this is something that they support. It's basically undefined behavior, and use at your own risk.

I really don't think it is a good idea to do this. I also asked on #ansible-devel, and one of the core team members confirmed that this is something that they not support. It's basically undefined behavior, and use at your own risk.

Damn, I missed the most important word in this: this is something they do not support. While it works now, it can stop working at a random point in the future.

Don't you think it's very convenient for the user and that if it were to stop working, we could adapt it quite easily?
The usability benefit are huge (and I think this could be useful to other module if it started to be supported in the future).
Shouldn't we try it? And if not, what would you suggest instead?

Don't you think it's very convenient for the user

It definitely is! But that can (and should) also be achieved differently than relying on undefined behavior.

and that if it were to stop working, we could adapt it quite easily?

Well, it depends on who will do the work. Such cleanup work often falls back to collection maintainers because the people who originally implemented it are no longer around to clean it up.

The usability benefit are huge (and I think this could be useful to other module if it started to be supported in the future).

The benefits of extra validation are indeed great, but doing that by instantiating AnsibleModule twice is IMO the wrong solution for that problem.

And if not, what would you suggest instead?

Implement the validation manually, preferably as a module_utils so it can be used by other modules as well. Useful parts for that already exist in Ansible(-base/-core)'s module_utils, but unfortunately they change a lot between the Ansible versions (2.9, 2.10, 2.11) and thus most of them cannot be used. What can be used are the things in https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/common/validation.py that were already present in https://github.com/ansible/ansible/blob/stable-2.9/lib/ansible/module_utils/common/validation.py, which covers all check_type_xxx functions, and check_required_arguments.

drzraf · 2021-04-05T15:35:54Z

In 372f6c5 I tried to add some tests.
Integration can't be run (disabled) so it's mostly an example.
The units tests are more tricky (just mostly added boilerplate code)

plugins/modules/source_control/gitlab/gitlab_service.py

tests/unit/plugins/modules/source_control/gitlab/test_gitlab_service.py

tests/integration/targets/gitlab_service/aliases

drzraf · 2021-04-11T16:47:00Z

Some months ago, GitLab services were renamed to GitLab integrations. Should I take that into account :S ?

felixfontein · 2021-04-17T12:04:28Z

Some months ago, GitLab services were renamed to GitLab integrations. Should I take that into account :S ?

Just wait a couple of more months until it is renamed something else again, and only then change it. Just joking ;) I guess it's better to update the module now before it has been released, since renaming the module, options, return values etc. later is a huge PITA, and having a different name than it is used in GitLab itself is also confusing to users.

drzraf · 2021-04-18T00:43:45Z

Done.
Note that python-gitlab still name them Project services

Co-authored-by: Nejc Habjan <hab.nejc@gmail.com>

felixfontein · 2022-02-23T22:41:42Z

plugins/modules/source_control/gitlab/gitlab_integration.py

+def mimic_ansible_module_validation(specs, constraints):
+    from ansible.module_utils.basic import ModuleArgumentSpecValidator, _load_params
+    from ansible.module_utils.errors import UnsupportedError
+    import os


These imports must be done on the global level, not locally in this function.

(This will also likely show that this does not work with Ansible 2.9 and ansible-base 2.10.)

Also you should not use _load_params. That's an internal API.

Finally, you should not re-validate the whole argument spec, but only the relevant option.

Ok about global level function import

re-validate the whole argument spec, but only the relevant option

But ... this implies, again, more code, more complexity than revalidating it (which does not hurt).

About you should not use _load_params, please please, have a look at the its inline documentation:

We cannot guarantee API stability for it (it may change between versions) however we will try not to break it gratuitously. It is certainly more future-proof to call this function and consume its outputs than to implement the logic inside it as a copy in your own code.

=> The programmer who actually wrote this function tells us exactly the contrary : if you need me, then use me.

It feels bad how far fetched this debate about AnsibleModule is leading us.
My intent to do proper validation of arguments, of being strict and DRY ends up in more code, more workarounds, more complication, more maintenance hurdle, more duplication of the private code base because... well the private method is private so lets copy the LoC instead of calling it.

The documentation of _load_params is exactly telling us that : Yes, I'm private but you know, it's better to reuse me than to rewrite me (and it's basically the same about the whole AnsibleModule which wasn't misused at all but just instantiating twice what was never forbidden in the first place ).

I feel we are loosing the big picture and the goal by blindly overfocusing on strict unquestioned "norms" (interpreted for the worst instead of the better) and that I'm pushed into a path that will lead to complex, bloated and inelegant code (that won't be merged either).

But ... this implies, again, more code, more complexity than revalidating it (which does not hurt).

It does not.

About you should not use _load_params, please please, have a look at the its inline documentation:

It's still something that should really not be used in module code. It's ok to use in tests though.

felixfontein · 2022-02-23T22:42:33Z

plugins/modules/source_control/gitlab/gitlab_integration.py

+    if error:
+        msg = validation_result.errors.msg
+        if isinstance(error, UnsupportedError):
+            msg = "Unsupported parameters for ({name}) {kind}: {msg}".format(name=os.path.basename(__file__), kind='module', msg=msg)


There is no need to use os.path.basename() here.

Why?
(lib/ansible/module_utils/basic.py does it that way and I stuck to the standard error output)

Because this is a specific module, not a module utils.

ansibullbot · 2022-03-12T14:55:41Z

cc @suukit
click here for bot help

felixfontein · 2022-04-26T20:09:38Z

This New Module PR contains a symbolic link from plugins/modules/ to the actual Python file. Since #4562 this is no longer necessary and will soon be flagged as an error. Instead you need to add an entry to meta/runtime.yml, similar to this one: https://github.com/ansible-collections/community.general/blob/main/meta/runtime.yml#L21-L22

See also our updated instructions on creating new modules: https://github.com/ansible-collections/community.general/blob/main/CONTRIBUTING.md#creating-new-modules-or-plugins

felixfontein · 2022-10-11T05:39:07Z

tests/integration/targets/gitlab_integration/aliases

@@ -0,0 +1,2 @@
+shippable/posix/group1


Suggested change

shippable/posix/group1

azp/posix/1

(see #5344)

felixfontein · 2022-11-03T06:02:15Z

Please note that in #5461 the collection repository was restructured to remove the directory tree in plugins/modules/, and the corresponding tree in tests/unit/plugins/modules/. Your PR adds new files into this hierarchy. Please rebase with the current main branch and move your files directly into plugins/modules/. You also can remove the changes to meta/runtime.yml, these are not needed anymore (the corresponding section was removed from CONTRIBUTING.md), and make sure to adjust the new entries in .github/BOTMETA.yml as well.

felixfontein · 2022-11-08T19:32:11Z

Do you still plan to update this PR according to the review comments?

needs_info

drzraf · 2022-11-11T21:06:54Z

Do you still plan to update this PR according to the review comments?

Sorry but I still disagree with the reviewers suggestion of reimplementing, badly, the _load_params function instead of reusing it (as original author anticipated in code comment). Drop suboptions validation doesn't seem right either.

I'm using a version of this module locally on a regular basis but I won't produce yet another PR containing bad/inefficient/unmaintainable/inelegant code... without a technical reason being given.
I hope reviewers will reconsider their choice.

Although I did the initial PR Jul 17th, 2020, the future of this module's [possible inclusion] is definitely not in my hands.

Good luck to you and/or to other (still) motivated Ansible contributors.

felixfontein · 2022-11-12T08:36:35Z

Sorry but I still disagree with the reviewers suggestion of reimplementing, badly, _load_params instead of reusing it (as code author anticipated) or having to drop suboptions validation altogether.

You can reuse it, but not in the way you are doing it right now.

If you don't want to change the code, you will have to find someone else who can get your module merged. I don't merge things that are simply wrong from my point of view, sorry.

felixfontein · 2023-05-10T05:46:02Z

needs_info

drzraf · 2023-05-11T11:15:32Z

You can reuse it, but not in the way you are doing it right now.

...

I'm happy you changed your mind but how do you suggest to use it differently?

felixfontein · 2023-05-11T18:25:46Z

I didn't change my mind. Using internal things in production code is still a bad idea.

russoz · 2024-10-07T05:42:16Z

Is this still going to be worked on?

needs_info

Akasurde requested changes Mar 30, 2021

View reviewed changes

felixfontein reviewed Mar 31, 2021

View reviewed changes

felixfontein reviewed Apr 2, 2021

View reviewed changes

ansibullbot added integration tests/integration tests tests unit tests/unit labels Apr 5, 2021

felixfontein reviewed Apr 7, 2021

View reviewed changes

ansibullbot added community_review needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR community_review labels Apr 11, 2021

ansibullbot removed the needs_triage label Apr 17, 2021

ansibullbot added community_review and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR community_review labels Apr 18, 2021

Update plugins/modules/source_control/gitlab/gitlab_integration.py

4e2d075

Co-authored-by: Nejc Habjan <hab.nejc@gmail.com>

ansibullbot removed the ci_verified Push fixes to PR branch to re-run CI label Feb 22, 2022

remove references to api_job_token

6df81d1

ansibullbot added the ci_verified Push fixes to PR branch to re-run CI label Feb 22, 2022

felixfontein reviewed Feb 23, 2022

View reviewed changes

ansibullbot added the stale_ci CI is older than 7 days, rerun before merging label Mar 4, 2022

felixfontein removed the backport-4 label Apr 27, 2022

felixfontein reviewed Oct 11, 2022

View reviewed changes

ansibullbot added the needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html label Nov 3, 2022

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label Nov 8, 2022

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label Nov 11, 2022

ansibullbot removed the ci_verified Push fixes to PR branch to re-run CI label Apr 20, 2023

felixfontein added the backport-7 Automatically create a backport for the stable-7 branch label May 10, 2023

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label May 10, 2023

ansibullbot removed the needs_info This issue requires further information. Please answer any outstanding questions label May 11, 2023

ansibullbot added needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI and removed stale_ci CI is older than 7 days, rerun before merging labels Aug 31, 2023

felixfontein removed the affects_2.10 label Apr 26, 2024

felixfontein removed the backport-7 Automatically create a backport for the stable-7 branch label May 19, 2024

ansibullbot added the needs_info This issue requires further information. Please answer any outstanding questions label Oct 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New module: Gitlab integration #2129

New module: Gitlab integration #2129

drzraf commented Mar 29, 2021 •

edited

Loading

ansibullbot commented Mar 30, 2021

felixfontein Apr 2, 2021

drzraf Apr 5, 2021 •

edited

Loading

felixfontein Apr 7, 2021

felixfontein Apr 17, 2021

drzraf Apr 18, 2021

felixfontein Apr 18, 2021

drzraf commented Apr 5, 2021

drzraf commented Apr 11, 2021 •

edited

Loading

felixfontein commented Apr 17, 2021

drzraf commented Apr 18, 2021

felixfontein Feb 23, 2022

drzraf Feb 24, 2022 •

edited

Loading

felixfontein Feb 24, 2022

felixfontein Feb 23, 2022

drzraf Feb 24, 2022

felixfontein Feb 24, 2022

ansibullbot commented Mar 12, 2022

felixfontein commented Apr 26, 2022

felixfontein Oct 11, 2022

felixfontein commented Nov 3, 2022

felixfontein commented Nov 8, 2022

drzraf commented Nov 11, 2022 •

edited

Loading

felixfontein commented Nov 12, 2022

felixfontein commented May 10, 2023

drzraf commented May 11, 2023

felixfontein commented May 11, 2023

russoz commented Oct 7, 2024

New module: Gitlab integration #2129

Are you sure you want to change the base?

New module: Gitlab integration #2129

Conversation

drzraf commented Mar 29, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

EXTENDED DEFINITION OF SERVICE

ansibullbot commented Mar 30, 2021

Choose a reason for hiding this comment

drzraf Apr 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drzraf commented Apr 5, 2021

drzraf commented Apr 11, 2021 • edited Loading

felixfontein commented Apr 17, 2021

drzraf commented Apr 18, 2021

Choose a reason for hiding this comment

drzraf Feb 24, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ansibullbot commented Mar 12, 2022

felixfontein commented Apr 26, 2022

Choose a reason for hiding this comment

felixfontein commented Nov 3, 2022

felixfontein commented Nov 8, 2022

drzraf commented Nov 11, 2022 • edited Loading

felixfontein commented Nov 12, 2022

felixfontein commented May 10, 2023

drzraf commented May 11, 2023

felixfontein commented May 11, 2023

russoz commented Oct 7, 2024

drzraf commented Mar 29, 2021 •

edited

Loading

drzraf Apr 5, 2021 •

edited

Loading

drzraf commented Apr 11, 2021 •

edited

Loading

drzraf Feb 24, 2022 •

edited

Loading

drzraf commented Nov 11, 2022 •

edited

Loading