adopt(V1Secret) fails

[kopf-archiver]

An issue by fsniper at 2020-01-03 16:24:10+00:00
Original URL: zalando-incubator/kopf#286
 

Long story short

When trying to adopt a V1Secret object, adoption fails with
AttributeError: 'V1Secret' object has no attribute 'setdefault'

Description

The code snippet to reproduce the issue

     metadata = kubernetes.client.V1ObjectMeta(name = meta['name'])
     payload = {
             'HOST': os.environ['HOST'],
     }
     data = kubernetes.client.V1Secret(string_data=payload, metadata=metadata)
     kopf.adopt(data)

The exact command to reproduce the issue

kopf run ...

The full output of the command that failed

[2020-01-03 16:09:15,471] kopf.objects         [ERROR   ] [test/example-azure-postgresql-database] Handler 'on_create_handler' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 523, in _execute_handler
    lifecycle=lifecycle,  # just a default for the sub-handlers, not used directly.
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 612, in _call_handler
    **kwargs,
  File "/usr/local/lib/python3.7/site-packages/kopf/reactor/invocation.py", line 115, in invoke
    result = await loop.run_in_executor(config.WorkersConfig.get_syn_executor(), real_fn)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "apg_op.py", line 62, in on_create_handler
    create_secret(meta, spec, dbname, roles)
  File "apg_op.py", line 28, in create_secret
    kopf.adopt(data)
  File "/usr/local/lib/python3.7/site-packages/kopf/toolkits/hierarchies.py", line 139, in adopt
    append_owner_reference(objs, owner=real_owner)
  File "/usr/local/lib/python3.7/site-packages/kopf/toolkits/hierarchies.py", line 28, in append_owner_reference
    refs = obj.setdefault('metadata', {}).setdefault('ownerReferences', [])
AttributeError: 'V1Secret' object has no attribute 'setdefault'

Environment

• Kopf version: 0.24
• Kubernetes version: v1.13.10
• Python version: 3.7.5
• OS/platform: MacOs Mojave

Python packages installed

adal==1.2.0
aiohttp==3.6.2
aiojobs==0.2.2
altgraph==0.16.1
arrow==0.12.1
asn1crypto==0.24.0
astroid==2.2.5
async-timeout==3.0.1
attrs==19.1.0
azure-status==0.0.2
beautifulsoup4==4.6.3
binaryornot==0.4.4
bleach==3.1.0
boto==2.49.0
boto3==1.9.60
botocore==1.12.109
CacheControl==0.12.5
cachetools==3.0.0
cachy==0.2.0
certifi==2018.11.29
cffi==1.11.5
chardet==3.0.4
cleo==0.6.8
Click==7.0
click-default-group==1.2
click-plugins==1.0.4
colorama==0.3.9
colorlog==3.1.4
configparser==3.5.0
cookiecutter==1.6.0
cryptography==2.4.2
dateparser==0.7.0
delegator.py==0.1.1
docutils==0.14
envoy==0.0.3
future==0.17.1
gitdb2==2.0.5
GitPython==2.1.11
google-auth==1.6.1
greenlet==0.4.15
html5lib==1.0.1
humanfriendly==4.17
humanize==0.5.1
hvac==0.9.1
idna==2.7
iso8601==0.1.12
isort==4.3.21
Jinja2==2.10
jinja2-time==0.2.0
jmespath==0.9.3
jsonschema==3.0.2
kopf==0.24
kubernetes==10.0.1
lazy-object-proxy==1.4.2
lockfile==0.12.2
macholib==1.11
marathon==0.10.0
MarkupSafe==1.1.0
maya==0.5.0
mccabe==0.6.1
memoized-property==1.0.3
msgpack==0.6.0
multidict==4.7.2
oauthlib==2.1.0
pastel==0.1.1
pefile==2019.4.18
pendulum==1.5.1
pexpect==4.6.0
pip==19.3.1
pkginfo==1.5.0.1
ply==3.10
poetry==0.12.17
poyo==0.4.2
prometheus-client==0.1.0
ptc==0.1.1
ptc-dcos==0.1.6
ptc-msai==0.2.2
ptc-startproject==0.1.0
ptc-terraform==0.1.6
ptyprocess==0.6.0
py==1.7.0
py-postgresql==1.2.1
pyasn1==0.4.4
pyasn1-modules==0.2.2
pybitbucket==0.12.0
pycparser==2.19
Pygments==2.3.0
pyhcl==0.3.10
PyInstaller==3.4
PyJWT==1.7.0
pykube-ng==19.12.1
pylev==1.3.0
pylint==2.3.1
pynvim==0.3.1
pyparsing==2.4.2
pyrsistent==0.14.11
pyrx-ats==0.3.1
python-dateutil==2.7.5
python-terraform==0.10.0
pytz==2018.7
pytzdata==2018.7
PyYAML==3.13
readme-renderer==24.0
regex==2018.11.22
requests==2.21.0
requests-oauthlib==0.8.0
requests-toolbelt==0.8.0
rsa==3.4.2
s3transfer==0.1.13
setuptools==40.6.3
shellingham==1.3.1
simplejson==3.17.0
six==1.11.0
smmap2==2.0.5
snaptime==0.2.4
tabulate==0.8.2
toml==0.10.0
tomlkit==0.5.5
tqdm==4.29.0
twine==1.12.1
typed-ast==1.4.0
typing-extensions==3.7.4.1
tzlocal==1.5.1
uritemplate==0.6
urllib3==1.25.7
virtualenv==16.2.0
voluptuous==0.11.7
webencodings==0.5.1
websocket-client==0.54.0
wheel==0.32.3
whichcraft==0.5.2
wrapt==1.11.2
xar==19.4.22
yarl==1.4.2

---

Commented by nolar at 2020-01-07 10:07:46+00:00
 

Thanks for reporting.

Currently, only raw dicts are supported. The classes of other libraries are not yet supported. Though, it would be a good addition for this specific toolkit — hierarchy management — so that it can also accept kubernetes's and pykube-ng's classes for adopting/labelling/annotating/owning.

---

Commented by janvdvegt at 2020-01-27 17:29:02+00:00
 

I'm interested in picking this up, although I only started using kopf yesterday. Is my observation correct that the base kubernetes client is not used anywhere? I'm looking at that part of the code now and I would need some guidance on what would be a clean way to fit it in. Where would the conditions on whether it is a dictionary vs a kubernetes object go?

---

Commented by nolar at 2020-01-27 21:10:16+00:00
 

janvdvegt That is correct: Kubernetes API clients are not used in Kopf. Only aiohttp is used — for API communication (low-level).

However, Kopf has slightly extended behaviour if kubernetes and/or pykube-ng are installed — it will reuse their authentication methods. But it does not require them to be present (effectively limiting the authentication capabilities to simple tokens & username-passwords & ssl certs).

The hierarchy logic is fully located in kopf.toolkits.hierarchies — 3rd-party classes support should go there too.

Can you please briefly describe what is your intention — i.e. what are you going to change/implement to fix this issue (conceptually)?

PS: Please, be aware of #298 before you start implementing anything — it is a huge shuffle of classes & modules — the logic remains exactly the same though. It does not affect the hierarchies toolkit, luckily.

---

Commented by janvdvegt at 2020-01-28 08:32:29+00:00
 

On a base level it would be the methods in kopf.toolkits.hierarchies to also accept the kubernetes objects like V1Secret so that users can manipulate these objects as well with regards to ownership. This would mean the definition of a K8sObject would include these objects from kubernetes. It looks like all the logic can stay inside the hierarchies module.

[nolar]

This is now solved in #672 and will be released soon. UPD: The fix is released in kopf>=1.30.0rc1.

Docs: https://kopf.readthedocs.io/en/latest/hierarchies/