Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cattr.structure isn't working for classes uncluding dicts #151

Closed
Natanela opened this issue May 30, 2021 · 2 comments
Closed

cattr.structure isn't working for classes uncluding dicts #151

Natanela opened this issue May 30, 2021 · 2 comments

Comments

@Natanela
Copy link

Natanela commented May 30, 2021

  • cattrs version: 1.7.1 (attrs-21.2.0 cattrs-1.7.1)
  • Python version: 3.8.1
  • Operating System: both Windows 10 and Linux

Description

I'm trying to structure a JSON file into a class defined with attrs.

Here is my class definition:

@attr.s(frozen=True, auto_attribs=True)
class SchedulingDefinition:
    schedule: str = attr.ib()
    job_name: str = attr.ib(default=attr.Factory(lambda self: self.notebook_name, takes_self=True))
    job_id: str = attr.ib(default=attr.Factory(lambda: hashlib.sha224(uuid.uuid4().bytes).hexdigest()[:10]))
    timeout_in_min: int = 60
    retry_count: int = 1
    parameters: Optional[Dict] = attr.ib(default=None)

I'm using the following line to destructure the JSON file:

    with open(os.path.join(scheduling_definitions_dir, scheduling_definition_file), 'rb') as f:
            definition = cattr.structure(json.load(f), SchedulingDefinition)

JSON files that contain a parameters property fail with the following error message:
Unsupported type: ~KT. Register a structure hook for it.

here is the full trace:

File "./scheduling.py", line 170, in load_scheduling_definitions
  definition = cattr.structure(json.load(f), SchedulingDefinition)
File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/cattr/converters.py", line 223, in structure
  return self._structure_func.dispatch(cl)(obj, cl)
File "", line 17, in structure_SchedulingDefinition
File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/cattr/converters.py", line 439, in _structure_union
  return self._structure_func.dispatch(other)(obj, other)
File "", line 2, in structure_mapping
File "", line 2, in <dictcomp>
File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/cattr/converters.py", line 304, in _structure_default
  raise ValueError(msg)

ValueError: Unsupported type: ~KT. Register a structure hook for it.

Here is an example JSON input file which fails to structure:

{
    "schedule": "0 15 8 ? * * *",
    "job_name": "Backfill",
    "job_id": "602608e0c9",
    "timeout_in_min": 60,
    "retry_count": 1,
    "parameters": {
        "days_back": 90,
        "shard_size": "2D"
    }
}

This logic worked perfectly fine up until v1.6.0 (included) but started to fail on 1.7.0 and 1.7.1

@Natanela Natanela changed the title cattr.structure isn't working for unannotated dicts cattr.structure isn't working for dicts May 30, 2021
@Natanela Natanela changed the title cattr.structure isn't working for dicts cattr.structure isn't working for classes uncluding dicts May 30, 2021
@Tinche
Copy link
Member

Tinche commented May 30, 2021

Hm, a regression. I will fix this! It looks like the issue is handling the parameters field.

In the meantime, your options are:

  • keep using version 1.6 until this is fixed
  • change the field definition to read: parameters: Optional[dict] = attr.ib(default=None) (so, using the built-in dict instead of the Dict from typing). This is the preferred way in 3.9 forward.
  • change the field definition to read: parameters: Optional[Dict[Any, Any]] = attr.ib(default=None) (basically flesh out the type parameters)
  • upgrade to Python 3.9, where this use case seems to work out of the box

@Tinche
Copy link
Member

Tinche commented May 31, 2021

This should be fixed on master now!

@Tinche Tinche closed this as completed Sep 22, 2021
mergify bot pushed a commit to aws/jsii that referenced this issue Jan 6, 2022
…1 in /packages/@jsii/python-runtime (#3315)

Updates the requirements on [cattrs](https://github.com/python-attrs/cattrs) to permit the latest version.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/python-attrs/cattrs/blob/main/HISTORY.rst">cattrs's changelog</a>.</em></p>
<blockquote>
<h2>1.10.0 (2022-01-04)</h2>
<ul>
<li>Add PEP 563 (string annotations) support for dataclasses.
(<code>[#195](python-attrs/cattrs#195) &lt;https://github.com/python-attrs/cattrs/issues/195&gt;</code>_)</li>
<li>Fix handling of dictionaries with string Enum keys for bson, orjson, and tomlkit.</li>
<li>Rename the <code>cattr.gen.make_dict_unstructure_fn.omit_if_default</code> parameter to <code>_cattrs_omit_if_default</code>, for consistency. The <code>omit_if_default</code> parameters to <code>GenConverter</code> and <code>override</code> are unchanged.</li>
<li>Following the changes in <code>attrs</code> 21.3.0, add a <code>cattrs</code> package mirroring the existing <code>cattr</code> package. Both package names may be used as desired, and the <code>cattr</code> package isn't going away.</li>
</ul>
<h2>1.9.0 (2021-12-06)</h2>
<ul>
<li>Python 3.10 support, including support for the new union syntax (<code>A | B</code> vs <code>Union[A, B]</code>).</li>
<li>The <code>GenConverter</code> can now properly structure generic classes with generic collection fields.
(<code>[#149](python-attrs/cattrs#149) &lt;https://github.com/python-attrs/cattrs/issues/149&gt;</code>_)</li>
<li><code>omit=True</code> now also affects generated structuring functions.
(<code>[#166](python-attrs/cattrs#166) &lt;https://github.com/python-attrs/cattrs/issues/166&gt;</code>_)</li>
<li><code>cattr.gen.{make_dict_structure_fn, make_dict_unstructure_fn}</code> now resolve type annotations automatically when PEP 563 is used.
(<code>[#169](python-attrs/cattrs#169) &lt;https://github.com/python-attrs/cattrs/issues/169&gt;</code>_)</li>
<li>Protocols are now unstructured as their runtime types.
(<code>[#177](python-attrs/cattrs#177) &lt;https://github.com/python-attrs/cattrs/pull/177&gt;</code>_)</li>
<li>Fix an issue generating structuring functions with renaming and <code>_cattrs_forbid_extra_keys=True</code>.
(<code>[#190](python-attrs/cattrs#190) &lt;https://github.com/python-attrs/cattrs/issues/190&gt;</code>_)</li>
</ul>
<h2>1.8.0 (2021-08-13)</h2>
<ul>
<li>Fix <code>GenConverter</code> mapping structuring for unannotated dicts on Python 3.8.
(<code>[#151](python-attrs/cattrs#151) &lt;https://github.com/python-attrs/cattrs/issues/151&gt;</code>_)</li>
<li>The source code for generated un/structuring functions is stored in the <code>linecache</code> cache, which enables more informative stack traces when un/structuring errors happen using the <code>GenConverter</code>. This behavior can optionally be disabled to save memory.</li>
<li>Support using the attr converter callback during structure.
By default, this is a method of last resort, but it can be elevated to the default by setting <code>prefer_attrib_converters=True</code> on <code>Converter</code> or <code>GenConverter</code>.
(<code>[#138](python-attrs/cattrs#138) &lt;https://github.com/python-attrs/cattrs/issues/138&gt;</code>_)</li>
<li>Fix structuring recursive classes.
(<code>[#159](python-attrs/cattrs#159) &lt;https://github.com/python-attrs/cattrs/issues/159&gt;</code>_)</li>
<li>Converters now support un/structuring hook factories. This is the most powerful and complex venue for customizing un/structuring. This had previously been an internal feature.</li>
<li>The <code>Common Usage Examples &lt;https://cattrs.readthedocs.io/en/latest/usage.html#using-factory-hooks&gt;</code>_ documentation page now has a section on advanced hook factory usage.</li>
<li><code>cattr.override</code> now supports the <code>omit</code> parameter, which makes <code>cattrs</code> skip the atribute entirely when unstructuring.</li>
<li>The <code>cattr.preconf.bson</code> module is now tested against the <code>bson</code> module bundled with the <code>pymongo</code> package, because that package is much more popular than the standalone PyPI <code>bson</code> package.</li>
</ul>
<h2>1.7.1 (2021-05-28)</h2>
<ul>
<li><code>Literal</code> s are not supported on Python 3.9.0 (supported on 3.9.1 and later), so we skip importing them there.
(<code>[#150](python-attrs/cattrs#150) &lt;https://github.com/python-attrs/cattrs/issues/150&gt;</code>_)</li>
</ul>
<h2>1.7.0 (2021-05-26)</h2>
<ul>
<li><code>cattr.global_converter</code> (which provides <code>cattr.unstructure</code>, <code>cattr.structure</code> etc.) is now an instance of <code>cattr.GenConverter</code>.</li>
<li><code>Literal</code> s are now supported and validated when structuring.</li>
<li>Fix dependency metadata information for <code>attrs</code>.
(<code>[#147](python-attrs/cattrs#147) &lt;https://github.com/python-attrs/cattrs/issues/147&gt;</code>_)</li>
<li>Fix <code>GenConverter</code> mapping structuring for unannotated dicts.
(<code>[#148](python-attrs/cattrs#148) &lt;https://github.com/python-attrs/cattrs/issues/148&gt;</code>_)</li>
</ul>

</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/python-attrs/cattrs/commit/7d3a6ba5e0df942391349e332cb87f6871088015"><code>7d3a6ba</code></a> Bump to 1.10.0</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/22b24c28fbeb2b8ca90d568dc4939bdc98ec5902"><code>22b24c2</code></a> Tin/import cattrs (<a href="https://github-redirect.dependabot.com/python-attrs/cattrs/issues/203">#203</a>)</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/a0e56f43f061c43814d6f938833d1c325ed61525"><code>a0e56f4</code></a> Fix test with 32-bit time_t</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/bc9432e606177fe10aa3d2d11e715970c92526be"><code>bc9432e</code></a> Documentation tweaks</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/6260c58aa185200a08c76ecc99a941a26a93eeb8"><code>6260c58</code></a> Rename gen.make_dict_unstructure_fn.omit_if_default</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/bb4383c2d97aae0e8a01db64f142d07350861a17"><code>bb4383c</code></a> Remove walrus</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/81d7756541129a73b82f076b860d61c550296666"><code>81d7756</code></a> Clean up test</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/de16200c3d02d259d04960ab466f49fcf8fa47a4"><code>de16200</code></a> Fix preconf string Enum keys</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/077c9ea8521372f706346f1f39101db716ca3089"><code>077c9ea</code></a> CI tweak</li>
<li><a href="https://github.com/python-attrs/cattrs/commit/3ad74d4456c19598e7278deef62e3af85eabc99d"><code>3ad74d4</code></a> setup.cfg B gone</li>
<li>Additional commits viewable in <a href="https://github.com/python-attrs/cattrs/compare/v1.8.0...v1.10.0">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
kaxil added a commit to astronomer/airflow that referenced this issue Jan 14, 2022
This was pinned because of issue mentioned in apache#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9
kaxil added a commit to apache/airflow that referenced this issue Jan 14, 2022
This was pinned because of issue mentioned in #16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9
potiuk pushed a commit to apache/airflow that referenced this issue Jan 22, 2022
This was pinned because of issue mentioned in #16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

(cherry picked from commit 8881458)
jedcunningham pushed a commit to apache/airflow that referenced this issue Jan 27, 2022
This was pinned because of issue mentioned in #16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

(cherry picked from commit 8881458)
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Jun 4, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

(cherry picked from commit 88814587d451be7493e005e4d477609a39caa1d9)

GitOrigin-RevId: 33e36225c3c2e202c264e0e78952961768748884
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Jul 10, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Aug 27, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Oct 4, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
aglipska pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Oct 7, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Dec 7, 2022
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
leahecole pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Jan 27, 2023
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
kosteev pushed a commit to kosteev/composer-airflow-test-copybara that referenced this issue Sep 12, 2024
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Sep 17, 2024
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this issue Nov 7, 2024
This was pinned because of issue mentioned in apache/airflow#16172 . However this was fixed in 1.8.0 of cattrs by python-attrs/cattrs#151

Changelog entry - https://cattrs.readthedocs.io/en/latest/history.html#id9

GitOrigin-RevId: 88814587d451be7493e005e4d477609a39caa1d9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants