[WIP] Remove dataclasses json
dependency from flytekit V2
#2709
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I am going to collaborate with @guyarad, an engineer from
Nimble Way
to finish this.Tracking issue
flyteorg/flyte#4418
Why are the changes needed?
Reduce container's size.
What changes were proposed in this pull request?
The philosophy of removing dataclasses-json from flytekit but keep ensuring backward compatibility is to try
import dataclasses_json
exceptImportError
for all cases.When we can't use api from
dataclasses-json
, we can usemashumaro
.example: https://github.com/flyteorg/flytekit/pull/2709/files#diff-542ea91d83fb95892fd56588f09967f76665eed2f6cb40afd1fddde776800875R164-R171
(1) get a new class called
BaseClass
(2) change every dataclasses with "DataClassJsonMixin" parent class to
BaseClass
Before
After
command line input (click_types)
example: fcb49c8#diff-cc3172a3cdf60bb2ffb23b81cb5b92d6c4a89063d691e86932b312e1df09f292R317-R328
from_json
andto_json
functionsexample: fcb49c8#diff-c56afc489e5060224889036253b85ae7c7f1df9f0f7b1130916d9c614293cbb9R963-R968
the
from_json
andto_json
functions are fromDataClassJsonMixin
we can should use
decoder.decode(json_str)
whenget_literal_type
example: https://github.com/flyteorg/flytekit/pull/2709/files#diff-68624e201f644896b015de1a5716eabef9b8b5a106caba1d9f21c51c0708684bR442-R443
from marshmallow_jsonschema import JSONSchema
from mashumaro.jsonschema import build_json_schema
dataclass_json()
example: https://github.com/flyteorg/flytekit/pull/2709/files#diff-68624e201f644896b015de1a5716eabef9b8b5a106caba1d9f21c51c0708684bR1849-R1853
for unit tests, we should use
pytest skip if dataclasses_json
is not in the dependencyadd CI to test cases when we remove dataclass-json
add warning logs when
dataclasses-json
is neededHow was this patch tested?
Setup process
Screenshots
Check all the applicable boxes
Related PRs
#2557
Docs link