Google Cloud Providers - Fix _MethodDefault deepcopy failure #29518
Conversation
boring-cyborg
bot
added
provider:cncf-kubernetes
IKholopov
force-pushed
the
ikholopov/deepcopy-google-operators
branch
from
dec03c2 to
bf92064
Compare
| class GoogleBaseOperator(BaseOperator): | ||
| """ | ||
| Abstract base class that takes care of common specifics of the operators built | ||
| on top of Google API client libraries. | ||
| """ |
There was a problem hiding this comment.
Our goal is to split Google provider to smaller providers #15933
This class seems to be GCP specific so lets keep it in the cloud directory and not in common as this directory will not stay after the split.
Possibly also rename the class toGcpBaseOperator ?
There was a problem hiding this comment.
Good idea, especially since AFAIU only Cloud providers are affected. GCP is a bit of a deprecated term, but "Google Cloud Base" should do that.
IKholopov
force-pushed
the
ikholopov/deepcopy-google-operators
branch
6 times, most recently
from
67d97d5 to
67bd5be
Compare
IKholopov
changed the title
| """ | ||
|
|
||
| def __deepcopy__(self, memo): | ||
| memo[id(DEFAULT)] = DEFAULT |
There was a problem hiding this comment.
Would be nice to add explanation why we are doing it.
There was a problem hiding this comment.
(Link to some description/issue explaining it because otherwise it is pretty magical.
|
@IKholopov - can you please split this one in two PRs:
That will be much better when it comes to historical look at the changes. It's quite hard to find the deepcopy fix among all the changed base operators. |
IKholopov
force-pushed
the
ikholopov/deepcopy-google-operators
branch
2 times, most recently
from
b1a5080 to
5cc71b1
Compare
IKholopov
changed the title
|
Sounds good, created #29680 |
IKholopov
force-pushed
the
ikholopov/deepcopy-google-operators
branch
2 times, most recently
from
a6bdccc to
5ac4d75
Compare
merged. |
This is the attempt to fix apache#28751 by setting a memo to the same instance of DEFAULT, which is the stub for a Literal value type introduced to support Python 3.7 with mypy in Google's API core Python client. Without this any operator that has the parameter set to DEFAULT constant (which is often the default value for retry parameters) will throw the error in mini-scheduler after execution as the attemt to deepcopy this project would fail.
IKholopov
force-pushed
the
ikholopov/deepcopy-google-operators
branch
from
5ac4d75 to
9c53f94
Compare
|
@potiuk, any other feedback that you would like me to address or are we good to merge? |
This is the attempt to fix #28751 by setting a memo to the same instance of DEFAULT, which is the stub for a Literal value type introduced to support Python 3.7 with mypy in Google's API core Python client.
closes: #28751