[AIRFLOW-3795] provide_context param is now used

[ssoto]

Make sure you have checked all steps below.

Jira

• [x ] My PR addresses the following Airflow Jira issues and references them in the PR title.
  • https://issues.apache.org/jira/browse/AIRFLOW-3795

Description

• Here are some details about my PR:

For a PythonOperator, I can set provide_context=True. For a PythonVirtualenvOperator, I am not able to set provide_context=True, and provide_context=False gets passed to the super function. I do not see why this is hardcoded, and I would like to be able to use the context with a PythonVirtualenvOperator.

See:

airflow/airflow/operators/python_operator.py

Line 242 in 83cb9c3

provide_context=False,

Tests

• [ x ] My PR adds the following unit tests.

Commits

• [ x] My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• [ x] In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

Code Quality

• [ x] Passes flake8

[ssoto]

Anyone can help me with these fails? Don't know tests not related with my change are failing.

[OmerJog]

@ssoto The error is on the function that you wrote

======================================================================
69) ERROR: test_provide_context (tests.operators.test_virtualenv_operator.TestPythonVirtualenvOperator)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/operators/test_virtualenv_operator.py line 214 in test_provide_context
      provide_context=True
    airflow/utils/decorators.py line 98 in wrapper
      result = func(*args, **kwargs)
    airflow/operators/python_operator.py line 284 in __init__
      self.__class__.__name__)
   AirflowException: ('{} only supports functions for python_callable arg', 'PythonVirtualenvOperator')

[ssoto]

This built failed only on python3.5 Docker Sqlite environment.

Test that fails is test_invoke_lambda_function, test_impersonation, test_impersonation_custom test_impersonation_subdag...

[Fokko]

I've restarted the failing job to check if the error is still there.

[codecov-io]

Codecov Report

Merging #4735 into master will increase coverage by 0.77%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #4735      +/-   ##
==========================================
+ Coverage   74.49%   75.26%   +0.77%     
==========================================
  Files         430      450      +20     
  Lines       28003    30400    +2397     
==========================================
+ Hits        20861    22881    +2020     
- Misses       7142     7519     +377

Impacted Files	Coverage Δ
airflow/operators/python_operator.py	97.72% <100%> (+2.46%)	⬆️
airflow/utils/dag_processing.py	56.16% <0%> (-2.66%)	⬇️
airflow/sensors/sql_sensor.py	100% <0%> (ø)	⬆️
airflow/contrib/hooks/gcp_vision_hook.py	82.85% <0%> (ø)
airflow/_vendor/nvd3/linePlusBarChart.py	25% <0%> (ø)
airflow/contrib/operators/gcp_vision_operator.py	100% <0%> (ø)
airflow/api/common/experimental/get_code.py	83.33% <0%> (ø)
airflow/_vendor/nvd3/lineChart.py	81.81% <0%> (ø)
airflow/_vendor/nvd3/__init__.py	100% <0%> (ø)
airflow/_vendor/nvd3/discreteBarChart.py	31.57% <0%> (ø)
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c48e83e...6452864. Read the comment docs.

[Fokko]

Thanks @ssoto

[ssoto]

Thanks for merge @Fokko , How can I know which version of apache-airflow will have this fix?

[ashb]

@ssoto I've marked it for inclusion in 1.10.3 which I'm hopeful will be out by the end of next week. But I can't promise that timeline.

[ashb]

@ssoto That said: provide context might have a few quirks/edge cases for the venv operator: https://issues.apache.org/jira/browse/AIRFLOW-2738