part 2: add pyspark support to tree surrogate model in error analysis

[imatiach-msft]

Description

This is a follow-up PR to #1251. This PR includes the ErrorAnalysisDashboard changes to add pyspark support to error analysis. This will allow users to run ErrorAnalysisDashboard in Azure Synapse or Databricks as a widget using the lightgbm model from https://github.com/microsoft/SynapseML/ as the surrogate model for computing the decision tree trained on errors which is shown in the dashboard on large, TB size data.

Checklist

• I have added screenshots above for all UI changes.
• Documentation was updated if it was needed.
• New tests were added or changes were manually verified.

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[codecov-commenter]

Codecov Report

Merging #1388 (bb91a46) into main (1a54cb0) will increase coverage by 5.10%.
The diff coverage is 67.60%.

@@            Coverage Diff             @@
##             main    #1388      +/-   ##
==========================================
+ Coverage   82.16%   87.27%   +5.10%     
==========================================
  Files          31      108      +77     
  Lines        1402     5108    +3706     
==========================================
+ Hits         1152     4458    +3306     
- Misses        250      650     +400     

Flag	Coverage Δ
unittests	87.27% <67.60%> (+5.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...dgets/raiwidgets/error_analysis_dashboard_input.py	66.21% <67.60%> (-3.34%)	⬇️
responsibleai/responsibleai/_managers/__init__.py	100.00% <0.00%> (ø)
.../responsibleai/modelanalysis/constants/__init__.py	100.00% <0.00%> (ø)
..._core_flask/environments/databricks_environment.py	56.25% <0.00%> (ø)
...sponsibleai/responsibleai/rai_insights/__init__.py	100.00% <0.00%> (ø)
erroranalysis/erroranalysis/analyzer/__init__.py	100.00% <0.00%> (ø)
responsibleai/responsibleai/_internal/constants.py	100.00% <0.00%> (ø)
...core_flask/rai_core_flask/environments/__init__.py	100.00% <0.00%> (ø)
...re_flask/environments/local_ipython_environment.py	88.88% <0.00%> (ø)
rai_core_flask/rai_core_flask/__init__.py	100.00% <0.00%> (ø)
... and 68 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a54cb0...bb91a46. Read the comment docs.

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[xuke444]

Is this for Synapse and/or databricks only?
Let's hold this to post build, to freeze the sdk version for build for now.

[imatiach-msft]

yes, it is for synapse/databricks support only

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html

[github-actions]

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/ilmat/ea-spark/dashboard/index.html