Support pod identity in spark-run

[nurdann]

Changes

• Picking up from Add support for use-web-identity in spark-run #3787
• args.no_aws_credentials was removed in MLCOMPUTE-967 | changes for Spark drivers on k8s #3679 [error]
• Waiting on being merged Allow get_aws_credentials to assume_role_with_web_identity service_configuration_lib#137
• get_signalfx_url() was deleted in Remove Spark SignalFx dashboard URLs service_configuration_lib#143
  • Should be safe because the dashboard says that the license has expired

[nurdann]

Manually setting pod identity worked https://fluffy.yelpcorp.com/i/3Gcd1qTzDjNP1z4zKD3t7c4D1d7SHDd8.html

Note that I had to manually set duration to --aws-role-duration 3600 because that's role's max session duration, and the default is above that

paasta/paasta_tools/cli/cmds/spark_run.py

Line 386 in c4dde2d

default=43200,

[nurdann]

Don't get why setting autospec=True fails for 98b1217 with

FAILED tests/cli/test_cmds_spark_run.py::test_get_aws_credentials - TypeError: autospec creates the mock for you. Can't specify autospec and new.

[88manpreet]

Change lgtm. Can you update the documentation https://yelpwiki.yelpcorp.com/pages/viewpage.action?pageId=283750257, to append the section with following details:

• In what cases user should use --use-web-identity argument
• An example

If possible would suggest to push on Monday.

[88manpreet]

+1. Thanks for adding in documentation.