Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files Browse the repository at this point in the history
…into response_backcompat_mixin

* 'main' of https://github.com/Azure/azure-sdk-for-python: (43 commits)
  Clean-up changelog scripts (Azure#20779)
  [AutoRelease] t2-containerinstance-2021-09-17-04542 (Azure#20733)
  [SchemaRegistry] remove cache from client (Azure#20760)
  Update add_sanitizer and doc (Azure#20769)
  Enable release stage selection at queue time (Azure#20767)
  [Key Vault] Add certificate import sample (Azure#20641)
  Fixing VCR redirection (Azure#20747)
  Link to Log Analytics throttling guidance from Monitor Query README (Azure#20759)
  Fix broken anchor tags (Azure#20751)
  avroserializer pins its dependency, which makes regression testing a … (Azure#20735)
  [AutoRelease] t2-eventhub-2021-09-17-55263(Do not merge) (Azure#20739)
  Fix broken link in python repo (Azure#20746)
  [Storage]Unify service version and update changelog (Azure#20723)
  Sync eng/common directory with azure-sdk-tools for PR 2010 (Azure#20729)
  [AutoRelease] t2-azurearcdata-2021-09-15-74995 (Azure#20714)
  [Test Proxy] Make add_sanitizer a module-level method (Azure#20701)
  [Test proxy] Add migration guide (Azure#20469)
  Sync eng/common directory with azure-sdk-tools for PR 2011 (Azure#20702)
  protobuf to handle python 2.7 issues (Azure#20725)
  add troubleshoot doc (Azure#20684)
  ...
iscai-msft committed Sep 22, 2021
2 parents 1763994 + eca1c81 commit eb44d19
Showing 531 changed files with 59,410 additions and 4,351 deletions.
2 changes: 1 addition & 1 deletion doc/dev/README.md
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ Overview of the documents:
- [Developer Set-Up](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/dev_setup.md) : How to create a development environment for this repo
- [Release](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/release.md) : How to release a package when ready
- [Packaging](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md) : How to organize packaging information for packages under `azure`
- [Testing](./tests.md): How to write unit and functional tests for a library
- [Testing](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md): How to write unit and functional tests for a library
- [Docstrings and Type hints](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring_typehint.md): How to document an SDK for compatability with internal tools (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site.

The [mgmt](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`)
2 changes: 1 addition & 1 deletion doc/dev/docstring_typehint.md
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ All public methods should have docstrings to document the parameters, keywords,

* [Docstrings](#docstrings)
* [Method Docstrings](#method_docstrings)
* [Model and Client Docstrings](#Model_and_Client_Docstrings)
* [Model and Client Docstrings](#model_and_client_docstrings)
* [Type Hints](#type_hints)
* [Type Hints for Python 2.7 and 3.5+](#type_hints_for_python_2.7_and_3.5+)
* [Type Hints for Python 3.5+](#type_hints_for_python_3.5+)
177 changes: 177 additions & 0 deletions doc/dev/test_proxy_migration_guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# Guide for migrating to the test proxy from vcrpy

This guide describes the changes that service SDKs should make to their test frameworks in order to take advantage of
the Azure SDK test proxy.

Documentation of the motivations and goals of the test proxy can be found [here][general_docs] in the azure-sdk-tools
GitHub repository, and documentation of how to set up and use the proxy can be found [here][detailed_docs].

## Update existing tests

### Current test structure

Test classes currently inherit from AzureTestCase, and test methods can optionally use decorators:

```py
from devtools_testutils import AzureTestCase

class TestExample(AzureTestCase):

def test_example(self):
...

@ExamplePreparer()
def test_example_with_preparer(self):
...
```

### New test structure

To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a
RecordedByProxy decorator:

```py
from devtools_testutils import AzureRecordedTestCase, RecordedByProxy

class TestExample(AzureRecordedTestCase):

@RecordedByProxy
def test_example(self):
...

@ExamplePreparer()
@RecordedByProxy
def test_example_with_preparer(self):
...
```

For async tests, import the RecordedByProxyAsync decorator from `devtools_testutils.aio` and use it in the same
way as RecordedByProxy.

> **Note:** since AzureRecordedTestCase doesn't inherit from `unittest.TestCase`, test class names need to start
> with "Test" in order to be properly collected by pytest by default. For more information, please refer to
> [pytest's documentation][pytest_collection].
## Run the tests

### Perform one-time setup

The test proxy is made available for your tests via a Docker container. Some tests require an SSL connection to work, so
the Docker image used for the container has a certificate imported that you need to trust on your machine. Instructions
on how to do so can be found [here][proxy_cert_docs].

### Start the proxy server

There is a [PowerShell script][docker_start_proxy] in `eng/common/testproxy` that will fetch the proxy Docker image if
you don't already have it, and will start or stop a container running the image for you. You can run the following
command from the root of the `azure-sdk-for-python` directory to start the container whenever you want to make the test
proxy available for running tests:

```powershell
.\eng\common\testproxy\docker-start-proxy.ps1 "start"
```

Note that the proxy is available as long as the container is running. In other words, you don't need to start and
stop the container for each test run or between tests for different SDKs. You can run the above command in the morning
and just stop the container whenever you'd like. To stop the container, run the same command but with `"stop"` in place
of `"start"`. In the future, the proxy container will be set up and started automatically when tests are run, and
starting it manually will be optional.

For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].

### Record or play back tests

Configuring live and playback tests is done with the `AZURE_TEST_RUN_LIVE` environment variable. When this variable is
set to "true" or "yes", live tests will run and produce recordings. When this variable is set to "false" or "no", or
not set at all, tests will run in playback mode and attempt to match existing recordings.

Recordings for a given package will end up in that package's `/tests/recordings` directory, just like they currently
do.

> **Note:** at this time, support for configuring live or playback tests with a `testsettings_local.cfg` file has been
> deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
### Register sanitizers

Since the test proxy doesn't use [`vcrpy`][vcrpy], tests don't use a scrubber to sanitize values in recordings.
Instead, sanitizers (as well as matchers and transforms) can be registered on the proxy as detailed in
[this][sanitizers] section of the proxy documentation. At the time of writing, sanitizers can be registered via the
`add_sanitizer` method in `devtools_testutils`.

Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that
are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to
[pytest's scoped fixture documentation][pytest_fixtures] for more details.

For example, to sanitize URIs in recordings, you can set up a URI sanitizer for all tests in the pytest session by
adding something like the following in the package's `conftest.py` file:

```python
from devtools_testutils import add_sanitizer

# autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
@pytest.fixture(scope="session", autouse=True)
def sanitize_uris():
add_sanitizer(ProxyRecordingSanitizer.URI, value="fakeendpoint")
```

`add_sanitizer` accepts a sanitizer, matcher, or transform type from the ProxyRecordingSanitizer enum as a required
parameter. Keyword-only arguments can be provided to customize the sanitizer; for example, in the snippet above, any
request URIs that match the default URI regular expression will have their domain name replaced with "fakeendpoint". A
request made to `https://tableaccount.table.core.windows.net` will be recorded as being made to
`https://fakeendpoint.table.core.windows.net`.

## Implementation details

### What does the test proxy do?

The gist of the test proxy is that it stands in between your tests and the service. What this means is that test
requests which would usually go straight to the service should instead point to the locally-hosted test proxy.

For example, if an operation would typically make a GET request to
`https://fakeazsdktestaccount.table.core.windows.net/Tables`, that operation should now be sent to
`https://localhost:5001/Tables` instead. The original endpoint should be stored in an `x-recording-upstream-base-uri` --
the proxy will send the original request and record the result.

The RecordedByProxy and RecordedByProxyAsync decorators patch test requests to do this for you.

### How does the test proxy know when and what to record or play back?

This is achieved by making POST requests to the proxy server that say whether to start or stop recording or playing
back, as well as what test is being run.

To start recording a test, the server should be primed with a POST request:

```
URL: https://localhost:5001/record/start
headers {
"x-recording-file": "<path-to-test>/recordings/<testfile>.<testname>"
}
```

This will return a recording ID in an `x-recording-id` header. This ID should be sent as an `x-recording-id` header in
all further requests during the test.

After the test has finished, a POST request should be sent to indicate that recording is complete:

```
URL: https://localhost:5001/record/stop
headers {
"x-recording-id": "<x-recording-id>"
}
```

Running tests in playback follows the same pattern, except that requests will be sent to `/playback/start` and
`/playback/stop` instead. A header, `x-recording-mode`, should be set to `record` for all requests when recording and
`playback` when playing recordings back. More details can be found [here][detailed_docs].

The RecordedByProxy and RecordedByProxyAsync decorators send the appropriate requests at the start and end of each test
case.

[detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
[docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
[proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md
[pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
[pytest_fixtures]: https://docs.pytest.org/latest/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session
[sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitiziers-and-matchers
[vcrpy]: https://vcrpy.readthedocs.io
Loading

0 comments on commit eb44d19

Please sign in to comment.