Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eg sync stack testproxy #36656

Merged
merged 47 commits into from
Oct 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
130895c
Add missing sync test cases
billwert Jun 12, 2023
5dc3fca
Enable sync-stack
billwert Jun 23, 2023
2daff85
Remove extra set of aeg-channel-name.
billwert Jun 23, 2023
6a0874a
enable AssertingHttpClient
billwert Jun 23, 2023
b80c1a8
add back missing Context arguments
billwert Jun 29, 2023
043e39a
Don't stop trying dev credentials on failures
billwert Jul 14, 2023
dbc2b6c
Tests now use test proxy and recordings are pushed to assets repo
billwert Sep 6, 2023
18dd529
resolve merge conflicts
srnagar Sep 14, 2023
f9ed9a0
address pr comments
srnagar Sep 14, 2023
64f5e98
fix merge conflicts
srnagar Sep 14, 2023
1b35edc
add exports
srnagar Sep 15, 2023
dcf148a
fix version
srnagar Sep 15, 2023
c6a2bb3
remove unused imports
srnagar Sep 15, 2023
26cc35a
use unreleased test version
srnagar Sep 19, 2023
dc41e49
fix tag
srnagar Sep 19, 2023
fb20989
fix tag
srnagar Sep 19, 2023
5560758
refactor
srnagar Sep 20, 2023
44e43fd
fix spring test after sync stack migration
srnagar Sep 20, 2023
1d2d820
log the test proxy response on error
srnagar Sep 20, 2023
48911db
fix header sanitizer in Service Bus
srnagar Sep 22, 2023
edbe426
fix regex for quantum job tests
srnagar Sep 24, 2023
f4d8493
Add missing sync test cases
billwert Jun 12, 2023
503813c
Enable sync-stack
billwert Jun 23, 2023
a1f0197
Remove extra set of aeg-channel-name.
billwert Jun 23, 2023
5087c06
enable AssertingHttpClient
billwert Jun 23, 2023
b224700
add back missing Context arguments
billwert Jun 29, 2023
e9a00d1
Tests now use test proxy and recordings are pushed to assets repo
billwert Sep 6, 2023
9b5a6eb
address pr comments
srnagar Sep 14, 2023
e01a753
add exports
srnagar Sep 15, 2023
7ec717e
fix version
srnagar Sep 15, 2023
67c932a
use unreleased test version
srnagar Sep 19, 2023
4cffdee
fix tag
srnagar Sep 19, 2023
66939e8
fix tag
srnagar Sep 19, 2023
a7ca1f7
refactor
srnagar Sep 20, 2023
799c12a
fix spring test after sync stack migration
srnagar Sep 20, 2023
507db27
log the test proxy response on error
srnagar Sep 20, 2023
baaf27d
fix header sanitizer in Service Bus
srnagar Sep 22, 2023
5ad9038
fix regex for quantum job tests
srnagar Sep 24, 2023
b29abdd
Put a semaphore around starting the proxy.
billwert Sep 25, 2023
531a43c
make the methods of `TestProxyManager` static and synchronized.
billwert Sep 27, 2023
189b6a3
Use the working directory to search for the root of the repo from.
billwert Sep 29, 2023
7d2ed33
add missing sanitizer setup
billwert Oct 6, 2023
3b8518e
base class has this so it's not needed here
billwert Oct 6, 2023
ceb9d57
removed this in merge accidentally
billwert Oct 6, 2023
10da9a6
resolve merge conflicts
srnagar Oct 9, 2023
6dc76f6
Merge branch 'main' into eg-sync-stack-testproxy
srnagar Oct 10, 2023
dca5125
pin azure-core version
srnagar Oct 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions eng/versioning/version_client.txt
Original file line number Diff line number Diff line change
Expand Up @@ -441,6 +441,7 @@ com.azure.tools:azure-sdk-build-tool;1.0.0;1.1.0-beta.1
# note: The unreleased dependencies will not be manipulated with the automatic PR creation code.
# In the pom, the version update tag after the version should name the unreleased package and the dependency version:
# <!-- {x-version-update;unreleased_com.azure:azure-core;dependency} -->
unreleased_com.azure:azure-messaging-eventgrid;4.19.0-beta.1

# Released Beta dependencies: Copy the entry from above, prepend "beta_", remove the current
# version and set the version to the released beta. Released beta dependencies are only valid
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,9 @@

import com.azure.core.test.utils.TestProxyManager;
import com.azure.core.util.logging.ClientLogger;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.TestInfo;

import java.nio.file.Path;
import java.nio.file.Paths;

import static com.azure.core.test.utils.TestUtils.toURI;

/**
* Base class for running live and playback tests using test-proxy
*/
Expand All @@ -30,8 +24,6 @@ public TestProxyTestBase() {
super();
}

private static TestProxyManager testProxyManager;

/**
* Before tests are executed, determines the test mode by reading the {@code AZURE_TEST_MODE} environment variable.
* If it is not set, {@link TestMode#PLAYBACK}
Expand All @@ -40,20 +32,8 @@ public TestProxyTestBase() {
@BeforeAll
public static void setupTestProxy(TestInfo testInfo) {
testMode = initializeTestMode();
Path testClassPath = Paths.get(toURI(testInfo.getTestClass().get().getResource(testInfo.getTestClass().get().getSimpleName() + ".class")));
if (isTestProxyEnabled() && (testMode == TestMode.PLAYBACK || testMode == TestMode.RECORD)) {
testProxyManager = new TestProxyManager(testClassPath);
testProxyManager.startProxy();
}
}

/**
* Performs cleanup actions after all tests are executed.
*/
@AfterAll
public static void teardownTestProxy() {
if (testProxyManager != null) {
testProxyManager.stopProxy();
TestProxyManager.startProxy();
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import com.azure.core.test.utils.HttpURLConnectionHttpClient;
import com.azure.core.test.utils.TestProxyUtils;
import com.azure.core.util.Context;
import com.azure.core.util.logging.ClientLogger;
import com.azure.core.util.serializer.JacksonAdapter;
import com.azure.core.util.serializer.SerializerAdapter;
import com.azure.core.util.serializer.SerializerEncoding;
Expand All @@ -32,6 +33,7 @@
import java.util.List;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.TimeUnit;

import static com.azure.core.test.implementation.TestingHelpers.X_RECORDING_FILE_LOCATION;
import static com.azure.core.test.implementation.TestingHelpers.X_RECORDING_ID;
Expand All @@ -46,6 +48,7 @@
*/
public class TestProxyPlaybackClient implements HttpClient {

private static final ClientLogger LOGGER = new ClientLogger(TestProxyPlaybackClient.class);
private final HttpClient client;
private final URL proxyUrl;
private String xRecordingId;
Expand Down Expand Up @@ -89,11 +92,7 @@ public Queue<String> startPlayback(File recordFile, Path testClassPath) {
SerializerEncoding.JSON))
.setHeader(HttpHeaderName.ACCEPT, "application/json")
.setHeader(HttpHeaderName.CONTENT_TYPE, "application/json");
} catch (IOException e) {
throw new RuntimeException(e);
}

try (HttpResponse response = client.sendSync(request, Context.NONE)) {
HttpResponse response = sendRequestWithRetries(request);
checkForTestProxyErrors(response);
xRecordingId = response.getHeaderValue(X_RECORDING_ID);
xRecordingFileLocation = new String(Base64.getUrlDecoder().decode(
Expand Down Expand Up @@ -127,13 +126,42 @@ public Queue<String> startPlayback(File recordFile, Path testClassPath) {
}
}

private HttpResponse sendRequestWithRetries(HttpRequest request) {
int retries = 0;
while (true) {
try {
HttpResponse response = client.sendSync(request, Context.NONE);
if (response.getStatusCode() / 100 != 2) {
throw new RuntimeException("Test proxy returned a non-successful status code. "
+ response.getStatusCode() + "; response: " + response.getBodyAsString().block());
}
return response;
} catch (Exception e) {
retries++;
if (retries >= 3) {
throw e;
}
sleep(1);
LOGGER.warning("Retrying request to test proxy. Retry attempt: " + retries);
}
}
}

private void sleep(int durationInSeconds) {
try {
TimeUnit.SECONDS.sleep(durationInSeconds);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}

/**
* Stops playback of a test recording.
*/
public void stopPlayback() {
HttpRequest request = new HttpRequest(HttpMethod.POST, proxyUrl + "/playback/stop")
.setHeader(X_RECORDING_ID, xRecordingId);
client.sendSync(request, Context.NONE).close();
sendRequestWithRetries(request);
}

/**
Expand Down Expand Up @@ -210,7 +238,7 @@ public void addMatcherRequests(List<TestProxyRequestMatcher> matchers) {
}
matcherRequests.forEach(request -> {
request.setHeader(X_RECORDING_ID, xRecordingId);
client.sendSync(request, Context.NONE).close();
sendRequestWithRetries(request);
});
} else {
this.matchers.addAll(matchers);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,37 +24,34 @@
/**
* Manages running the test recording proxy server
*/
public class TestProxyManager {
public final class TestProxyManager {
private static final ClientLogger LOGGER = new ClientLogger(TestProxyManager.class);
private Process proxy;
private final Path testClassPath;
private static Process proxy;
private static final Path WORKING_DIRECTORY = Paths.get(System.getProperty("user.dir"));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this running in IntelliJ as well as mvn from both the SDK directory and the root of the repo using -pl.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think user.dir is consistent.
https://stackoverflow.com/questions/16239130/java-user-dir-property-what-exactly-does-it-mean
We should avoid hard coding that path here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is consistent enough for our purposes - java.exe (through Maven typically) will be started in some path in the repo, which is all we need. My goal is to stop using the target dir here as it isn't something we need in this class so plumbing it in isn't necessary. Happy to hear of another alternative.


/**
* Construct a {@link TestProxyManager} for controlling the external test proxy.
* @param testClassPath the test class path
*/
public TestProxyManager(Path testClassPath) {
this.testClassPath = testClassPath;
// This is necessary to stop the proxy when the debugger is stopped.
Runtime.getRuntime().addShutdownHook(new Thread(this::stopProxy));
static {
Runtime.getRuntime().addShutdownHook(new Thread(TestProxyManager::stopProxy));
if (runningLocally()) {
TestProxyDownloader.installTestProxy(testClassPath);
TestProxyDownloader.installTestProxy(WORKING_DIRECTORY);
}
}

@Deprecated
private TestProxyManager() { }

/**
* Start an instance of the test proxy.
* @throws UncheckedIOException There was an issue communicating with the proxy.
* @throws RuntimeException There was an issue starting the proxy process.
*/
public void startProxy() {
public static synchronized void startProxy() {
try {
// if we're not running in CI we will check to see if someone has started the proxy, and start one if not.
if (runningLocally() && !checkAlive(1, Duration.ofSeconds(1), null)) {
String commandLine = Paths.get(TestProxyDownloader.getProxyDirectory().toString(),
TestProxyUtils.getProxyProcessName()).toString();

Path repoRoot = TestUtils.getRepoRootResolveUntil(testClassPath, "eng");
Path repoRoot = TestUtils.getRepoRootResolveUntil(WORKING_DIRECTORY, "eng");

// Resolve the path to the repo root 'target' folder and create the folder if it doesn't exist.
// This folder will be used to store the 'test-proxy.log' file to enable simpler debugging of Test Proxy
Expand Down Expand Up @@ -128,7 +125,7 @@ private static boolean checkAlive(int loops, Duration waitTime, Process proxy) t
/**
* Stop the running instance of the test proxy.
*/
public void stopProxy() {
private static void stopProxy() {
if (proxy != null && proxy.isAlive()) {
proxy.destroy();
}
Expand All @@ -138,7 +135,7 @@ public void stopProxy() {
* Checks the environment variables commonly set in CI to determine if the run is local.
* @return True if the run is local.
*/
private boolean runningLocally() {
private static boolean runningLocally() {
return Configuration.getGlobalConfiguration().get("TF_BUILD") == null
&& Configuration.getGlobalConfiguration().get("CI") == null;
}
Expand Down
2 changes: 1 addition & 1 deletion sdk/eventgrid/azure-messaging-eventgrid/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
### Other Changes

#### Dependency Updates
-

- Upgraded `azure-core` from `1.42.0` to version `1.43.0`.
- Upgraded `azure-core-http-netty` from `1.13.6` to version `1.13.7`.

Expand Down
6 changes: 6 additions & 0 deletions sdk/eventgrid/azure-messaging-eventgrid/assets.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "java",
"TagPrefix": "java/eventgrid/azure-messaging-eventgrid",
"Tag": "java/eventgrid/azure-messaging-eventgrid_7bb36a1579"
}
2 changes: 2 additions & 0 deletions sdk/eventgrid/azure-messaging-eventgrid/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@
--add-opens com.azure.messaging.eventgrid/com.azure.messaging.eventgrid.implementation=ALL-UNNAMED

--add-reads com.azure.messaging.eventgrid=com.azure.core.serializer.json.jackson
--add-exports com.azure.core/com.azure.core.implementation.util=ALL-UNNAMED
--add-exports com.azure.core/com.azure.core.implementation=ALL-UNNAMED
</javaModulesSurefireArgLine>
</properties>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,11 @@
import com.azure.core.annotation.ServiceMethod;
import com.azure.core.credential.AzureKeyCredential;
import com.azure.core.credential.AzureSasCredential;
import com.azure.core.http.HttpHeaders;
import com.azure.core.http.HttpPipeline;
import com.azure.core.http.policy.AddHeadersFromContextPolicy;
import com.azure.core.http.rest.Response;
import com.azure.core.models.CloudEvent;
import com.azure.core.util.BinaryData;
import com.azure.core.util.Context;
import com.azure.core.util.CoreUtils;
import com.azure.core.util.logging.ClientLogger;
import com.azure.core.util.tracing.Tracer;
import com.azure.messaging.eventgrid.implementation.Constants;
Expand All @@ -39,7 +36,6 @@
import java.util.Base64;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Objects;

import static com.azure.core.util.FluxUtil.monoError;
Expand Down Expand Up @@ -155,7 +151,6 @@
@ServiceClient(builder = EventGridPublisherClientBuilder.class, isAsync = true)
public final class EventGridPublisherAsyncClient<T> {

private static final String PARTNER_CHANNEL_HEADER_NAME = "aeg-channel-name";
private final String hostname;

private final EventGridPublisherClientImpl impl;
Expand Down Expand Up @@ -312,25 +307,9 @@ Mono<Response<Void>> sendEventsWithResponse(Iterable<T> events, String channelNa
if (context == null) {
context = Context.NONE;
}
if (!CoreUtils.isNullOrEmpty(channelName)) {
String requestHttpHeadersKey = AddHeadersFromContextPolicy.AZURE_REQUEST_HTTP_HEADERS_KEY;
Map<Object, Object> keyValues = context.getValues();
if (keyValues != null && keyValues.containsKey(requestHttpHeadersKey)) {
// if the given Context instance already contains custom headers,
// add partner channel header to HttpHeaders
Object value = keyValues.get(requestHttpHeadersKey);
if (value instanceof HttpHeaders) {
HttpHeaders headers = (HttpHeaders) value;
headers.add(PARTNER_CHANNEL_HEADER_NAME, channelName);
}
} else {
context = context.addData(requestHttpHeadersKey,
new HttpHeaders().add(PARTNER_CHANNEL_HEADER_NAME, channelName));
}
}

if (this.eventClass == CloudEvent.class) {
return this.sendCloudEventsWithResponse((Iterable<CloudEvent>) events, context);
return this.sendCloudEventsWithResponse((Iterable<CloudEvent>) events, channelName, context);
} else if (this.eventClass == EventGridEvent.class) {
return this.sendEventGridEventsWithResponse((Iterable<EventGridEvent>) events, context);
} else {
Expand Down Expand Up @@ -395,15 +374,15 @@ Mono<Response<Void>> sendEventGridEventsWithResponse(Iterable<EventGridEvent> ev
.flatMap(list -> this.impl.publishEventGridEventsWithResponseAsync(this.hostname, list, finalContext));
}

Mono<Response<Void>> sendCloudEventsWithResponse(Iterable<CloudEvent> events, Context context) {
Mono<Response<Void>> sendCloudEventsWithResponse(Iterable<CloudEvent> events, String channelName, Context context) {
if (events == null) {
return monoError(logger, new NullPointerException("'events' cannot be null."));
}
final Context finalContext = context != null ? context : Context.NONE;
this.addCloudEventTracePlaceHolder(events);
return Flux.fromIterable(events)
.collectList()
.flatMap(list -> this.impl.publishCloudEventEventsWithResponseAsync(this.hostname, list, null, finalContext));
.flatMap(list -> this.impl.publishCloudEventEventsWithResponseAsync(this.hostname, list, channelName, finalContext));
}

Mono<Response<Void>> sendCustomEventsWithResponse(Iterable<BinaryData> events, Context context) {
Expand Down
Loading