-
Notifications
You must be signed in to change notification settings - Fork 207
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
initial bulk publish impl for java (#789)
* initial bulk publish impl for java Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * add UTs and clean up java doc for new interface methods. Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * add more interface methods for bulk publish Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * adding examples and ITs for bulk publish Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * addressing review comments Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * use latest ref from dapr branch Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * add example validation Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix bindings example validation Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * make changes for latest bulk publish dapr changes Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix examples Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix examples Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix typo Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * test against java 11 only Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * change to latest dapr commit Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * run only pubsub IT, upload failsafe reports as run artifact Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix checkstyle Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix IT report upload condition Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix compile issues Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix spotbugs issue Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * run pubsubIT only Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * change upload artifact name for IT Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix tests Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * introduce sleep Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * test bulk publish with redis Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * change longvalues test to kafka Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * change bulk pub to kafka and revert long values changes Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * remove kafka pubsub from pubsub IT Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * change match order in examples Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * set fail fast as false Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix Internal Invoke exception in ITs Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * address review comments Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix IT Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix app scopes in examples Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * add content to daprdocs Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * address review comments Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix mm.py step comment Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * reset bindings examples readme Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> * fix example, IT and make classes immutable Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com> Co-authored-by: Artur Souza <asouza.pro@gmail.com>
- Loading branch information
1 parent
eb8565c
commit 81591b9
Showing
32 changed files
with
1,942 additions
and
50 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
101 changes: 101 additions & 0 deletions
101
examples/src/main/java/io/dapr/examples/pubsub/BulkPublisher.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| /* | ||
| * Copyright 2023 The Dapr Authors | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package io.dapr.examples.pubsub; | ||
|
|
||
| import io.dapr.client.DaprClient; | ||
| import io.dapr.client.DaprClientBuilder; | ||
| import io.dapr.client.DaprPreviewClient; | ||
| import io.dapr.client.domain.BulkPublishResponse; | ||
| import io.dapr.client.domain.BulkPublishResponseFailedEntry; | ||
| import io.dapr.examples.OpenTelemetryConfig; | ||
| import io.opentelemetry.api.OpenTelemetry; | ||
| import io.opentelemetry.api.trace.Span; | ||
| import io.opentelemetry.api.trace.Tracer; | ||
| import io.opentelemetry.context.Scope; | ||
| import io.opentelemetry.sdk.OpenTelemetrySdk; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.List; | ||
|
|
||
| import static io.dapr.examples.OpenTelemetryConfig.getReactorContext; | ||
|
|
||
| /** | ||
| * Message publisher. | ||
| * 1. Build and install jars: | ||
| * mvn clean install | ||
| * 2. cd [repo root]/examples | ||
| * 3. Run the program: | ||
| * dapr run --components-path ./components/pubsub --app-id bulk-publisher -- \ | ||
| * java -Ddapr.grpc.port="50010" -jar target/dapr-java-sdk-examples-exec.jar io.dapr.examples.pubsub.BulkPublisher | ||
| */ | ||
| public class BulkPublisher { | ||
|
|
||
| private static final int NUM_MESSAGES = 10; | ||
|
|
||
| private static final String TOPIC_NAME = "bulkpublishtesting"; | ||
|
|
||
| //The name of the pubsub | ||
| private static final String PUBSUB_NAME = "messagebus"; | ||
|
|
||
| /** | ||
| * main method. | ||
| * | ||
| * @param args incoming args | ||
| * @throws Exception any exception | ||
| */ | ||
| public static void main(String[] args) throws Exception { | ||
| OpenTelemetry openTelemetry = OpenTelemetryConfig.createOpenTelemetry(); | ||
| Tracer tracer = openTelemetry.getTracer(BulkPublisher.class.getCanonicalName()); | ||
| Span span = tracer.spanBuilder("Bulk Publisher's Main").setSpanKind(Span.Kind.CLIENT).startSpan(); | ||
| try (DaprPreviewClient client = (new DaprClientBuilder()).buildPreviewClient()) { | ||
| DaprClient c = (DaprClient) client; | ||
| c.waitForSidecar(10000); | ||
| try (Scope scope = span.makeCurrent()) { | ||
| System.out.println("Using preview client..."); | ||
| List<String> messages = new ArrayList<>(); | ||
| System.out.println("Constructing the list of messages to publish"); | ||
| for (int i = 0; i < NUM_MESSAGES; i++) { | ||
| String message = String.format("This is message #%d", i); | ||
| messages.add(message); | ||
| System.out.println("Going to publish message : " + message); | ||
| } | ||
| BulkPublishResponse<?> res = client.publishEvents(PUBSUB_NAME, TOPIC_NAME, "text/plain", messages) | ||
| .subscriberContext(getReactorContext()).block(); | ||
| System.out.println("Published the set of messages in a single call to Dapr"); | ||
| if (res != null) { | ||
| if (res.getFailedEntries().size() > 0) { | ||
| // Ideally this condition will not happen in examples | ||
| System.out.println("Some events failed to be published"); | ||
| for (BulkPublishResponseFailedEntry<?> entry : res.getFailedEntries()) { | ||
| System.out.println("EntryId : " + entry.getEntry().getEntryId() | ||
| + " Error message : " + entry.getErrorMessage()); | ||
| } | ||
| } | ||
| } else { | ||
| throw new Exception("null response from dapr"); | ||
| } | ||
| } | ||
| // Close the span. | ||
|
|
||
| span.end(); | ||
| // Allow plenty of time for Dapr to export all relevant spans to the tracing infra. | ||
| Thread.sleep(10000); | ||
| // Shutdown the OpenTelemetry tracer. | ||
| OpenTelemetrySdk.getGlobalTracerManagement().shutdown(); | ||
|
|
||
| System.out.println("Done"); | ||
| } | ||
| } | ||
| } | ||
|
|
94 changes: 94 additions & 0 deletions
94
examples/src/main/java/io/dapr/examples/pubsub/CloudEventBulkPublisher.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| /* | ||
| * Copyright 2023 The Dapr Authors | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| */ | ||
|
|
||
| package io.dapr.examples.pubsub; | ||
|
|
||
| import io.dapr.client.DaprClientBuilder; | ||
| import io.dapr.client.DaprPreviewClient; | ||
| import io.dapr.client.domain.BulkPublishEntry; | ||
| import io.dapr.client.domain.BulkPublishRequest; | ||
| import io.dapr.client.domain.BulkPublishResponse; | ||
| import io.dapr.client.domain.BulkPublishResponseFailedEntry; | ||
| import io.dapr.client.domain.CloudEvent; | ||
|
|
||
| import java.util.ArrayList; | ||
| import java.util.HashMap; | ||
| import java.util.List; | ||
| import java.util.Map; | ||
| import java.util.UUID; | ||
|
|
||
| /** | ||
| * Message publisher. | ||
| * 1. Build and install jars: | ||
| * mvn clean install | ||
| * 2. cd [repo root]/examples | ||
| * 3. Run the program: | ||
| * dapr run --components-path ./components/pubsub --app-id publisher -- \ | ||
| * java -Ddapr.grpc.port="50010" \ | ||
| * -jar target/dapr-java-sdk-examples-exec.jar io.dapr.examples.pubsub.CloudEventBulkPublisher | ||
| */ | ||
| public class CloudEventBulkPublisher { | ||
|
|
||
| private static final int NUM_MESSAGES = 10; | ||
|
|
||
| private static final String TOPIC_NAME = "bulkpublishtesting"; | ||
|
|
||
| //The name of the pubsub | ||
| private static final String PUBSUB_NAME = "messagebus"; | ||
|
|
||
| /** | ||
| * main method. | ||
| * | ||
| * @param args incoming args | ||
| * @throws Exception any exception | ||
| */ | ||
| public static void main(String[] args) throws Exception { | ||
| try (DaprPreviewClient client = (new DaprClientBuilder()).buildPreviewClient()) { | ||
| System.out.println("Using preview client..."); | ||
| List<BulkPublishEntry<CloudEvent<Map<String, String>>>> entries = new ArrayList<>(); | ||
| for (int i = 0; i < NUM_MESSAGES; i++) { | ||
| CloudEvent<Map<String, String>> cloudEvent = new CloudEvent<>(); | ||
| cloudEvent.setId(UUID.randomUUID().toString()); | ||
| cloudEvent.setType("example"); | ||
| cloudEvent.setSpecversion("1"); | ||
| cloudEvent.setDatacontenttype("application/json"); | ||
| String val = String.format("This is message #%d", i); | ||
| cloudEvent.setData(new HashMap<>() { | ||
| { | ||
| put("dataKey", val); | ||
| } | ||
| }); | ||
| BulkPublishEntry<CloudEvent<Map<String, String>>> entry = new BulkPublishEntry<>( | ||
| "" + (i + 1), cloudEvent, CloudEvent.CONTENT_TYPE, null); | ||
| entries.add(entry); | ||
| } | ||
| BulkPublishRequest<CloudEvent<Map<String, String>>> request = new BulkPublishRequest<>(PUBSUB_NAME, TOPIC_NAME, | ||
| entries); | ||
| BulkPublishResponse<?> res = client.publishEvents(request).block(); | ||
| if (res != null) { | ||
| if (res.getFailedEntries().size() > 0) { | ||
| // Ideally this condition will not happen in examples | ||
| System.out.println("Some events failed to be published"); | ||
| for (BulkPublishResponseFailedEntry<?> entry : res.getFailedEntries()) { | ||
| System.out.println("EntryId : " + entry.getEntry().getEntryId() | ||
| + " Error message : " + entry.getErrorMessage()); | ||
| } | ||
| } | ||
| } else { | ||
| throw new Exception("null response"); | ||
| } | ||
| System.out.println("Done"); | ||
| } | ||
| } | ||
| } | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.