Support rocketmq #1412

JoeKerouac · 2024-01-31T12:10:23Z

support RocketMQ
see #1356

JoeKerouac · 2024-01-31T12:12:15Z

Signed-off-by: Adrian Cole <adrian@tetrate.io>

codefromthecrypt · 2024-02-01T01:16:35Z

ok I re-pushed into a single commit as we squash on merge anyway, and already spent the work to rebase this on master.

NEXT STEP:

The docker image for 5.1.4 is not multi-platform, so we need a multi-platform image or people like me cannot run docker tests (on apple silicon aka arm64). @JoeKerouac can you check on what is holding up apache/rocketmq-docker#98?

Signed-off-by: Adrian Cole <adrian@tetrate.io>

codefromthecrypt · 2024-02-01T01:35:16Z

so this is the current error until there's a multi-platform image. Our other images we make on own, because they are shared with a zipkin server transport. As there's no rocketmq server transport, this one is special case and we need to address the docker image in some way, at least upstream would be good to find out why not.

[INFO] Running brave.rocketmq.client.ITRocketMQTracingTest
08:31:48,591 WARN  [main] containers.GenericContainer (GenericContainer.java:488) - The architecture 'amd64' for image 'apache/rocketmq:5.1.4' (ID sha256:f14b643f5d86369b50362c0039938cf0346f451b559d95f4f2c530083f9e579f) does not match the Docker server architecture 'arm64'. This will cause the container to execute much more slowly due to emulation and may lead to timeout failures.
[ERROR] Tests run: 6, Failures: 1, Errors: 5, Skipped: 0, Time elapsed: 34.80 s <<< FAILURE! -- in brave.rocketmq.client.ITRocketMQTracingTest
[ERROR] brave.rocketmq.client.ITRocketMQTracingTest.tracingMessageListenerConcurrently -- Time elapsed: 6.291 s <<< ERROR!

codefromthecrypt · 2024-02-01T01:42:36Z

@anuraaga @reta @shakuzen @jeqo I think this is a good example of us needing to add the new apple silicon runners and use them in the docker tests. If anyone feels up to task of github action slinging here and otherwise, please do! (can do an org search for testcontainers, but should be mostly zipkin, -dependencies, -reporter, here, brave-cassandra and the contrib storage things)

codefromthecrypt · 2024-02-01T02:39:10Z

nevermind about CI catching this as there's not yet a free runner that works with arm64 and docker https://github.com/orgs/community/discussions/69211

JoeKerouac · 2024-02-02T08:07:23Z

ok I re-pushed into a single commit as we squash on merge anyway, and already spent the work to rebase this on master.

NEXT STEP:

The docker image for 5.1.4 is not multi-platform, so we need a multi-platform image or people like me cannot run docker tests (on apple silicon aka arm64). @JoeKerouac can you check on what is holding up apache/rocketmq-docker#98?

I will do my best to resolve this issue; however, it may not be straightforward. I need some time, and there might not be a definite deadline for it.

JoeKerouac · 2024-02-02T08:10:15Z

However, regarding the current test case failure, I will address it as soon as possible.

JoeKerouac · 2024-02-04T15:42:39Z

@codefromthecrypt I don't know why the RocketMQ version in my code was upgraded from 4.6.0 to 5.1.4. Since there are significant differences between 5.1.4 and 4.x, the current goal is RocketMQ 4.6.0 instead of 5.1.4.

codefromthecrypt · 2024-02-04T23:54:05Z

@JoeKerouac I don't think we have ever not used a current version as the first version of instrumentation. Can you mention why there is an issue with 5.x or why 5.x is not desired? When we don't use current versions, we get security (CVE) problems to deal with, and also usually eventually have to support latest anyway. We don't likely want to have something both for rocket 4.x and 5.x

codefromthecrypt · 2024-02-04T23:55:21Z

for example, old and new dubbo caused a lot of work for the project. Not saying this is like dubbo, but dubbo was one of the most high maintenance and also slowest tests in the project. We don't want to repeat something like we had when we had 2 dubbos

JoeKerouac · 2024-02-06T05:33:40Z

@codefromthecrypt Since that's the case, I'll go ahead and upgrade to version 5.1.4; it's already at version 5.1.4 now.

codefromthecrypt · 2024-02-06T08:48:49Z

thanks @JoeKerouac. If there's a compatability issue with 5 against 4, we can add a maven-invoker-plugin test, but if it should work, let's just stick with 5.x.

instrumentation/rocketmq-client/src/test/java/brave/rocketmq/client/ITRocketMQTracingTest.java

codefromthecrypt

getting closer! After these changes will need another pass, and then I will start asking for and suggesting non IT unit tests, using examples from kafka or jms where we have them

brave-bom/pom.xml

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTracing.java

.../rocketmq-client/src/main/java/brave/rocketmq/client/TracingMessageListenerConcurrently.java

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/Util.java

codefromthecrypt · 2024-02-08T01:59:47Z

nice to see passing tests @JoeKerouac!

JoeKerouac · 2024-02-20T05:57:31Z

Sorry, I just finished the Chinese New Year holiday. I will handle it as soon as possible.

JoeKerouac · 2024-02-26T14:41:32Z

@codefromthecrypt I have completed all the modifications suggested during the code review.

codefromthecrypt

Thanks for the progress! Besides notes mentioned, please audit the other messaging libraries and look for IT tests you may have missed. For example, custom sampling and continuing a trace. You can look at least at kafka-clients to see what is left out.

Also, has there been movement on a multi-platform image? Once you get the current docker setup simpler (ideally no custom shell script), we can look into making our own image, as we can't release this until it can run on arm64.

codefromthecrypt · 2024-02-26T23:44:49Z

instrumentation/rocketmq-client/src/test/resources/start.sh

+}
+
+function createTopic() {
+    sh /home/rocketmq/rocketmq-5.1.4/bin/mqadmin updateTopic -n 127.0.0.1:9876 -b 127.0.0.1:10911 -t $1


is there no way to do this in java?

codefromthecrypt · 2024-02-26T23:47:39Z

instrumentation/rocketmq-client/src/test/resources/start.sh

+sh  /home/rocketmq/rocketmq-5.1.4/bin/mqbroker -n 127.0.0.1:9876 -c /broker.conf > ~/broker.log 2>&1 &
+check "boot success" ~/broker.log
+
+createTopic testSend


the better way is to use less shell scripting and set these up in java, even if java uses a shell command to create the topic. Then, there is less maintenance.

Here's an example https://github.com/openzipkin/zipkin-reporter-java/blob/master/amqp-client/src/test/java/zipkin2/reporter/amqp/RabbitMQContainer.java#L44-L55

codefromthecrypt · 2024-02-26T23:47:56Z

instrumentation/rocketmq-client/src/test/resources/start.sh

+createTopic tracingMessageListenerOrderly
+createTopic testAll
+
+echo "--JoeKerouac--"


this is cute but we don't want to echo individuals in the logs ;)

codefromthecrypt · 2024-02-26T23:48:44Z

instrumentation/rocketmq-client/src/test/resources/start.sh

+
+echo "--JoeKerouac--"
+
+while true


seems odd that the container doesn't itself stay alive. Maybe when topic setup is moved to java/testcontainers side, you don't need this anymore?

codefromthecrypt · 2024-02-26T23:49:32Z

instrumentation/rocketmq-client/src/test/resources/broker.conf

@@ -0,0 +1,3 @@
+brokerName=JoeKerouac-Test


use java to setup these things and leave the broker to default or just "test" or something because it is in a container so there should be no conflict.

codefromthecrypt · 2024-02-26T23:56:04Z

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTags.java

+
+  // TODO: maybe like HttpTags.PATH if we support extended tags on first version
+  public static final String ROCKETMQ_TAGS = "rocketmq.tags";
+  public static final String ROCKETMQ_TOPIC = "rocketmq.topic";


delete this file as we will only start with rocketmq.topic

codefromthecrypt · 2024-02-26T23:56:53Z

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTracing.java

+import java.util.Map;
+
+public class RocketMQTracing {
+


note this won't be public, and this is like jms, rabbit, etc.

Suggested change

static final String ROCKETMQ_TOPIC = "rocketmq.topic";

codefromthecrypt · 2024-02-26T23:57:26Z

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTracing.java

+
+import java.util.Map;
+
+public class RocketMQTracing {


Suggested change

public class RocketMQTracing {

/**

* Use this class to decorate your RocketMQ consumer / producer and enable Tracing.

*

* @since 6.1

*/

public final class RocketMQTracing {

codefromthecrypt · 2024-02-26T23:58:08Z

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTracing.java

+  private static final int defaultDelayLevelWhenNextConsume = 0;
+
+  public static RocketMQTracing create(Tracing tracing) {
+    return new RocketMQTracing(MessagingTracing.create(tracing), RocketMQTags.ROCKETMQ_SERVICE);


don't pass around this constant.. you'll see why, but it is simpler and less parameters

codefromthecrypt · 2024-02-26T23:59:29Z

instrumentation/rocketmq-client/src/main/java/brave/rocketmq/client/RocketMQTracing.java

+                                       String remoteServiceName) {
+    return new RocketMQTracing(messagingTracing, remoteServiceName);
+  }
+


Main thing is to be similar to other code, look at jms and you'll see this.. adopt that and adapt it for rabbit. This makes it simpler for configuration tools who don't need to look for different conventions depending on the library.

public static JmsTracing create(Tracing tracing) { return newBuilder(tracing).build(); } public static JmsTracing create(MessagingTracing messagingTracing) { return newBuilder(messagingTracing).build(); } public static Builder newBuilder(Tracing tracing) { return newBuilder(MessagingTracing.create(tracing)); } public static Builder newBuilder(MessagingTracing messagingTracing) { return new Builder(messagingTracing); } public static final class Builder { final MessagingTracing messagingTracing; String remoteServiceName = "jms"; Builder(MessagingTracing messagingTracing) {

codefromthecrypt · 2024-04-11T00:33:50Z

note: status is still the same since my last comments, except the license format has now changed and will need to be updated along with past comments here.

codefromthecrypt force-pushed the support_rocketmq branch 2 times, most recently from 01266d6 to 6c23605 Compare February 1, 2024 01:08

adds RocketMQTracing

35452f8

Signed-off-by: Adrian Cole <adrian@tetrate.io>

codefromthecrypt force-pushed the support_rocketmq branch from 6c23605 to 35452f8 Compare February 1, 2024 01:13

fix version and touch files to get 2024 license

d07e7a0

Signed-off-by: Adrian Cole <adrian@tetrate.io>

Downgrade the RocketMQ version from 5.1.4 to 4.6.0.

9e9d1ed

upgrade rocketMQ

9c72c70

fix todo

054034e

codefromthecrypt reviewed Feb 6, 2024

View reviewed changes

instrumentation/rocketmq-client/src/test/java/brave/rocketmq/client/ITRocketMQTracingTest.java Outdated Show resolved Hide resolved

fix test

13e3a09

codefromthecrypt reviewed Feb 8, 2024

View reviewed changes

fix code review

98f6d8a

codefromthecrypt reviewed Feb 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support rocketmq #1412

Support rocketmq #1412

JoeKerouac commented Jan 31, 2024

JoeKerouac commented Jan 31, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

JoeKerouac commented Feb 2, 2024

JoeKerouac commented Feb 2, 2024

JoeKerouac commented Feb 4, 2024

codefromthecrypt commented Feb 4, 2024

codefromthecrypt commented Feb 4, 2024

JoeKerouac commented Feb 6, 2024

codefromthecrypt commented Feb 6, 2024

codefromthecrypt left a comment

codefromthecrypt commented Feb 8, 2024

JoeKerouac commented Feb 20, 2024

JoeKerouac commented Feb 26, 2024

codefromthecrypt left a comment

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt Feb 26, 2024

codefromthecrypt commented Apr 11, 2024

-public class RocketMQTracing {
+/**
+ * Use this class to decorate your RocketMQ consumer / producer and enable Tracing.
+ *
+ * @since 6.1
+ */
+public final class RocketMQTracing {

Support rocketmq #1412

Are you sure you want to change the base?

Support rocketmq #1412

Conversation

JoeKerouac commented Jan 31, 2024

JoeKerouac commented Jan 31, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

codefromthecrypt commented Feb 1, 2024

JoeKerouac commented Feb 2, 2024

JoeKerouac commented Feb 2, 2024

JoeKerouac commented Feb 4, 2024

codefromthecrypt commented Feb 4, 2024

codefromthecrypt commented Feb 4, 2024

JoeKerouac commented Feb 6, 2024

codefromthecrypt commented Feb 6, 2024

codefromthecrypt left a comment

Choose a reason for hiding this comment

codefromthecrypt commented Feb 8, 2024

JoeKerouac commented Feb 20, 2024

JoeKerouac commented Feb 26, 2024

codefromthecrypt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codefromthecrypt commented Apr 11, 2024