KAFKA-14582: Move JmxTool to tools by fvaleri · Pull Request #13136 · apache/kafka

fvaleri · 2023-01-20T16:56:22Z

This PR is based on #13131.

Output example:

$ bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://:9990/jmxrmi \
  --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec \
  --attributes FifteenMinuteRate,FiveMinuteRate \
  --date-format "yyyyMMdd-hh:mm:ss" \
  --reporting-interval 1000 \
  --report-format tsv
Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://:9999/jmxrmi
time    20230120-06:23:14
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate    0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
time	20230120-06:23:15
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate	0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
time	20230120-06:23:16
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate	0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
^C

System test using it:

$ TC_PATHS="tests/kafkatest/tests/core/throttling_test.py::ThrottlingTest.test_throttled_reassignment" \
  _DUCKTAPE_OPTIONS='--parameters '\''{"bounce_brokers":"false"}'\' \
    bash tests/docker/run_tests.sh
docker exec ducker01 bash -c "cd /opt/kafka-dev && ducktape --cluster-file /opt/kafka-dev/tests/docker/build/cluster.json  ./tests/kafkatest/tests/core/throttling_test.py::ThrottlingTest.test_throttled_reassignment --parameters '{"bounce_brokers":"false"}'"
/usr/local/lib/python3.9/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[INFO:2023-01-21 09:21:39,367]: starting test run with session id 2023-01-21--017...
[INFO:2023-01-21 09:21:39,368]: running 1 tests...
[INFO:2023-01-21 09:21:39,368]: Triggering test 1 of 1...
[INFO:2023-01-21 09:21:39,373]: RunnerClient: Loading test {'directory': '/opt/kafka-dev/tests/kafkatest/tests/core', 'file_name': 'throttling_test.py', 'cls_name': 'ThrottlingTest', 'method_name': 'test_throttled_reassignment', 'injected_args': {'bounce_brokers': 'false'}}
[INFO:2023-01-21 09:21:39,375]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: on run 1/1
[INFO:2023-01-21 09:21:39,376]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Setting up...
[INFO:2023-01-21 09:21:41,701]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Running...
[INFO:2023-01-21 09:26:23,770]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Tearing down...
[WARNING - 2023-01-21 09:26:24,090 - service_registry - stop_all - lineno:53]: Error stopping service <ProducerPerformanceService-0-139773137726720: num_nodes: 1, nodes: ['ducker09']>: 1
[INFO:2023-01-21 09:27:12,415]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: PASS
[INFO:2023-01-21 09:27:12,415]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Data: None
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.3
session_id:       2023-01-21--017
run time:         5 minutes 33.062 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
================================================================================
test_id:    kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false
status:     PASS
run time:   5 minutes 33.040 seconds
--------------------------------------------------------------------------------

fvaleri · 2023-01-24T08:21:32Z

@mimaison @clolov @vamossagar12 this is ready for review if you have some time.

mimaison

Thanks @fvaleri for the PR. I left a few comments. I think we should use this opportunity to add a few tests too. Ideally write them against the previous tool first to ensure the new version works the same way.

checkstyle/import-control.xml

core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

tools/src/main/java/org/apache/kafka/tools/JmxCommand.java

fvaleri · 2023-01-25T14:31:01Z

Thanks @mimaison.

I addressed all your comments and now working on the test suite.

…lasses These classes are required by most commands, so they must be migrated first. Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri · 2023-01-27T10:35:57Z

@mimaison I've rebased and added some tests.

This is now ready for another review. Thanks.

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison

Thanks for the PR. I left a few questions.

checkstyle/import-control.xml

core/src/main/scala/kafka/Kafka.scala

core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

tools/src/test/java/org/apache/kafka/tools/JmxCommandTest.java

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java

tools/src/main/java/org/apache/kafka/tools/JmxTool.java

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison

LGTM

ijuma · 2023-02-11T17:33:43Z

Looks like this caused some system test failures: http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/trunk/2023-02-11--001.system-test-kafka-trunk--1676126329--confluentinc--master--d0d9e3f297/report.html

TimeoutError('ubuntu@worker9: Jmx tool took too long to start')
Traceback (most recent call last):
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/tests/runner_client.py", line 262, in run_test
return self.test_context.function(self.test)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/mark/_mark.py", line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/tests/client/quota_test.py", line 139, in test_quota
self.kafka.start()
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/kafka/kafka.py", line 654, in start
self.wait_for_start(node, monitor, timeout_sec)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/kafka/kafka.py", line 890, in wait_for_start
self.start_jmx_tool(self.idx(node), node)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/monitor/jmx.py", line 90, in start_jmx_tool
wait_until(lambda: self._jmx_has_output(node), timeout_sec=30, backoff_sec=.5, err_msg="%s: Jmx tool took too long to start" % node.account)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/utils/util.py", line 58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: ubuntu@worker9: Jmx tool took too long to start

fvaleri marked this pull request as draft January 20, 2023 16:56

fvaleri force-pushed the move-jmx-tool branch 2 times, most recently from 2e97c4a to 16b90df Compare January 20, 2023 17:31

fvaleri marked this pull request as ready for review January 21, 2023 14:54

fvaleri force-pushed the move-jmx-tool branch 2 times, most recently from 1b927c6 to b7daf79 Compare January 23, 2023 13:12

mimaison reviewed Jan 24, 2023

View reviewed changes

fvaleri force-pushed the move-jmx-tool branch from 28d9f91 to cbdb6ea Compare January 24, 2023 18:22

fvaleri added 3 commits January 26, 2023 20:47

KAFKA-14628: Move CommandLineUtils and CommandDefaultOptions shared c…

b648e19

…lasses These classes are required by most commands, so they must be migrated first. Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

KAFKA-14582: Move JmxTool to tools

e40010d

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Add MM feedback

a167801

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison mentioned this pull request Jan 27, 2023

KAFKA-14559: Fix JMX tool to handle the object names with wild cards and optional attributes #13060

Merged

3 tasks

Add integration test

37b998a

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri force-pushed the move-jmx-tool branch from f05b104 to 37b998a Compare January 27, 2023 10:29

fvaleri added 2 commits January 27, 2023 16:39

Switch back to the original class name

4213449

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Minor improvements

607ff24

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison reviewed Jan 31, 2023

View reviewed changes

Add changes from Mickael feedback

f825f87

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison reviewed Feb 1, 2023

View reviewed changes

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java Outdated Show resolved Hide resolved

fvaleri and others added 2 commits February 1, 2023 18:12

Make Idea JUnit pluging happy

d86dc35

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Remove printStackTrace()

aa99752

mimaison approved these changes Feb 2, 2023

View reviewed changes

mimaison merged commit 50e0e3c into apache:trunk Feb 2, 2023

fvaleri deleted the move-jmx-tool branch February 2, 2023 10:23

chia7712 mentioned this pull request Feb 11, 2023

MINOR: the package of JmxTool is incorrect when running quota_test.py #13233

Merged

3 tasks

fvaleri added the tools label May 22, 2023

Conversation

fvaleri commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fvaleri commented Jan 24, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fvaleri commented Jan 25, 2023

Uh oh!

fvaleri commented Jan 27, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

ijuma commented Feb 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fvaleri commented Jan 20, 2023 •

edited

Loading

ijuma commented Feb 11, 2023 •

edited

Loading