Fixed `do_kafka_test.sh` script #28

dmccoystephenson · 2023-05-10T15:19:38Z

Changes

The do_kafka_test.sh was created six years ago, but it was discovered last year that it was not functioning properly due to inconsistencies in container name references and issues with dependent scripts.

The script has been refactored and thoroughly tested to ensure its functionality when executed in WSL.

Additional Changes

The dockerfile.testing file has been added to the project for testing purposes. This includes a python installation.
The majority of the code has been moved into methods in do_kafka_test.sh
Color has been added to the do_kafka_test.sh, do_bsm_test.sh and do_tim_test.sh scripts.
The standalone.sh and standalone-multi.sh scripts have been refactored.
Descriptions as comments have been added to a number of scripts.
The DOCKER_HOST_IP environment variable is now required to be set for the do_kafka_test.sh script to run.
The docker-compose-kafka.yml file has been added for testing purposes.

Testing

These changes have been verified to be working when the script is run locally in WSL.

…petitive behavior.

…dability.

…ipt.

…to the `standalone.sh` script.

…one.sh` script

…time to be ready.

drewjj

This looks good to me

Dockerfile-nsv

do_kafka_test.sh

payneBrandon

Looking good, just a couple comments

…ndalone`

payneBrandon · 2023-06-21T17:42:10Z

@dmccoystephenson the do_kafka_test.sh script seems to be failing on all the tests for me

payneBrandon

Tests currently failing - maybe I missed a step somewhere?

dmccoystephenson · 2023-06-22T14:57:37Z

Tests currently failing - maybe I missed a step somewhere?

The tests fail sometimes when data cannot be verified to be consumed from the kafka cluster. I have found that the tests are more likely to pass after a fresh restart of the host computer. It's possible that some background processing with docker is causing the issue (which would explain why a restart would help). I'll see if I can reproduce the fail case and identify what the problem is.

dmccoystephenson · 2023-06-22T20:50:03Z

Tests pass for me. I am running do_kafka_test.sh in WSL with no other active containers present at the time of execution.

I will continue to experiment until the tests fail and then investigate further.

dmccoystephenson · 2023-06-23T19:05:34Z

Compilation was failing when using Ubuntu 20.04 as the base image. After switching back to using Ubuntu 18.04, the tests are passing again.

…ut to the kafka-topics.sh script.

dmccoystephenson · 2023-06-23T21:28:18Z

These changes have been re-deployed to our dev cluster and data has been verified to still be flowing.

…topic creation check.

drewjj

Looks like the tests are not able to consume any of the produced messages. Not sure if this is a timing issue or something else. Definitely needs some more investigation.

Sync Master branch to develop

… PPM logs is above 100 instead of waiting a fixed amount of time for PPM readiness.

dmccoystephenson · 2023-07-11T20:53:10Z

I suspect that the do_kafka_test.sh script was failing sometimes because data got produced before the PPM was ready to consume it. I have modified the standalone test scripts to wait until the PPM logs reach 100 lines instead of waiting a fixed 20 seconds for PPM readiness.

Please let me know if you run into more issues @drewjj

drewjj

I am still seeing my tests fail. I am running it in Ubuntu, I have the proper IP information setup due to the script successfully creating the messages but the test seems to fail at reading them for its test case. Unsure what to do here.

drewjj

The test script works great now. I hadn't fully updated my local repository before.

dmccoystephenson

Self-review

README.md

do_kafka_test.sh

…`standalone-multi.sh`

…`do_kafka_test.sh`

…script calls in `do_bsm_test.sh` & `do_tim_test.sh`

dmccoystephenson added 12 commits April 13, 2022 15:23

Made some changes to resolve some issues with do_kafka_test.sh script.

970f1c2

Merge branch 'develop' into fixing-kafka-test-script

03440ab

Refactored scripts to improve clarity of execution and to minimize re…

11600d9

…petitive behavior.

Organized code in do_kafka_test.sh script into methods to improve rea…

57aebcc

…dability.

Made the DOCKER_HOST_IP env var required for the do_kafka_test.sh scr…

2496024

…ipt.

Modified topic creation check in do_kafka_test.sh script.

7d6d8c8

Moved the responsibility of starting/stopping the PPM container back …

19af792

…to the `standalone.sh` script.

Modified & added color to echo messages.

c6bb676

Gave the PPM container an arbitrary 10 seconds to spin up in `standal…

58ae4c8

…one.sh` script

Refactored standalone_multi.sh script and gave PPM containers more …

cb3b5a9

…time to be ready.

Added descriptions as comments to a number of scripts.

2685f49

Removed duplicate python installation in Dockerfile.standalone

adfb334

dmccoystephenson requested review from payneBrandon and drewjj May 10, 2023 15:19

drewjj approved these changes Jun 14, 2023

View reviewed changes

Created separate dockerfile for testing.

b8b1124

payneBrandon reviewed Jun 19, 2023

View reviewed changes

Dockerfile-nsv Outdated Show resolved Hide resolved

payneBrandon reviewed Jun 19, 2023

View reviewed changes

do_kafka_test.sh Show resolved Hide resolved

payneBrandon requested changes Jun 19, 2023

View reviewed changes

dmccoystephenson added 2 commits June 19, 2023 11:41

Removed python installation from Dockerfile-nsv and `Dockerfile.sta…

4dbfdb0

…ndalone`

Added documentation for the do_kafka_test.sh script to the README.

96976ff

dmccoystephenson requested a review from payneBrandon June 19, 2023 20:49

Merge branch 'develop' into fixing-kafka-test-script

9dfa167

payneBrandon requested changes Jun 21, 2023

View reviewed changes

dmccoystephenson requested a review from payneBrandon June 22, 2023 22:47

dmccoystephenson added 2 commits June 23, 2023 09:12

Modified 'Kafka Test Script' section of README.

42cda70

Modified Build & Exec Script section of the README.

a22253b

dmccoystephenson marked this pull request as draft June 23, 2023 17:43

Switched back to using ubuntu:18.04 for the base image.

30ca2af

dmccoystephenson marked this pull request as ready for review June 23, 2023 19:05

Switched over to using $DOCKER_HOST_IP instead of 172.17.0.1 as inp…

0300a34

…ut to the kafka-topics.sh script.

dmccoystephenson marked this pull request as draft June 23, 2023 20:10

dmccoystephenson marked this pull request as ready for review June 23, 2023 20:16

Added a note regarding WSL hanging to the README.

84e5153

dmccoystephenson requested a review from drewjj June 23, 2023 21:09

Switched back to using 172.17.0.1 instead of $DOCKER_HOST_IP for the …

af674cf

…topic creation check.

drewjj requested changes Jun 23, 2023

View reviewed changes

dmccoystephenson marked this pull request as draft June 23, 2023 22:58

dmccoystephenson pushed a commit that referenced this pull request Jul 11, 2023

Merge pull request #28 from usdot-jpo-ode/master

6e43b7f

Sync Master branch to develop

Modified standalone test scripts to wait until the number of lines of…

76de636

… PPM logs is above 100 instead of waiting a fixed amount of time for PPM readiness.

dmccoystephenson requested a review from drewjj July 11, 2023 20:53

dmccoystephenson marked this pull request as ready for review July 11, 2023 20:54

drewjj requested changes Jul 13, 2023

View reviewed changes

dmccoystephenson requested a review from drewjj July 14, 2023 20:25

drewjj approved these changes Jul 14, 2023

View reviewed changes

dmccoystephenson commented Jul 18, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

do_kafka_test.sh Outdated Show resolved Hide resolved

do_kafka_test.sh Outdated Show resolved Hide resolved

dmccoystephenson added 4 commits July 18, 2023 12:23

Modified note about WSL hanging in the README.

9f4b90b

Modified PPM container name in do_kafka_test.sh, standalone.sh & …

144a91f

…`standalone-multi.sh`

Used name of current directory when defining kafka container name in …

69af1ac

…`do_kafka_test.sh`

Printed setup info in do_kafka_test.sh and added timeout to python …

bb0f062

…script calls in `do_bsm_test.sh` & `do_tim_test.sh`

dmccoystephenson mentioned this pull request Jul 26, 2023

Optimize Dockerfiles for Smaller and Faster Deployments #32

Merged

payneBrandon approved these changes Sep 27, 2023

View reviewed changes

payneBrandon merged commit bec5045 into develop Sep 27, 2023

payneBrandon deleted the fixing-kafka-test-script branch September 27, 2023 21:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed `do_kafka_test.sh` script #28

Fixed `do_kafka_test.sh` script #28

dmccoystephenson commented May 10, 2023 •

edited

Loading

drewjj left a comment

payneBrandon left a comment

payneBrandon commented Jun 21, 2023

payneBrandon left a comment

dmccoystephenson commented Jun 22, 2023 •

edited

Loading

dmccoystephenson commented Jun 22, 2023

dmccoystephenson commented Jun 23, 2023

dmccoystephenson commented Jun 23, 2023

drewjj left a comment

dmccoystephenson commented Jul 11, 2023

drewjj left a comment

drewjj left a comment

dmccoystephenson left a comment

Fixed do_kafka_test.sh script #28

Fixed do_kafka_test.sh script #28

Conversation

dmccoystephenson commented May 10, 2023 • edited Loading

Changes

Additional Changes

Testing

drewjj left a comment

Choose a reason for hiding this comment

payneBrandon left a comment

Choose a reason for hiding this comment

payneBrandon commented Jun 21, 2023

payneBrandon left a comment

Choose a reason for hiding this comment

dmccoystephenson commented Jun 22, 2023 • edited Loading

dmccoystephenson commented Jun 22, 2023

dmccoystephenson commented Jun 23, 2023

dmccoystephenson commented Jun 23, 2023

drewjj left a comment

Choose a reason for hiding this comment

dmccoystephenson commented Jul 11, 2023

drewjj left a comment

Choose a reason for hiding this comment

drewjj left a comment

Choose a reason for hiding this comment

dmccoystephenson left a comment

Choose a reason for hiding this comment

Fixed `do_kafka_test.sh` script #28

Fixed `do_kafka_test.sh` script #28

dmccoystephenson commented May 10, 2023 •

edited

Loading

dmccoystephenson commented Jun 22, 2023 •

edited

Loading