Standardizing Logging Level Strings, Fixed do_kafka_test.sh script
#31
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| FROM ubuntu:18.04 | ||
| USER root | ||
|
|
||
| WORKDIR /cvdi-stream | ||
|
|
||
| ENV DEBIAN_FRONTEND=noninteractive | ||
|
|
||
| # Add build tools. | ||
| RUN apt-get update && apt-get install -y software-properties-common wget git make gcc-7 g++-7 gcc-7-base && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 100 && update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-7 100 | ||
|
|
||
| # Install cmake. | ||
| RUN apt install -y libprotobuf-dev protobuf-compiler | ||
| RUN apt install -y cmake | ||
|
|
||
| # Install librdkafka. | ||
| RUN apt-get install -y libsasl2-dev libsasl2-modules libssl-dev librdkafka-dev | ||
|
|
||
| # install python for testing | ||
| RUN apt-get install -y python3 | ||
|
|
||
| # add the source and build files | ||
| ADD CMakeLists.txt /cvdi-stream | ||
| ADD ./src /cvdi-stream/src | ||
| ADD ./cv-lib /cvdi-stream/cv-lib | ||
| ADD ./include /cvdi-stream/include | ||
| ADD ./kafka-test /cvdi-stream/kafka-test | ||
| ADD ./unit-test-data /cvdi-stream/unit-test-data | ||
| ADD ./config /cvdi-stream/config | ||
|
|
||
| # Do the build. | ||
| RUN export LD_LIBRARY_PATH=/usr/local/lib && mkdir /cvdi-stream-build && cd /cvdi-stream-build && cmake /cvdi-stream && make | ||
|
There was a problem hiding this comment. Is this /cvdi-stream-build folder or a build folder inside /cvdi-stream? like /cvdi-stream/build There was a problem hiding this comment. Can we reuse the ./devcontainer and replace the Dockerfile.testing with the vscode test task for the testing? There was a problem hiding this comment. Project files are stored in There was a problem hiding this comment. The I don't think this should be replaced by using the dev container, since the dev container is meant for development and the |
||
|
|
||
| # Add test data. This changes frequently so keep it low in the file. | ||
| ADD ./docker-test /cvdi-stream/docker-test | ||
|
|
||
| # Run the tool. | ||
| RUN chmod 7777 /cvdi-stream/docker-test/ppm_no_map.sh | ||
|
|
||
| CMD ["/cvdi-stream/docker-test/ppm_no_map.sh"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,71 +1,172 @@ | ||
| #!/bin/bash | ||
|
|
||
| # This script tests the PPM against a kafka cluster. It sets up variables for container and input data | ||
| # file names. It starts a Kafka container using another script and checks that required topics are created. | ||
| # If the container or topics are missing, the script exits. It builds a Docker image using the current | ||
| # directory and specified name/tag. It runs a series of tests using a script with different properties | ||
| # and input data files, outputting results to the console. It stops the Kafka container after the tests | ||
| # are completed. The script performs five steps: set up the test environment, wait for Kafka to create | ||
| # topics, build the PPM image, run the tests, and clean up. | ||
|
|
||
| CYAN='\033[0;36m' | ||
| YELLOW='\033[1;33m' | ||
| NC='\033[0m' # No Color | ||
|
|
||
| CURRENT_DIR_NAME=${PWD##*/} | ||
|
There was a problem hiding this comment. Test Scenario 1: Test Scenario 2: IT will start the containers and then stop them at step 2. Should we remove the stop_kafka.sh from the start_kafka.sh script? There was a problem hiding this comment. If Zookeeper and Kafka are already running, the If Zookeeper and Kafka are not running, the Another comment of yours mentions that the kafka container name uses an underscore, not a hyphen. It's possible that this is why the container is not being found in Step 2. Interestingly, when I run the script the container name uses a hyphen, not an underscore. I wonder if this could be due to differences in docker versions. I am using Docker Desktop 4.17.0, what are you using? There was a problem hiding this comment. I am using ubuntu 18.04 and docker version 20.10.6 |
||
| KAFKA_CONTAINER_NAME=$CURRENT_DIR_NAME-kafka-1 | ||
| MAP_FILE=data/I_80.edges | ||
| BSM_DATA_FILE=data/I_80_test.json | ||
| TIM_DATA_FILE=data/I_80_test_TIMS.json | ||
| PPM_CONTAINER_NAME=test_ppm_instance | ||
| PPM_IMAGE_TAG=do-kafka-test-ppm-image | ||
| PPM_IMAGE_NAME=jpo-cvdp_ppm | ||
|
|
||
| setup() { | ||
| if [ -z $DOCKER_HOST_IP ]; then | ||
| echo "DOCKER_HOST_IP is not set. Exiting." | ||
| exit 1 | ||
| fi | ||
|
|
||
| # print setup info | ||
| echo "=== Setup Info ===" | ||
| echo "DOCKER_HOST_IP: $DOCKER_HOST_IP" | ||
| echo "KAFKA_CONTAINER_NAME: $KAFKA_CONTAINER_NAME" | ||
|
There was a problem hiding this comment.
There was a problem hiding this comment. On my machine, the kafka container name is separated by a hyphen. This may have to do with differences in Docker Desktop versions. I am using v4.17.0, what are you using? We may want to consider modifying the script to check for container names separated by an underscore and a hyphen, just to cover all of our bases. |
||
| echo "MAP_FILE: $MAP_FILE" | ||
| echo "BSM_DATA_FILE: $BSM_DATA_FILE" | ||
| echo "TIM_DATA_FILE: $TIM_DATA_FILE" | ||
| echo "PPM_CONTAINER_NAME: $PPM_CONTAINER_NAME" | ||
| echo "PPM_IMAGE_TAG: $PPM_IMAGE_TAG" | ||
| echo "PPM_IMAGE_NAME: $PPM_IMAGE_NAME" | ||
| echo "==================" | ||
|
|
||
| ./start_kafka.sh | ||
| } | ||
|
|
||
| waitForKafkaToCreateTopics() { | ||
| maxAttempts=100 | ||
| attempts=0 | ||
| while true; do | ||
| attempts=$((attempts+1)) | ||
| if [ $(docker ps | grep $KAFKA_CONTAINER_NAME | wc -l) == "0" ]; then | ||
| echo "Kafka container '$KAFKA_CONTAINER_NAME' is not running. Exiting." | ||
| ./stop_kafka.sh | ||
| exit 1 | ||
| fi | ||
|
|
||
| ltopics=$(docker exec -it $KAFKA_CONTAINER_NAME /opt/kafka/bin/kafka-topics.sh --list --zookeeper 172.17.0.1) | ||
| allTopicsCreated=true | ||
| if [ $(echo $ltopics | grep "topic.FilteredOdeBsmJson" | wc -l) == "0" ]; then | ||
| allTopicsCreated=false | ||
| elif [ $(echo $ltopics | grep "topic.FilteredOdeTimJson" | wc -l) == "0" ]; then | ||
| allTopicsCreated=false | ||
| elif [ $(echo $ltopics | grep "topic.OdeBsmJson" | wc -l) == "0" ]; then | ||
| allTopicsCreated=false | ||
| elif [ $(echo $ltopics | grep "topic.OdeTimJson" | wc -l) == "0" ]; then | ||
| allTopicsCreated=false | ||
| fi | ||
|
|
||
| if [ $allTopicsCreated == true ]; then | ||
| echo "Kafka has created all required topics" | ||
| break | ||
| fi | ||
|
|
||
| sleep 1 | ||
|
|
||
| if [ $attempts -ge $maxAttempts ]; then | ||
| echo "Kafka has not created all required topics after $maxAttempts attempts. Exiting." | ||
| ./stop_kafka.sh | ||
| exit 1 | ||
| fi | ||
| done | ||
| } | ||
|
|
||
| buildPPMImage() { | ||
| docker build . -t $PPM_IMAGE_NAME:$PPM_IMAGE_TAG -f Dockerfile.testing | ||
| } | ||
|
|
||
| run_tests() { | ||
| echo "--- File Being Used ---" | ||
| echo $MAP_FILE | ||
| echo $BSM_DATA_FILE | ||
| echo $TIM_DATA_FILE | ||
| echo "-----------------" | ||
|
|
||
| numberOfTests=10 | ||
| echo -e $YELLOW"Test 1/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c1.properties $BSM_DATA_FILE BSM 0 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 2/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c2.properties $BSM_DATA_FILE BSM 10 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 3/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c3.properties $BSM_DATA_FILE BSM 18 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 4/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c4.properties $BSM_DATA_FILE BSM 23 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 5/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c5.properties $BSM_DATA_FILE BSM 33 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 6/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/bsm-test/c6.properties $BSM_DATA_FILE BSM 43 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 7/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/tim-test/c1.properties $TIM_DATA_FILE TIM 0 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 8/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/tim-test/c2.properties $TIM_DATA_FILE TIM 10 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 9/$numberOfTests"$NC | ||
| ./test-scripts/standalone.sh $MAP_FILE config/tim-test/c3.properties $TIM_DATA_FILE TIM 18 | ||
| echo "" | ||
| echo "" | ||
|
|
||
| echo -e $YELLOW"Test 10/$numberOfTests (2 tests in one)"$NC | ||
| ./test-scripts/standalone_multi.sh $MAP_FILE config/bsm-test/c6.properties config/tim-test/c3.properties $BSM_DATA_FILE $TIM_DATA_FILE 48 23 | ||
| } | ||
|
|
||
| cleanup() { | ||
| echo "[log] stopping Kafka" | ||
| ./stop_kafka.sh | ||
| } | ||
|
|
||
| run() { | ||
| numberOfSteps=5 | ||
| echo "" | ||
| echo -e $CYAN"Step 1/$numberOfSteps: Set up test environment"$NC | ||
| setup | ||
|
|
||
| echo "" | ||
| echo -e $CYAN"Step 2/$numberOfSteps: Wait for Kafka to create topics"$NC | ||
| waitForKafkaToCreateTopics | ||
|
|
||
| echo "" | ||
| echo -e $CYAN"Step 3/$numberOfSteps: Build PPM image"$NC | ||
| buildPPMImage | ||
|
|
||
| echo "" | ||
| echo -e $CYAN"Step 4/$numberOfSteps: Run tests"$NC | ||
| run_tests | ||
|
|
||
| echo "" | ||
| echo -e $CYAN"Step 5/$numberOfSteps: Cleanup"$NC | ||
| cleanup | ||
| } | ||
|
|
||
| run | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| version: '2' | ||
| services: | ||
| zookeeper: | ||
| image: wurstmeister/zookeeper | ||
| ports: | ||
| - "2181:2181" | ||
| kafka: | ||
| image: wurstmeister/kafka | ||
| ports: | ||
| - "9092:9092" | ||
| environment: | ||
| KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_HOST_IP} | ||
| KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 | ||
| KAFKA_CREATE_TOPICS: "topic.OdeBsmJson:1:1,topic.FilteredOdeBsmJson:1:1,topic.OdeTimJson:1:1,topic.FilteredOdeTimJson:1:1" | ||
| volumes: | ||
| - /var/run/docker.sock:/var/run/docker.sock |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
which repository does this feature reference to?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a warning for this feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure it is just my local environment. Sometimes the code take some time to run with the feature on and the terminal does not return control to user.
