diff --git a/doc/started.md b/doc/started.md index fdbe46e..c86c57e 100644 --- a/doc/started.md +++ b/doc/started.md @@ -10,7 +10,7 @@ If you are running Windows, you will be using WSL for development, so make sure ## Prepare data directories -In production environments, Capillaries server components ([Daemon](glossary.md#daemon), [Toolbelt](glossary.md#toolbelt), [Webapi](glossary.md#webapi)) need access to configuration files, source data files and target directories. In dev environments, we want Capillaries components to access those files and directories in the uniform way: for dockerized component and for the scenarios when [Daemon](glossary.md#daemon), [Toolbelt](glossary.md#toolbelt), and [Webapi](glossary.md#webapi) are run by developers. We use `/tmp/capitest_*` directories that can be accessed using the same path - from the host machine and from containers (see [docker-compose.yml](../docker-compose.yml) for volume definitions). +In production environments, Capillaries server components ([Daemon](glossary.md#daemon), [Toolbelt](glossary.md#toolbelt), [Webapi](glossary.md#webapi)) need access to configuration files, source data files and target directories. In dev environments, we want Capillaries components to access those files and directories in the uniform way: for dockerized component and for the scenarios when [Daemon](glossary.md#daemon), [Toolbelt](glossary.md#toolbelt), and [Webapi](glossary.md#webapi) are run by developers. We use `/tmp/capi_*` directories that can be accessed using the same path - from the host machine and from containers (see [docker-compose.yml](../docker-compose.yml) for volume definitions). Run these commands from the root project directory, they will create those data directories and populate them with sample configurations and sample source data: @@ -103,7 +103,7 @@ If you are running Windows, you will be using WSL for development, so make sure Is there a way to develop and debug Capillaries server components in a dev environment like VSCode without running it from WSL? Yes, but you will have to solve two problems. -1. Data directories `/tmp/capitest_*` will not be available from Windows, so you will have to tweak all configuration files and shell scripts so they reference Windows paths. This is doable, but it's a tedious job. +1. Data directories `/tmp/capi_*` will not be available from Windows, so you will have to tweak all configuration files and shell scripts so they reference Windows paths. This is doable, but it's a tedious job. 2. When [test_tag_and_denormalize](../test/code/tag_and_denormalize/README.md) integration test runs in WSL and uses Webapi executed from Windows dev environment, `curl` command will not be able to connect to Webapi's `http://localhost:6543` because of the known WSL limitation descussed at https://github.com/microsoft/WSL/issues/5211 and at https://superuser.com/questions/1679757/how-to-access-windows-localhost-from-wsl2 . You will need to use host IP address or use `$(localhost).local` instead of `localhost` in the shell script. diff --git a/test/data/cfg/tag_and_denormalize_quicktest/script_params_one_run_input_https.json b/test/data/cfg/tag_and_denormalize_quicktest/script_params_one_run_input_https.json index 3f7a8d0..2cb0441 100644 --- a/test/data/cfg/tag_and_denormalize_quicktest/script_params_one_run_input_https.json +++ b/test/data/cfg/tag_and_denormalize_quicktest/script_params_one_run_input_https.json @@ -2,6 +2,6 @@ "totals_start_policy": "auto", "uri_suffix": "?raw=1", "dir_cfg": "https://github.com/capillariesio/capillaries/blob/main/test/data/cfg/tag_and_denormalize_quicktest", - "dir_in": "https://github.com/capillariesio/capillaries/blob/main/test/data/in/tag_and_denormalize", + "dir_in": "https://github.com/capillariesio/capillaries/blob/main/test/data/in/tag_and_denormalize_quicktest", "dir_out": "/tmp/capi_out/tag_and_denormalize_quicktest" } \ No newline at end of file diff --git a/test/data/in/lookup/olist_order_items_dataset.csv b/test/data/in/lookup_quicktest/olist_order_items_dataset.csv similarity index 100% rename from test/data/in/lookup/olist_order_items_dataset.csv rename to test/data/in/lookup_quicktest/olist_order_items_dataset.csv diff --git a/test/data/in/lookup/olist_orders_dataset.csv b/test/data/in/lookup_quicktest/olist_orders_dataset.csv similarity index 100% rename from test/data/in/lookup/olist_orders_dataset.csv rename to test/data/in/lookup_quicktest/olist_orders_dataset.csv diff --git a/test/data/in/portfolio/accounts.csv b/test/data/in/portfolio_quicktest/accounts.csv similarity index 100% rename from test/data/in/portfolio/accounts.csv rename to test/data/in/portfolio_quicktest/accounts.csv diff --git a/test/data/in/portfolio/holdings.csv b/test/data/in/portfolio_quicktest/holdings.csv similarity index 100% rename from test/data/in/portfolio/holdings.csv rename to test/data/in/portfolio_quicktest/holdings.csv diff --git a/test/data/in/portfolio/txns.csv b/test/data/in/portfolio_quicktest/txns.csv similarity index 100% rename from test/data/in/portfolio/txns.csv rename to test/data/in/portfolio_quicktest/txns.csv diff --git a/test/data/in/py_calc/olist_order_items_dataset00.csv b/test/data/in/py_calc_quicktest/olist_order_items_dataset00.csv similarity index 100% rename from test/data/in/py_calc/olist_order_items_dataset00.csv rename to test/data/in/py_calc_quicktest/olist_order_items_dataset00.csv diff --git a/test/data/in/py_calc/olist_order_items_dataset01.csv b/test/data/in/py_calc_quicktest/olist_order_items_dataset01.csv similarity index 100% rename from test/data/in/py_calc/olist_order_items_dataset01.csv rename to test/data/in/py_calc_quicktest/olist_order_items_dataset01.csv diff --git a/test/data/in/py_calc/olist_order_items_dataset02.csv b/test/data/in/py_calc_quicktest/olist_order_items_dataset02.csv similarity index 100% rename from test/data/in/py_calc/olist_order_items_dataset02.csv rename to test/data/in/py_calc_quicktest/olist_order_items_dataset02.csv diff --git a/test/data/in/py_calc/olist_order_items_dataset03.csv b/test/data/in/py_calc_quicktest/olist_order_items_dataset03.csv similarity index 100% rename from test/data/in/py_calc/olist_order_items_dataset03.csv rename to test/data/in/py_calc_quicktest/olist_order_items_dataset03.csv diff --git a/test/data/in/py_calc/olist_order_items_dataset04.csv b/test/data/in/py_calc_quicktest/olist_order_items_dataset04.csv similarity index 100% rename from test/data/in/py_calc/olist_order_items_dataset04.csv rename to test/data/in/py_calc_quicktest/olist_order_items_dataset04.csv diff --git a/test/data/in/tag_and_denormalize/flipcart_products.tsv b/test/data/in/tag_and_denormalize_quicktest/flipcart_products.tsv similarity index 100% rename from test/data/in/tag_and_denormalize/flipcart_products.tsv rename to test/data/in/tag_and_denormalize_quicktest/flipcart_products.tsv diff --git a/test/data/in/tag_and_denormalize/tags.csv b/test/data/in/tag_and_denormalize_quicktest/tags.csv similarity index 100% rename from test/data/in/tag_and_denormalize/tags.csv rename to test/data/in/tag_and_denormalize_quicktest/tags.csv diff --git a/test/data/out/lookup/order_date_value_grouped_inner_baseline.csv b/test/data/out/lookup_quicktest/order_date_value_grouped_inner_baseline.csv similarity index 100% rename from test/data/out/lookup/order_date_value_grouped_inner_baseline.csv rename to test/data/out/lookup_quicktest/order_date_value_grouped_inner_baseline.csv diff --git a/test/data/out/lookup/order_date_value_grouped_left_outer_baseline.csv b/test/data/out/lookup_quicktest/order_date_value_grouped_left_outer_baseline.csv similarity index 100% rename from test/data/out/lookup/order_date_value_grouped_left_outer_baseline.csv rename to test/data/out/lookup_quicktest/order_date_value_grouped_left_outer_baseline.csv diff --git a/test/data/out/lookup/order_item_date_inner_baseline.csv b/test/data/out/lookup_quicktest/order_item_date_inner_baseline.csv similarity index 100% rename from test/data/out/lookup/order_item_date_inner_baseline.csv rename to test/data/out/lookup_quicktest/order_item_date_inner_baseline.csv diff --git a/test/data/out/lookup/order_item_date_left_outer_baseline.csv b/test/data/out/lookup_quicktest/order_item_date_left_outer_baseline.csv similarity index 100% rename from test/data/out/lookup/order_item_date_left_outer_baseline.csv rename to test/data/out/lookup_quicktest/order_item_date_left_outer_baseline.csv diff --git a/test/data/out/lookup_quicktest/readme.txt b/test/data/out/lookup_quicktest/readme.txt new file mode 100644 index 0000000..a14a42a --- /dev/null +++ b/test/data/out/lookup_quicktest/readme.txt @@ -0,0 +1 @@ +Placeholder for out files \ No newline at end of file diff --git a/test/data/out/portfolio/account_period_sector_perf_baseline.csv b/test/data/out/portfolio_quicktest/account_period_sector_perf_baseline.csv similarity index 100% rename from test/data/out/portfolio/account_period_sector_perf_baseline.csv rename to test/data/out/portfolio_quicktest/account_period_sector_perf_baseline.csv diff --git a/test/data/out/portfolio/account_year_perf_baseline.csv b/test/data/out/portfolio_quicktest/account_year_perf_baseline.csv similarity index 100% rename from test/data/out/portfolio/account_year_perf_baseline.csv rename to test/data/out/portfolio_quicktest/account_year_perf_baseline.csv diff --git a/test/data/out/portfolio/account_year_perf_official.csv b/test/data/out/portfolio_quicktest/account_year_perf_official.csv similarity index 100% rename from test/data/out/portfolio/account_year_perf_official.csv rename to test/data/out/portfolio_quicktest/account_year_perf_official.csv diff --git a/test/data/out/py_calc_quicktest/readme.txt b/test/data/out/py_calc_quicktest/readme.txt new file mode 100644 index 0000000..a14a42a --- /dev/null +++ b/test/data/out/py_calc_quicktest/readme.txt @@ -0,0 +1 @@ +Placeholder for out files \ No newline at end of file diff --git a/test/data/out/py_calc/taxed_order_items_go_baseline.csv b/test/data/out/py_calc_quicktest/taxed_order_items_go_baseline.csv similarity index 100% rename from test/data/out/py_calc/taxed_order_items_go_baseline.csv rename to test/data/out/py_calc_quicktest/taxed_order_items_go_baseline.csv diff --git a/test/data/out/py_calc/taxed_order_items_py_baseline.csv b/test/data/out/py_calc_quicktest/taxed_order_items_py_baseline.csv similarity index 100% rename from test/data/out/py_calc/taxed_order_items_py_baseline.csv rename to test/data/out/py_calc_quicktest/taxed_order_items_py_baseline.csv diff --git a/test/data/out/tag_and_denormalize_quicktest/readme.txt b/test/data/out/tag_and_denormalize_quicktest/readme.txt new file mode 100644 index 0000000..a14a42a --- /dev/null +++ b/test/data/out/tag_and_denormalize_quicktest/readme.txt @@ -0,0 +1 @@ +Placeholder for out files \ No newline at end of file diff --git a/test/data/out/tag_and_denormalize/tag_totals_baseline.tsv b/test/data/out/tag_and_denormalize_quicktest/tag_totals_baseline.tsv similarity index 100% rename from test/data/out/tag_and_denormalize/tag_totals_baseline.tsv rename to test/data/out/tag_and_denormalize_quicktest/tag_totals_baseline.tsv diff --git a/test/data/out/tag_and_denormalize/tagged_products_for_operator_review_baseline.csv b/test/data/out/tag_and_denormalize_quicktest/tagged_products_for_operator_review_baseline.csv similarity index 100% rename from test/data/out/tag_and_denormalize/tagged_products_for_operator_review_baseline.csv rename to test/data/out/tag_and_denormalize_quicktest/tagged_products_for_operator_review_baseline.csv diff --git a/test/deploy/README.md b/test/deploy/README.md index 6e2852d..5321319 100644 --- a/test/deploy/README.md +++ b/test/deploy/README.md @@ -44,8 +44,8 @@ Host <genesis_bastion_ip> Just for convenience, let's store deploy tool arguments and other configuration settings in shell variables, for example: ``` export capideploy=../../build/capideploy.exe -export DEPLOY_ARGS="-prj capideploy_project_dreamhost.json -prj_params $HOME/capideploy_project_params_dreamhost.json" -export DEPLOY_ROOT_KEY=$HOME/.ssh/sampledeployment001_rsa +export DEPLOY_ARGS="-prj capideploy_project_genesis.json -prj_params $HOME/capideploy_project_params_genesis.json" +export DEPLOY_ROOT_KEY=$HOME/.ssh/sampledeployment002_rsa export BASTION_IP=<dreamhost_bastion_ip> ``` @@ -97,7 +97,9 @@ This command will populate /tmp/capi_in, /tmp/capi_cfg, /tmp/capi_out cd ./test/code/lookup ./1_create_quicktest_data.sh ./create_bigtest_data.sh -cd test/code/tag_and_denormalize +cd ../py_calc +./1_create_quicktest_data.sh +cd ../tag_and_denormalize ./1_create_quicktest_data.sh ```