Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

censor versioning [T971] #321

Merged
merged 23 commits into from
Mar 1, 2019
Merged

censor versioning [T971] #321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9d758e0
export version as metric
Lagovas Feb 21, 2019
2038bb8
export version as metric
Lagovas Feb 22, 2019
7995802
Merge remote-tracking branch 'origin/master' into lagovas/T1002-expor…
Lagovas Feb 22, 2019
ea7ac0d
export metrics for translator/connector
Lagovas Feb 22, 2019
379b247
move prometheus logic for exporting versions to cmd package
Lagovas Feb 22, 2019
bdf11d6
check censor version; run tests in a loop
Lagovas Feb 22, 2019
00f9683
fix circleci config
Lagovas Feb 26, 2019
b41841e
fix golint issues
Lagovas Feb 26, 2019
7b58bfe
fix identation
Lagovas Feb 26, 2019
b976062
fix
Lagovas Feb 26, 2019
9a7e4b6
update tests requirements
Lagovas Feb 26, 2019
4054945
update censor's tests
Lagovas Feb 26, 2019
666229c
update network tests to support go1.12
Lagovas Feb 26, 2019
af7fd57
Merge remote-tracking branch 'origin/master' into lagovas/T971-censor…
Lagovas Feb 26, 2019
8e1e5e2
fix tests for tls1.3
Lagovas Feb 26, 2019
7ce0741
run go test with GODEBUG=tls13=1
Lagovas Feb 26, 2019
9a52545
fix loop for tests
Lagovas Feb 26, 2019
71f0867
log iterations too
Lagovas Feb 27, 2019
d04df59
store retries in artifacts
Lagovas Feb 27, 2019
5fe8724
save log to artifacts, log status to distinguish timeout error from o…
Lagovas Feb 27, 2019
6b986a0
add tests for version functions
Lagovas Feb 27, 2019
d7850f3
correctly fetch status of bash commands; requested messages changes;
Lagovas Feb 28, 2019
0a145c7
change error text for ErrInvalidVersionFormat
Lagovas Feb 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .circleci/check_gotest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/usr/bin/env bash

# Run test in each go environment and log errors to $FILEPATH_ERROR_FLAG.
# If all tests pass successfully then the file will not be created at all.
cd $HOME
for version in $VERSIONS; do
export GOROOT=$HOME/go_root_$version/go;
export PATH=$GOROOT/bin/:$PATH;
export GOPATH=$HOME/$GOPATH_FOLDER;
rm -rf $HOME/$GOPATH_FOLDER/bin;
rm -rf $HOME/$GOPATH_FOLDER/pkg;

go test -v github.com/cossacklabs/acra/...;
status="$?"
if [[ "${status}" != "0" ]]; then
echo "$version-tls12" >> "$FILEPATH_ERROR_FLAG";
fi

# test with supported tls1.3
GODEBUG="tls13=1" go test -v github.com/cossacklabs/acra/...;
status="$?"
if [[ "${status}" != "0" ]]; then
echo "$version-tls13" >> "$FILEPATH_ERROR_FLAG";
fi
done

# if file exists (exit code of stat == 0 ) then something was wrong. cat file with versions of environments where was error and return exit 1
if [[ -f $FILEPATH_ERROR_FLAG ]]; then
cat "$FILEPATH_ERROR_FLAG";
rm "$FILEPATH_ERROR_FLAG";
exit 1;
fi
58 changes: 34 additions & 24 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,8 @@ jobs:
POSTGRES_USER: test
POSTGRES_DB: test
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.8 1.11.5
VERSIONS: 1.10.8 1.11.5 1.12
TEST_DB_PORT: 5432
GOPATH_FOLDER: gopath
TEST_RANDOM_DATA_FOLDER: /tmp/test_data
Expand All @@ -29,17 +28,18 @@ jobs:
- run: GOPATH=$HOME/$GOPATH_FOLDER .circleci/check_golint.sh
# delete file if exists
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then rm "$FILEPATH_ERROR_FLAG"; fi
# run test in each go environment and create $FILEPATH_ERROR_FLAG file if was any error. But all tests should
- run: cd $HOME && for version in $VERSIONS; do export GOROOT=$HOME/go_root_$version/go; export PATH=$GOROOT/bin/:$PATH; export GOPATH=$HOME/$GOPATH_FOLDER; rm -rf $HOME/$GOPATH_FOLDER/bin; rm -rf $HOME/$GOPATH_FOLDER/pkg; go test -v github.com/cossacklabs/acra/...; if [ "$?" != "0" ]; then echo "$version" >> "$FILEPATH_ERROR_FLAG"; fi done
# if file exists (exit code of stat == 0 ) then something was wrong. cat file with versions of environments where was error and return exit 1
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- run: .circleci/check_gotest.sh
# check python wrapper
- run: PYTHONPATH=`pwd`/wrappers/python python3 wrappers/python/acrawriter/tests.py
# generate test data for integration tests
- run: python3 tests/generate_random_data.py
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

postgresql:
docker:
Expand All @@ -50,9 +50,8 @@ jobs:
POSTGRES_USER: test
POSTGRES_DB: test
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.5 1.11.2
VERSIONS: 1.10.8 1.11.5 1.12
TEST_DB_PORT: 5432
GOPATH_FOLDER: gopath
TEST_RANDOM_DATA_FOLDER: /tmp/test_data
Expand All @@ -68,6 +67,10 @@ jobs:
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

mariadb-ssl:
docker:
Expand All @@ -81,9 +84,8 @@ jobs:
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.5 1.11.2
VERSIONS: 1.10.8 1.11.5 1.12
TEST_MYSQL: true
TEST_DB_PORT: 3306
GOPATH_FOLDER: gopath
Expand All @@ -102,15 +104,16 @@ jobs:
- run: GOPATH=$HOME/$GOPATH_FOLDER .circleci/check_golint.sh
# delete file if exists
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then rm "$FILEPATH_ERROR_FLAG"; fi
# run test in each go environment and create $FILEPATH_ERROR_FLAG file if was any error. But all tests should
- run: cd $HOME && for version in $VERSIONS; do export GOROOT=$HOME/go_root_$version/go; export PATH=$GOROOT/bin/:$PATH; export GOPATH=$HOME/$GOPATH_FOLDER; rm -rf $HOME/$GOPATH_FOLDER/bin; rm -rf $HOME/$GOPATH_FOLDER/pkg; go test -v github.com/cossacklabs/acra/...; if [ "$?" != "0" ]; then echo "$version" >> "$FILEPATH_ERROR_FLAG"; fi done
# if file exists (exit code of stat == 0 ) then something was wrong. cat file with versions of environments where was error and return exit 1
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- run: .circleci/check_gotest.sh
# generate test data for integration tests
- run: python3 tests/generate_random_data.py
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

mariadb:
docker:
Expand All @@ -124,9 +127,8 @@ jobs:
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.5 1.11.2
VERSIONS: 1.10.8 1.11.5 1.12
TEST_MYSQL: true
TEST_DB_PORT: 3306
GOPATH_FOLDER: gopath
Expand All @@ -144,6 +146,10 @@ jobs:
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

mysql:
docker:
Expand All @@ -157,9 +163,8 @@ jobs:
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.5 1.11.2
VERSIONS: 1.10.8 1.11.5 1.12
TEST_MYSQL: true
TEST_DB_PORT: 3306
GOPATH_FOLDER: gopath
Expand All @@ -178,15 +183,16 @@ jobs:
- run: GOPATH=$HOME/$GOPATH_FOLDER .circleci/check_golint.sh
# delete file if exists
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then rm "$FILEPATH_ERROR_FLAG"; fi
# run test in each go environment and create $FILEPATH_ERROR_FLAG file if was any error. But all tests should
- run: cd $HOME && for version in $VERSIONS; do export GOROOT=$HOME/go_root_$version/go; export PATH=$GOROOT/bin/:$PATH; export GOPATH=$HOME/$GOPATH_FOLDER; rm -rf $HOME/$GOPATH_FOLDER/bin; rm -rf $HOME/$GOPATH_FOLDER/pkg; go test -v github.com/cossacklabs/acra/...; if [ "$?" != "0" ]; then echo "$version" >> "$FILEPATH_ERROR_FLAG"; fi done
# if file exists (exit code of stat == 0 ) then something was wrong. cat file with versions of environments where was error and return exit 1
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- run: .circleci/check_gotest.sh
# generate test data for integration tests
- run: python3 tests/generate_random_data.py
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

mysql-ssl:
docker:
Expand All @@ -198,9 +204,8 @@ jobs:
MYSQL_PASSWORD: test
MYSQL_ROOT_PASSWORD: root
environment:
GOTHEMIS_IMPORT: github.com/cossacklabs/themis/gothemis
FILEPATH_ERROR_FLAG: /tmp/test_fail
VERSIONS: 1.9.7 1.10.5 1.11.2
VERSIONS: 1.10.8 1.11.5 1.12
TEST_MYSQL: true
TEST_DB_PORT: 3306
GOPATH_FOLDER: gopath
Expand All @@ -218,6 +223,11 @@ jobs:
# each iteration pass to test different ports for forks to avoid problems with TCP TIME WAIT between tests
- run: .circleci/integration.sh
- run: if [ -f $FILEPATH_ERROR_FLAG ]; then cat "$FILEPATH_ERROR_FLAG"; rm "$FILEPATH_ERROR_FLAG"; exit 1; fi
- store_test_results:
path: /home/user/tests_output
- store_artifacts:
path: /home/user/tests_output

workflows:
version: 2
tests:
Expand Down
33 changes: 27 additions & 6 deletions .circleci/integration.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
set -o pipefail

export TEST_ACRASERVER_PORT=6000
export TEST_CONNECTOR_PORT=7000
Expand All @@ -7,6 +8,12 @@ export TEST_DB_USER=test
export TEST_DB_USER_PASSWORD=test
export TEST_DB_NAME=test
export GOPATH=$HOME/$GOPATH_FOLDER;
# cirecle ci has timeout 10 minutes without output after that it stop execution
# set timeout 8 minutes to give a time to re-start tests execution
export TEST_RUN_TIMEOUT=480 # 8 minutes (8 * 60)

export TEST_OUTPUT_FOLDER="${HOME}/tests_output"
mkdir -p ${TEST_OUTPUT_FOLDER}

cd $HOME/project
# set correct permissions for ssl keys here because git by default recognize changing only executable bit
Expand All @@ -25,10 +32,24 @@ for version in $VERSIONS; do

echo "-------------------- Testing with TEST_TLS=${TEST_TLS}"

python3 tests/test.py -v;
if [[ "$?" != "0" ]]; then
echo "golang-$version tls_on=${TEST_TLS}" >> "$FILEPATH_ERROR_FLAG";
else
echo "no errors";
fi
for iteration in {1..3}; do
context="${iteration}-golang-${version}-tls-${TEST_TLS}"
export TEST_XMLOUTPUT="${TEST_OUTPUT_FOLDER}/${context}.xml"
LOG_OUTPUT="${TEST_OUTPUT_FOLDER}/${context}.log"
timeout ${TEST_RUN_TIMEOUT} python3 tests/test.py -v | tee "${LOG_OUTPUT}";
status="$?"
if [[ "${status}" != "0" ]]; then
status="$?"
ilammy marked this conversation as resolved.
Show resolved Hide resolved
echo "${context}. status=${status}" >> "$FILEPATH_ERROR_FLAG";
continue
else
echo "no errors";
if [[ "${iteration}" != "1" ]]; then
# if test run successful after retries then copy retries log to folder that will be available on circleci ui
cp "${FILEPATH_ERROR_FLAG}" "${TEST_OUTPUT_FOLDER}";
rm "${FILEPATH_ERROR_FLAG}";
fi
break
fi
done
done
35 changes: 32 additions & 3 deletions acra-censor/acra-censor_configuration_provider.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,19 @@ limitations under the License.
package acracensor

import (
"errors"
"github.com/cossacklabs/acra/acra-censor/common"
"github.com/cossacklabs/acra/acra-censor/handlers"
"github.com/cossacklabs/acra/logging"
"github.com/cossacklabs/acra/utils"
"github.com/sirupsen/logrus"
"gopkg.in/yaml.v2"
"strings"
)

// MinimalCensorConfigVersion min version of config that support acra-censor
var MinimalCensorConfigVersion = "0.84.2"

// Query handlers' names.
const (
DenyConfigStr = "deny"
Expand All @@ -36,24 +42,47 @@ const (

// Config shows handlers configuration: queries, tables, patterns
type Config struct {
Handlers []struct {
Version string `yaml:"version"`
IgnoreParseError bool `yaml:"ignore_parse_error"`
ParseErrorsLog string `yaml:"parse_errors_log"`
Handlers []struct {
Handler string
Queries []string
Tables []string
Patterns []string
FilePath string
}
IgnoreParseError bool `yaml:"ignore_parse_error"`
ParseErrorsLog string `yaml:"parse_errors_log"`
}

// ErrUnsupportedConfigVersion acra-censor's config has version less than MinimalCensorConfigVersion
var ErrUnsupportedConfigVersion = errors.New("acra-censor's config is outdated")

// LoadConfiguration loads configuration of AcraCensor
func (acraCensor *AcraCensor) LoadConfiguration(configuration []byte) error {
var censorConfiguration Config
err := yaml.Unmarshal(configuration, &censorConfiguration)
if err != nil {
return err
}
if len(censorConfiguration.Version) == 0 {
return ErrUnsupportedConfigVersion
}
configVersion, err := utils.ParseVersion(censorConfiguration.Version)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

current censor configs don't have version, so it can be parsed. shouldn't we return ErrUnsupportedConfigVersion`?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will return this error and log will looks like:

[...] "Can't setup censor" error="VERSION value has incorrect format (semver 2.0.0 format expected, https://semver.org/)"

There return another error value to distinguish cases when version is missed/has incorrect value and version is outdated. Maybe better to return 3 different error values: missed version/incorrect format/outdated?

Plus I think to change "VERSION value has incorrect format (semver 2.0.0 format expected, https://semver.org/)" -> "version value has incorrect format (semver 2.0.0 format expected, https://semver.org/)". Because at start it was a reference to variable VERSION

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a check on empty version and return ErrUnsupportedConfigVersion

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you, makes sense!

if err != nil {
return err
}
currentlySupportedVersion, err := utils.ParseVersion(MinimalCensorConfigVersion)
if err != nil {
return err
}
if currentlySupportedVersion.Compare(configVersion) == utils.Greater {
logrus.WithField(logging.FieldKeyEventCode, logging.EventCodeErrorCensorSetupError).
Errorln("AcraCensor config file version is not supported: probably AcraCensor configuration " +
"(acra-censor.yaml) is outdated, check docs for deprecation warnings " +
"https://docs.cossacklabs.com/pages/documentation-acra/#acracensor-acra-s-firewall")
// censor has version newer than config
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should log error in human readable form, smth like

acraCensor.logger.
  WithField(logging.FieldKeyEventCode, logging.EventCodeErrorCensorSetupError).
  Errorln("AcraCensor config file version is not supported: probably AcraCensor configuration (acra-censor.yaml) is outdated, check docs for deprecation warnings https://docs.cossacklabs.com/pages/documentation-acra/#acracensor-acra-s-firewall")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, added

return ErrUnsupportedConfigVersion
}
acraCensor.ignoreParseError = censorConfiguration.IgnoreParseError
if !strings.EqualFold(censorConfiguration.ParseErrorsLog, "") {
queryWriter, err := common.NewFileQueryWriter(censorConfiguration.ParseErrorsLog)
Expand Down
Loading