Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge private and public id sets #10189

Merged
merged 139 commits into from
Jan 7, 2021
Merged

Merge private and public id sets #10189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
441ed58
download private_id_set script, add to the build
ChanochShayner Nov 30, 2020
456524b
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Nov 30, 2020
85f5641
fix file name
ChanochShayner Nov 30, 2020
30d7d86
fix file name
ChanochShayner Nov 30, 2020
ea115c1
fix file name
ChanochShayner Nov 30, 2020
5c7cc2b
fix file name
ChanochShayner Nov 30, 2020
e84cacb
fix file name
ChanochShayner Nov 30, 2020
543def3
some tests
ChanochShayner Nov 30, 2020
37dfb5d
some tests
ChanochShayner Dec 1, 2020
b84735f
add service account
ChanochShayner Dec 1, 2020
0af07dc
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 1, 2020
b86bc13
add service account
ChanochShayner Dec 1, 2020
bf13451
add GOOGLE_APPLICATION_CREDENTIALS
ChanochShayner Dec 1, 2020
7ca513d
add artifacts
ChanochShayner Dec 1, 2020
1e82dd0
add artifacts
ChanochShayner Dec 1, 2020
a700fdc
xsoar-ci-artifacts
ChanochShayner Dec 1, 2020
b464b11
merge id sets
ChanochShayner Dec 1, 2020
a64316b
merge id sets
ChanochShayner Dec 1, 2020
6584fd3
change sdk to use merge id sets command
ChanochShayner Dec 1, 2020
2defc42
change sdk to use merge id sets command
ChanochShayner Dec 1, 2020
602d04b
change sdk to use merge id sets command
ChanochShayner Dec 1, 2020
c25f7bc
add prepare_private_id_set_for_merge and upload_private_id_set scripts
ChanochShayner Dec 3, 2020
6b939a1
add test files
ChanochShayner Dec 7, 2020
4a38b4c
add test
ChanochShayner Dec 7, 2020
c4b16bc
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 7, 2020
3386bc0
add test
ChanochShayner Dec 7, 2020
8dccad4
fix test
ChanochShayner Dec 7, 2020
81a2470
fix test
ChanochShayner Dec 7, 2020
27add4d
fix test
ChanochShayner Dec 7, 2020
cd9867e
remove test
ChanochShayner Dec 7, 2020
2616525
add test, (remove changes from build)
ChanochShayner Dec 8, 2020
429a763
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 8, 2020
7e9209b
fix test and validation
ChanochShayner Dec 8, 2020
4b41665
fix path
ChanochShayner Dec 8, 2020
74dddb1
try another bucket
ChanochShayner Dec 8, 2020
7d3455a
try another file
ChanochShayner Dec 8, 2020
15eb957
some test
ChanochShayner Dec 8, 2020
7faf179
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 8, 2020
e3a7fcf
some test
ChanochShayner Dec 8, 2020
10dec35
some test
ChanochShayner Dec 8, 2020
7e9e4de
service_account
ChanochShayner Dec 8, 2020
5059df6
public_and_private_id_sets file
ChanochShayner Dec 8, 2020
c346338
remove public_and_private_id_sets file
ChanochShayner Dec 8, 2020
17a06c2
remove cp to artifacts
ChanochShayner Dec 8, 2020
c46459b
remove cp to artifacts
ChanochShayner Dec 8, 2020
a29c9e2
$CIRCLE_ARTIFACTS
ChanochShayner Dec 8, 2020
6640b35
add file
ChanochShayner Dec 8, 2020
7062a81
merge to id_set file
ChanochShayner Dec 8, 2020
6a9d8e5
CR changes
ChanochShayner Dec 8, 2020
ad22614
CR changes
ChanochShayner Dec 9, 2020
aa8e962
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 9, 2020
294c329
fix validation
ChanochShayner Dec 9, 2020
1d03f81
fix validation
ChanochShayner Dec 9, 2020
ee2ce46
change private id set path in bucket
ChanochShayner Dec 9, 2020
f4a9efe
fix test, try download with gsutil
ChanochShayner Dec 9, 2020
557c5c0
try download with gsutil
ChanochShayner Dec 9, 2020
edaad7e
try download with gsutil
ChanochShayner Dec 9, 2020
ad235d8
try download with gsutil
ChanochShayner Dec 9, 2020
79cfd88
download with script
ChanochShayner Dec 9, 2020
363494a
try to decode gcs creds
ChanochShayner Dec 9, 2020
4337e18
try with gsutil
ChanochShayner Dec 9, 2020
a7cc7b1
try with gsutil
ChanochShayner Dec 10, 2020
c2a34d6
add file unified_id_sets
ChanochShayner Dec 10, 2020
43c51eb
remove file
ChanochShayner Dec 10, 2020
6be5e9d
change path of unified id sets
ChanochShayner Dec 10, 2020
930ec19
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 10, 2020
b9b4055
replace sdk branch
ChanochShayner Dec 10, 2020
99de447
add id_set_private_test file to bucket
ChanochShayner Dec 10, 2020
9e97b74
remove scripts use gsutil instead
ChanochShayner Dec 10, 2020
36aad69
use another branch
ChanochShayner Dec 10, 2020
271e3c3
remove script from this PR
ChanochShayner Dec 13, 2020
1aeaa24
add script and test
ChanochShayner Dec 13, 2020
4f0ae20
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 14, 2020
85f684e
remove ref for demisto sdk
ChanochShayner Dec 22, 2020
d56865d
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 22, 2020
37ae33f
some fixes
ChanochShayner Dec 22, 2020
26c0d55
some fixes
ChanochShayner Dec 22, 2020
1702e94
fix artifacts
ChanochShayner Dec 22, 2020
10b2d0c
unified id set file
ChanochShayner Dec 22, 2020
81a12ed
fixes
ChanochShayner Dec 22, 2020
c2c4da8
fixes
ChanochShayner Dec 22, 2020
9ff4fcc
open new file
ChanochShayner Dec 22, 2020
6bfa1df
fixes
ChanochShayner Dec 22, 2020
05d1385
fixes
ChanochShayner Dec 22, 2020
cb581a2
fixes
ChanochShayner Dec 22, 2020
6fab940
try to download with script
ChanochShayner Dec 22, 2020
30fcb5b
some fixes
ChanochShayner Dec 22, 2020
39e28b0
some fixes
ChanochShayner Dec 22, 2020
e122e47
ref sdk
ChanochShayner Dec 22, 2020
e40035a
fixes
ChanochShayner Dec 22, 2020
4d2064a
comment
ChanochShayner Dec 22, 2020
ec18775
remove ref
ChanochShayner Dec 22, 2020
a93c700
fixes
ChanochShayner Dec 22, 2020
b839394
try with prints
ChanochShayner Dec 23, 2020
5adffb9
add ref
ChanochShayner Dec 23, 2020
7877a1a
comment
ChanochShayner Dec 23, 2020
17ab653
comment
ChanochShayner Dec 23, 2020
0d23701
comment
ChanochShayner Dec 23, 2020
1a126ff
comment
ChanochShayner Dec 23, 2020
17b8a13
remove ref
ChanochShayner Dec 23, 2020
a55431e
comment
ChanochShayner Dec 23, 2020
c3e9d99
ref
ChanochShayner Dec 23, 2020
b35a95e
cache
ChanochShayner Dec 23, 2020
cc16ab2
comment
ChanochShayner Dec 23, 2020
6db71c7
try to remove duplicates
ChanochShayner Dec 23, 2020
a018a80
fix duplicate id
ChanochShayner Dec 23, 2020
0ff9d78
fix duplicate ID issue
ChanochShayner Dec 24, 2020
6280a04
add check that id sets merge successfully
ChanochShayner Dec 24, 2020
c7e9eec
add check that id sets merge successfully
ChanochShayner Dec 24, 2020
2d0d758
fix duplicate
ChanochShayner Dec 24, 2020
90339a9
add check in sdk command
ChanochShayner Dec 24, 2020
ea85d74
comment
ChanochShayner Dec 24, 2020
d99a196
add duplicates for test
ChanochShayner Dec 24, 2020
b8f8264
fix duplicates for test
ChanochShayner Dec 24, 2020
d977b6a
duplicates for test
ChanochShayner Dec 24, 2020
c7e6744
typo
ChanochShayner Dec 24, 2020
6494c8d
fix duplicates - ready
ChanochShayner Dec 24, 2020
e63c725
fix ref
ChanochShayner Dec 27, 2020
b7748f0
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 28, 2020
4f5ee59
unified ID set just in validate
ChanochShayner Dec 30, 2020
1a05b41
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 30, 2020
ac6d77e
check with ref sdk
ChanochShayner Dec 30, 2020
2cc6686
add id set path to validate script
ChanochShayner Dec 30, 2020
8e54f7e
comment
ChanochShayner Dec 30, 2020
9219640
comment
ChanochShayner Dec 30, 2020
9e0df43
ref sdk to master
ChanochShayner Dec 31, 2020
9af35de
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 31, 2020
756bc16
add artifacts to run validation step
ChanochShayner Dec 31, 2020
a67850e
add merge to create step
ChanochShayner Dec 31, 2020
316963f
fixes
ChanochShayner Dec 31, 2020
4e00343
fixes
ChanochShayner Dec 31, 2020
2c5cdb7
ref to sdk validate mappers
ChanochShayner Dec 31, 2020
f0281e3
comment
ChanochShayner Dec 31, 2020
89fbd8d
add step for merge id sets
ChanochShayner Dec 31, 2020
c752ce9
remove changes
ChanochShayner Dec 31, 2020
feec77f
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Dec 31, 2020
6cbf369
remove changes
ChanochShayner Dec 31, 2020
46209f6
CR changes
ChanochShayner Jan 6, 2021
d7d9859
Merge branch 'master' into prepare_paid_packs_to_create_id_set
ChanochShayner Jan 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,23 @@ references:
demisto-sdk create-id-set -o ./Tests/id_set.json
cp ./Tests/id_set.json $CIRCLE_ARTIFACTS

merge_public_and_private_id_sets: &merge_public_and_private_id_sets
run:
name: Merge public and private ID sets
when: always
command: |
GCS_PATH=$(mktemp)
echo "$GCS_MARKET_KEY" > "$GCS_PATH"
gcloud auth activate-service-account --key-file="$GCS_PATH" > auth.out 2>&1

# Download private ID set
gsutil cp gs://marketplace-dist/content/private_id_set.json $CIRCLE_ARTIFACTS/unified_id_set.json
gcloud auth revoke --all
rm $GCS_PATH

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
gcloud auth revoke --all
rm $GCS_PATH

It's important to also revoke the gcloud credentials.
In addition, it's important to do it before the sdk command because in case the sdk command fails - the step will finish immediately and the credentials will not be cleaned.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

# Merge public and private ID sets
demisto-sdk merge-id-sets -i1 ./Tests/id_set.json -i2 $CIRCLE_ARTIFACTS/unified_id_set.json -o $CIRCLE_ARTIFACTS/unified_id_set.json

build_content_descriptor: &build_content_descriptor
run:
name: Build Content Descriptor
Expand Down Expand Up @@ -545,6 +562,7 @@ jobs:
- *prepare_environment
- *secrets
- *create_id_set
- *merge_public_and_private_id_sets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The references syntax is mostly useful for steps we want to use more than once, since this step is only used once - I think it can be written here directly without the pointer, but it's up to you.

- *update_tests_step
- *validate_files_and_yaml
- run:
Expand Down
5 changes: 2 additions & 3 deletions Tests/scripts/validate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@ echo "CIRCLE_BRANCH: $CIRCLE_BRANCH CI: $CI DEMISTO_README_VALIDATION: $DEMISTO_

if [[ $CIRCLE_BRANCH = master ]] || [[ -n "${NIGHTLY}" ]] || [[ -n "${BUCKET_UPLOAD}" ]];
then
demisto-sdk validate -a
demisto-sdk validate -a --id-set-path $CIRCLE_ARTIFACTS/unified_id_set.json
else
demisto-sdk validate -g --post-commit
demisto-sdk validate -g --post-commit --id-set-path $CIRCLE_ARTIFACTS/unified_id_set.json
fi