Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add packs.json to the build #26975

Merged
merged 8 commits into from
May 28, 2023
Merged

add packs.json to the build #26975

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
0d0e2aa
packs.json in build
May 24, 2023
b60de4a
onpush
May 24, 2023
92bfbd2
fix path
May 28, 2023
2d274cc
upload packs json to bucket
May 28, 2023
279e0ae
upload json
May 28, 2023
51ff5e7
fix
May 28, 2023
5f54411
added docstring
May 28, 2023
5d7d1e3
Merge branch 'master' into export_packs_json
ilaner May 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitlab/ci/bucket-upload.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
if [[ $TEST_UPLOAD == "false" ]]; then
section_start "Upload content graph GraphML to GCP" --collapsed
gsutil cp $ARTIFACTS_FOLDER/content_graph/$MARKETPLACE_VERSION.zip "gs://$GCS_MARKET_BUCKET_DEV/content_graph/$MARKETPLACE_VERSION.zip"
# copy the packs.json file to the bucket, used in contribution management
gsutil cp $ARTIFACTS_FOLDER/packs.json "gs://$GCS_MARKET_BUCKET_DEV/content_graph/"$MARKETPLACE_VERSION"_packs.json"
section_end "Upload content graph GraphML to GCP"
fi

Expand Down
4 changes: 2 additions & 2 deletions .gitlab/ci/on-push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,10 +149,10 @@ validate-content-conf:
if [[ $MARKETPLACE_VERSION == "xsoar" ]];
then
echo "Starting to create artifacts with zip for XSOAR."
python Tests/scripts/create_artifacts_graph/create_artifacts.py --marketplace "xsoar" --artifacts-output $ARTIFACTS_FOLDER/content_packs --dependencies-output $ARTIFACTS_FOLDER/packs_dependencies.json --bucket-upload "$BUCKET_UPLOAD"
python Tests/scripts/create_artifacts_graph/create_artifacts.py --marketplace "xsoar" --artifacts-output $ARTIFACTS_FOLDER/content_packs --dependencies-output $ARTIFACTS_FOLDER/packs_dependencies.json --packs-output $ARTIFACTS_FOLDER/packs.json --bucket-upload "$BUCKET_UPLOAD"
else
echo "Starting to create artifacts without zip."
python Tests/scripts/create_artifacts_graph/create_artifacts.py --marketplace "$MARKETPLACE_VERSION" --artifacts-output $ARTIFACTS_FOLDER/content_packs --dependencies-output $ARTIFACTS_FOLDER/packs_dependencies.json --no-zip --bucket-upload "$BUCKET_UPLOAD"
python Tests/scripts/create_artifacts_graph/create_artifacts.py --marketplace "$MARKETPLACE_VERSION" --artifacts-output $ARTIFACTS_FOLDER/content_packs --dependencies-output $ARTIFACTS_FOLDER/packs_dependencies.json --packs-output $ARTIFACTS_FOLDER/packs.json --no-zip --bucket-upload "$BUCKET_UPLOAD"
fi

- gcloud auth activate-service-account --key-file="$GCS_ARTIFACTS_KEY"
Expand Down
20 changes: 19 additions & 1 deletion Tests/scripts/create_artifacts_graph/create_artifacts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,26 @@ def create_dependencies(content_dto: ContentDTO, is_bucket_upload: bool, output:
json.dump(pack_dependencies, f, indent=4)


def create_packs_json(content_dto: ContentDTO, packs_output: Path):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a docstring

"""Create packs.json file, to be used by contribution management project

Args:
content_dto (ContentDTO): Content Repository DTO
packs_output (Path): Output path for packs.json
"""
packs = content_dto.packs
packs_json = {pack.object_id: json.loads(
pack.json(include={"name", "description", "author", "current_version"}, by_alias=True)) for pack in packs}
with open(packs_output, "w") as f:
json.dump(packs_json, f, indent=4)


def main():
parser = ArgumentParser()
parser.add_argument("-mp", "--marketplace", type=MarketplaceVersions, help="marketplace version", default="xsoar")
parser.add_argument("-ao", "--artifacts-output", help="Artifacts output directory", required=True)
parser.add_argument("-do", "--dependencies-output", help="Dependencies output directory", required=True)
parser.add_argument("-do", "--dependencies-output", help="Dependencies output file", required=True)
parser.add_argument("-po", "--packs-output", help="Packs json output file", required=True)
parser.add_argument(
"-bu", "--bucket-upload", help="Upload to bucket", type=lambda x: str2bool(x or False), default=False
)
Expand All @@ -70,6 +85,9 @@ def main():
logger.info("Creating pack dependencies mapping")
create_dependencies(content_dto, args.bucket_upload, Path(args.dependencies_output))

logger.info("Creating packs.json")
create_packs_json(content_dto, Path(args.packs_output))

logger.info("Creating content artifacts zips")
create_zips(content_dto, Path(args.artifacts_output), args.marketplace, args.zip)

Expand Down