Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using AKS and EKS Terraform configuration directly with Epiphany. #2982

Closed
9 of 20 tasks
seriva opened this issue Feb 15, 2022 · 2 comments
Closed
9 of 20 tasks

Using AKS and EKS Terraform configuration directly with Epiphany. #2982

seriva opened this issue Feb 15, 2022 · 2 comments
Assignees
@przemyslavic @cicharka
Labels
priority/high Task with high priority

Comments

@seriva
Copy link
Collaborator

seriva commented Feb 15, 2022 •
edited
Loading

Is your spike related to a problem or idea? Please describe.
Currently we have modules to deploy EKS and AKS alongside an Epiphany cluster. As these Terraform deployments work separate from each other both have separate Terraform states making it hard to maintain the 2 configured together.

Describe the outcome you'd like
With the following issue completed: #2976 it should be possible to deploy custom Terraform templates alongside the ones Epicli generates. With this spike we want to deploy an Epiphany cluster and see if we can deploy AKS and EKS alongside it using custom Terraform which can be partially re-used from the modules:

Modules:
https://github.com/epiphany-platform/m-azure-kubernetes-service
https://github.com/epiphany-platform/m-aws-kubernetes-service

Note that this should also take into account the custom things we did with '''k8s_as_cloud_service''' to facilitate the original Epiphany+ modules deployment.

The outcome should preferable consist of the abstracted Terraform from the AWS and Azure modules, documentation how to deploy them. This can be a separate repo on our GitHub.

What is the reason or source for the spike
The module approach might not be something we want to continue to develop long-term. This approach should make it easier to integrate EKS/AKS with Epiphany and more maintainable from the Terraform perspective.

Additional context
None

DoD checklist

  • Changelog
    • updated
    • not needed
  • COMPONENTS.md
    • updated
    • not needed
  • Schema
    • updated
    • not needed
  • Backport tasks
    • created
    • not needed
  • Documentation
    • added
    • updated
    • not needed
  • Feature has automated tests
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
    • backup/restore
  • Idempotency tested
  • All conversations in PR resolved
  • Solution meets requirements and is done according to design doc
  • Usage compliant with license
@seriva seriva added the priority/high Task with high priority label Feb 16, 2022
@seriva seriva mentioned this issue Feb 17, 2022
Merged
@cicharka cicharka self-assigned this Apr 13, 2022
@cicharka cicharka mentioned this issue May 11, 2022
Merged
cicharka added a commit to cicharka/epiphany that referenced this issue May 13, 2022
@cicharka
k8s-modules: update documentation
7ac421b 
Result of spike: hitachienergy#2982

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>
@cicharka cicharka mentioned this issue May 13, 2022
Merged
cicharka added a commit to cicharka/epiphany that referenced this issue May 13, 2022
@cicharka
k8s-modules: update documentation
4bf1ddc 
Result of spike: hitachienergy#2982

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>
cicharka added a commit to cicharka/epiphany that referenced this issue May 13, 2022
@cicharka
k8s-modules: update documentation
0305cf0 
Result of spike: hitachienergy#2982

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>
@cicharka
Copy link
Contributor

As a result of this Spike, there are 3 PRs:

cicharka added a commit that referenced this issue May 18, 2022
@cicharka
[2.0.1] k8s-modules: update documentation (#3146)
8ee4a3f 
Result of spike: #2982

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>
@seriva
Copy link
Collaborator Author

seriva commented Jun 23, 2022

As a result of this Spike, there are 3 PRs:

4 PRs: hitachienergy/k8s-modules#2

@przemyslavic przemyslavic self-assigned this Jul 14, 2022
cicharka added a commit that referenced this issue Aug 3, 2022
@cicharka
Include aws-cli and git in Dockerfile #2982 (#3236)
818f1f5 
* Include aws-cli and git in Dockerfile
* add components entries

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>
seriva added a commit that referenced this issue Aug 9, 2022
@seriva @plirglo
@to-bar @sbbroot @przemyslavic @rafzei @cicharka @atsikham @tomasz-baran
Merge develop into 2.0 (#3250)
ad4c3d3 
* Document upgrade Red Hat / CentOS 7 to v 8.x (#3109)

* Migrate registry volume to named one (#3118)

* Fix dnf repoquery checks only latest kube* packages (#3123) (#3126)

* Switch to RHEL 8.4 for longer support (#3117) (#3129)

Co-authored-by: Tomasz Baran <46519524+to-bar@users.noreply.github.com>

* [develop] Fetch missing packages, add stderr handling   (#3132)

* Add handling stderr to repoquery and check if there are missing packages

* Skip missing_packages if dependencies, handle output of dnf download

* Unit tests fix

* Changes after review, Update libmodulemd to the latest

* Documentation update

* Remove duplicated run(), fix for offline mode

* Tests fix

* Lifecycle update (#3135)

* Lifecycle update

* Mark 1.2 as out of support

* Bump epicli version in develop (#3134)

Co-authored-by: przemyslavic <>

* Add policycoreutils package (#3139)

* Add allow_mismatch flag for ceph/ceph:v16.2.7 image (#3136) (#3138)

* [2.0.1] Filebeat upgrade to 7.12.1 (#3086)

* Filebeat update to 7.12.1

* Add missing tasks name in upgrade/filebeat.yml

* Update sha256

* Update changelogs

* Changelog change back to 2.0.1

Co-authored-by: przemyslavic <43173646+przemyslavic@users.noreply.github.com>

* [2.0.1] k8s-modules: update documentation (#3146)

Result of spike: #2982

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>

* Support 'epicli upgrade' for RHEL/AlmaLinux 8 (#3154)

* Upgrade only to RHEL 8.4

* Disable legacy containerd plugin to avoid instance auto-recovery on AWS

* Reboot system after update only when needed

* Update Leapp metadata file

* Enable yum repos after OS is updated

* Use target option

* Handle PostgreSQL packages

* Enable upgrade mode for RedHat OS family

* Add releasever parameter

* Fix update of libmodulemd package

* Remove releasever DNF variable

* Suspend HealthCheck process on AWS

* Install python3-psycopg2 package also for RedHat family

* Add ntsysv package for Azure

* Prevent auto-upgrade of repmgr10-4.0.6-1.el7

* Update changelog

* [2.0.1] Migration to OpenSearch (#3093)

* All in one commit - from PR #2983

* Ansible-lint adjustments

* Remove leftovers

* Tests fix

* Adjust download-requirements

* HA fix, improvements

* Fix defaults, schema

* Fixes in migration to opensearch and opensearch dashboards, add cleanup

* Improvements

* Changes after review

* Update doc, changelog and schema after review

* Spec tests update, rebase + changes after review

* Fix defaults

* Fix unittest

* Fix backup/restore

* Replace kibana with opensearch_dashboards

* Fix apply mode, cleanup, add opensearch spec test

* Disable upgrade of logging/opensearch, cleanup and rename vars

* [2.0.1] Add ARM architecture support for AlmaLinux 8.4 (#3151)

* merge ARM installation

* Add repositories ids

* Modify SHAs

* Merge in develop changes

* Add policycoreutils to packages list

* Add docs and lua package

* Modification after review

* Fix Config.py for supported architectures (#3175)

* [2.0.1] Allow temporary credentials (session token parameter) (#3076)

* filebeat: update template for new version (#3141)

* Fix after bumping up filebeat version (PR: #3086).
* related to k8s_as_cloud_service flag

* Source containerd version and allow downgrade (#3170)

* [2.0.1] Bumped Python packages (#3176)

* Bumping Python packages.

* Added changelog

* Added Sonarcloud status badges. (#3182)

* [2.0.1] Add sssd and dependencies to requirements (#3155)

* Add ssd packages needed to upgrade ssd to v2.6.2

* [2.0.1] Low hanging fruit SonarQube fixes. (#3183)

* SonarQube fixes

* [2.0.1] Fix `use_network_security_groups` is set to `false` (#3181)

* Fix `use_network_security_groups` is set to `false`
* SonarQube fix.
* Minor fix after review

* Ensure ca-certificates package is in the latest version (#3169)

* Ensure ca-certificates package is in the latest version

* Add tar to base packages for RHEL mode

* Ensure tar is not uninstalled too early

* Use constants instead of string literals

* Ignore non-critical DNF error

* Ensure dnf config-manager command

* Do not use constants for better readability

* Ensure epel repo is enabled

* Fix is_repo_enabled method

* Preserve epel-release package

* Remove accidental import

* Apply suggestions from code review

* Apply suggestions from 2nd review

* Fix `The same or higher version of epel-release is already installed` error

* Create a YAML build pipeline (#3187)

* Fix PostgreSQL tests (#3192)

* Fix postgresql tests

* Update default configurations

* Restore escaping for PostgreSQL tests (#3195)

* Ensure epicli upgrade works on cluster with upgraded RHEL from version 7 to 8 (#3191)

* Fix repmgr10 service

* Fix for K8s master with Calico

* Mark AWS instances as healthy

* Suspend ReplaceUnhealthy process

* Put all instances into Standby state and disable auto-recovery

* Keep ReplaceUnhealthy process suspended

* Remove unsupported Pylint options (#3197)

* Add ARM dependencies (#3185)

* Add ARM dependencies

* Add rook to unsupported roles

* Add FELIX_IPTABLESBACKEND variable to calico configuration

* Update documentation

* Add FELIX_IPTABLESBACKEND
for ARM only

* Remove rook from requirements

* Changelog: move ARM to 2.0.1

* [2.0.1] Adaptive mode for downloading requirements (#3188)

* Split available_roles and roles_mapping into separate yaml documents (#3097) (#3119)

* available_roles splitted into feature-mappings and features
  documents

* feature-mappings added to the Init by default

* Add manifest file parsing (#3105) (#3130)

* Add `-m/--manifest` flag to accept manifest.yml produced by
  `epicli init/prepare`

* Add `-v/--verbose` mode for printing out parsed manifest data

* Add ManifestReader class used for paring the manifest.yml file

* Move src/command/*.py to debian/redhat subdirs where needed

* Optimize Grafana dashboards downloading (#3131) (#3150)

* Optimize files downloading (#3116) (#3156)

* Add image-registry configuration reading (#3106) (#3159)

* Fix ansible-lint scan location (#3203)

* Fix ansible-lint scan location

* Update ansible_lint_error_threshold

* Allow excluding test groups (#3202)

* Add excluding test groups

* Exclude effective test groups

* Update doc

* Update configuration

* Do not use sets

* Apply suggestions from review

* Fix failed services after RHEL 7 upgrade on cluster created with epicli v1.3 (#3204)

* Fix failed services

* Fix for offline mode

* Workaround for esl-erlang package issue (#3211)

* Provide kubeconfig file for spec tests (#3206)

* Comply with Rubocop

* Print selected groups as yaml

* Provide kubeconfig file for spec tests

* Fix Pylint import-error issues in VSCode

* Self code review

* Add new option to launch configurations

* Fix after tests

* Update rubocop_linter_threshold

* Apply suggestions from code review

* Print selected test groups before preparing env

* Added checking enabled roles for feature. (#3213)

* ceph: fix tag for ceph image (#3199)

* Use a stable tag for the quay.io/ceph/ceph:v16.2.7 image

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>

* Added ability to disable OpenSearch audit logs (#3215)

* Added ability to disable OpenSearch audit logs

* Added Black Duck Scan plugin (#3219)

* Added Black Duck Scan plugin

* Add java to devcontainer to run BDS.

* Change to JAVA headless

* Cache is already cleaned, removed unneeded run.

* Support for original output coloring (#3220)

* Add click package

* Support for original output coloring

* Add click package to CI pipeline

* Fix CI task

* Use human friendly color codes

* Fix naming style

* Do not detect log level for colored loggers

* Apply --no-color option for epicli output formatter

* Highlight info on Ansible commands

* Fix UncolorJsonFormatter

* Update changelog

* Update pylint_score_cli_threshold

* Better formatting

* Add support for NO_COLOR env var

* Use 'python3 -m pip' instead of pip

* Ensure click

* Restore higher threshold

* Use python3 -m pylint

* Fix pylint_score_cli_threshold

* Resolve dependencies for specified package version (#3223)

* Fix issues reported by Pylint

* Resolve dependencies for specified version

* Update changelog

* Update crane to v0.11.0 (#3230)

* Fix disabling rook in feature-mappings (#3227)

* Run rook playbook on rook group

* Move rook images under rook group

* Update LIFECYCLE.md (#3235)

* Update LIFECYCLE.md

* Skip firewall role unless present in inventory (#3233)

* Update Calico and Canal to fix issue on ARM (#3228)

* Update Calico and Canal to fix issue on ARM

* Use single arch for CNI plugin images

* Fix incorrect checksums

* OpenSearch: add dedicated user for Filebeat (#3079) (#3221)

* removes previously used `logstash` user from filebeat configuration
* removes `logstash` user from demo users configured by opensearch
* enables creation of dedicated filebeat user - by default name set to
  `filebeatservice`
* add user detection in case of re-apply
* set user `kibanaserver` and `filebeatservice` installation
  dependent on inventory groups rather than user_active
  flag (previously configured by users)
* simplify documentation
* set dashboards hosts list based on their group
* use yaml anchors in user manipulation tasks

* Add filtering mechanism for the sensitive data (#3207) (#3208)

* Add filtering mechanism for the sensitive data (#3207)

* Include aws-cli and git in Dockerfile #2982 (#3236)

* Include aws-cli and git in Dockerfile
* add components entries

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>

* Fix getting package dependencies (#3239)

* Optimize get_package_dependencies method

* Update changelog

* Simplify method in CommandRunMock class

* Add test_get_package_dependencies_return_value

* Apply suggestions from code review

* Sort imports

* Move APT_CACHE_DEPENDS_DATA over test

* Fix k8s_as_cloud_service flag used in download-requirements (#3222) (#3242)

* Fix k8s_as_cloud_service flag used in download-requirements (#3222)

* filebeat: fix template for k8s_as_cloud_service (#3247)

Signed-off-by: cicharka <arkadiusz.cichon@outlook.com>

* k8s: controller managed attachment and detachment #3190 (#3237)

* enable configuration of enable-controller-attach-detach
  kubelet parameter in input manifest
* set enable-controller-attach-detach to true
* fix extend-kubeadm-config.yml task in order to keep
  consistent values in configMaps and kubeadm-config.yml
* move get and set cluster version utils to kubernetes_common
* Update docs/home/howto/kubernetes/PERSISTENT_STORAGE.md

* Remove leftovers of OpenDistro repository (#3248)

* Add haproxy to k8s images group (#3240)

* fix for #3231
* enhance test data for image_requirements
* add type hints

* Fix handling of download-requirements flag file (#3246)

Co-authored-by: Irek Głownia <48471627+plirglo@users.noreply.github.com>
Co-authored-by: Tomasz Baran <46519524+to-bar@users.noreply.github.com>
Co-authored-by: sbbroot <86356638+sbbroot@users.noreply.github.com>
Co-authored-by: przemyslavic <43173646+przemyslavic@users.noreply.github.com>
Co-authored-by: Rafal Zeidler <webdler@gmail.com>
Co-authored-by: cicharka <93913624+cicharka@users.noreply.github.com>
Co-authored-by: Anatoli Tsikhamirau <anatoliytihomirov@yahoo.com>
Co-authored-by: Tomasz Baran <110602076+tomasz-baran@users.noreply.github.com>
@seriva seriva removed the type/spike label Sep 15, 2022
@seriva seriva changed the title [SPIKE] Using AKS and EKS Terraform configuration directly with Epiphany. Using AKS and EKS Terraform configuration directly with Epiphany. Sep 15, 2022
@seriva seriva closed this as completed Sep 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@przemyslavic przemyslavic

@cicharka cicharka

Labels
priority/high Task with high priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@seriva @przemyslavic @cicharka