Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Containerization, automatic deployment and Kubernetes #78

Open
5 of 19 tasks
vincentnam opened this issue Sep 1, 2021 · 0 comments
Open
5 of 19 tasks

Containerization, automatic deployment and Kubernetes #78

vincentnam opened this issue Sep 1, 2021 · 0 comments
Assignees
@vincentnam
Milestone
Automatic deploym...

Comments

@vincentnam
Copy link
Owner

'Automatic deployment :

First step will be to create docker container for each service to deploy and create Ansible playbooks to deploy automatically on VM.

Second step will be to make this deployment possible on Kubernetes cluster.

  • MongoDB container
  • Openstack Swift container (container exists, need to fit it to the install)
  • Apache Airflow container
  • Processed Data zone container (Apache Airflow and Apache Spark)
    • Apache Airflow (from Apache docker hub)
    • Apache Spark
  • Service zone container (Web GUI and REST api (Flask))
  • Security, authentication and monitoring container (Openstack Keystone, Kerberos (?), ... ?)
  • Ansible Playbooks
    • Base playbooks
    • Raw data zone playbooks
    • Metadata management zone playbooks
    • Process zone playbooks
    • Processed data zone playbooks
    • Service zone playbooks
    • Security, authentication and monitoring zone playbooks
  • Ansible playbook launcher

Later :

  • Import on Kubernetes cluster
  • Apache Spark container

A merge request is associated to this issue ; see https://gitlab.irit.fr/datalake/docker_datalake/-/merge_requests/67
@vincentnam vincentnam self-assigned this Sep 1, 2021
vincentnam pushed a commit that referenced this issue Oct 8, 2021
#78 - Feature/11503 backend download handled data
a1059f3
vincentnam pushed a commit that referenced this issue Oct 8, 2021
Merge branch 'feature/11503-backend-download-handled-data' into 'modi…
b36fe94 
…s_master'

#78 - Feature/11503 backend download handled data

See merge request datalake/docker_datalake!83
vincentnam pushed a commit that referenced this issue Apr 8, 2022
#78 - Fixed import module when download raw data
c5ae9aa
vincentnam pushed a commit that referenced this issue Apr 8, 2022
Merge branch 'hotfix/raw-data-fix-import-module' into 'modis_master'
99c9f33 
#78 - Fixed import module when download raw data

See merge request datalake/docker_datalake!127
vincentnam pushed a commit that referenced this issue Apr 8, 2022
#78 - fixed filters in backend to download processed data
986902e
vincentnam pushed a commit that referenced this issue Apr 8, 2022
@TheoLGG-Modis
Merge branch 'hotfix/feature-11503-download-processed-data' into 'mod…
b485961 
…is_master'

#78 - fixed filters in backend to download processed data

See merge request datalake/docker_datalake!140
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vincentnam vincentnam

Labels
None yet
Projects
None yet
Milestone
Automatic deployment and containerisa...
Development

No branches or pull requests
1 participant
@vincentnam