Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring Sync Things Job #2171

Closed
wants to merge 78 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
bf69a5d
Add AWS configuration in ConfigHandler (#1948)
delager Mar 22, 2023
3ae1037
#1793 Add AmazonIoTClient and AmazonIotDataClient configurations (#1951)
delager Mar 27, 2023
34899e1
#1925 & #1928 Specific config handler and throw error if no CloudProv…
delager Mar 30, 2023
477e62d
#1929 Configuration logic by provider (#1984)
delager Apr 7, 2023
2455fb3
1942 - Add Cloud Formation Linter check in CI (#2002)
delager Apr 11, 2023
5d716b5
#1978 Add Secrets Manager Client (#2003)
delager Apr 11, 2023
de372d7
#1942 CI Cloudformation update (#2009)
delager Apr 12, 2023
310622f
1943 task add a button to deploy aws infra (#2022)
delager Apr 15, 2023
b0a97e4
#1937 Azure dependency fix for AWS (#2031)
delager Apr 19, 2023
0ba0dd1
1935 as a developper i want to use amazon s3 to store and expose devi…
ssgueye2 Apr 21, 2023
e0147af
#1940 CloudFormation script creation: AWS S3, PostgreSQL DB & AWS App…
ssgueye2 May 4, 2023
449e91c
1802 task create an aws thing type (#2041)
ssgueye2 May 4, 2023
b48e769
Status check on main branch only
kbeaugrand May 4, 2023
2441330
Make code cove patch informational
kbeaugrand May 4, 2023
47d4142
Disable E2E tests
kbeaugrand May 4, 2023
44ac607
Add support for Device model in AWS through AWS thing types (#2050)
ssgueye2 May 11, 2023
36f0a7e
add AWS Greegrass infrastructure (#2061)
ssgueye2 May 11, 2023
bce25c2
Synchronize AWS Thing Types (#2062)
ssgueye2 May 16, 2023
4daefb6
Create the greengrass role in cloud formation script (#2080)
ssgueye2 May 16, 2023
8e36543
Fix Device model max age configuration handler (#2086)
kbeaugrand May 17, 2023
8649588
Update SDK version
kbeaugrand May 17, 2023
1efd07b
Fix AWS cloud formation template (#2087)
kbeaugrand May 17, 2023
f0b1cf9
Feature: Add cd pipeline to build and deploy to aws app runner (#1998)
hocinehacherouf May 17, 2023
4c11324
Story 2067: Refactor thing type (#2095)
hocinehacherouf May 22, 2023
e7161fa
Fix SyncThingTypesJob (#2098)
hocinehacherouf May 22, 2023
113008b
(CI) Update OIDC parameters in AWS staging environment
kbeaugrand May 23, 2023
7ea0be2
(CI) Disable LoraWAN features in AWS staging environment
kbeaugrand May 23, 2023
cc145cf
Update README
kbeaugrand May 23, 2023
e318794
Update README
kbeaugrand May 23, 2023
81f5957
Reformat template + Add internet gateway to the App Runner and VPC co…
kbeaugrand May 23, 2023
9f1820f
Add AWS Greengrass deployment support (#2081)
ssgueye2 May 23, 2023
671ffa6
Fix #2100 - Sync thing type depending on iotEdge tag (#2107)
kbeaugrand May 24, 2023
3215ed4
Create dynamic thing group for device models (#2111)
kbeaugrand May 24, 2023
5d39507
Add AWS thing support for IoT Devices (#2109)
delager May 25, 2023
5e4f428
#1921 Update AWS IoT Device (#2114)
delager May 25, 2023
87a0789
Fix #2117 (#2118)
kbeaugrand May 26, 2023
98476f7
#1922 Delete AWS IoT Device (#2119)
delager May 26, 2023
bd9e43c
Update and delete green grass deployment (#2113)
ssgueye2 May 26, 2023
bcb3e24
Add multi instance dev container (#2105)
kbeaugrand May 25, 2023
7814b08
Update CI for v5.0 release (#2124)
kbeaugrand May 30, 2023
2fc5554
Sync GreenGrassDeployment (#2120)
ssgueye2 May 31, 2023
534865f
Remove device status from form in AWS (#2127)
ssgueye2 May 31, 2023
3262bc6
Add Greengrass automagic deployment + Greengrass create and delete fr…
kbeaugrand May 31, 2023
f2efeab
Update the mysql schema to match the model (#2132)
kbeaugrand May 31, 2023
468bfff
Add exception if coud provider not correctly set
kbeaugrand May 31, 2023
5e27ccb
Fix Mysql migration script
kbeaugrand May 31, 2023
b720dc4
Use git sha to deploy staging env
kbeaugrand May 31, 2023
03856d9
Update nucleus version
kbeaugrand May 31, 2023
2a5acfc
CI - Update publish task to push to ECR as part of ACR
kbeaugrand Jun 1, 2023
1417ac6
CI - Update documentation publish to match the release branch name
kbeaugrand Jun 1, 2023
cdfd84a
Update AWS Cloud Formation Template
kbeaugrand Jun 1, 2023
380be2f
Sync AWS Things with portal (#2133)
delager Jun 1, 2023
afd9a86
Get AWS iot Edge Device (#2136)
ssgueye2 Jun 1, 2023
ba32640
Use certificate to authenticate devices and edge devices (#2143)
inkonito Jun 1, 2023
d2088a0
CI Push image to Public ECR
kbeaugrand Jun 1, 2023
222ed22
CI update documentation
kbeaugrand Jun 1, 2023
fafdc42
CI public to ECR Public registry should use us-east-1 region
kbeaugrand Jun 1, 2023
1942f99
Fix documentation publication
kbeaugrand Jun 1, 2023
da718d8
Fix template generation for AWS public ECR
kbeaugrand Jun 1, 2023
f37aadc
CI - Update documentation publish
kbeaugrand Jun 2, 2023
dd56346
see detail device model tooltip changed (#2149)
ssgueye2 Jun 2, 2023
bb8e567
detach associated certificate in thing (#2151)
ssgueye2 Jun 2, 2023
8ff19c8
Feature: Improve edge models for AWS (#2142)
hocinehacherouf Jun 4, 2023
d3236e6
Update deployement parameters (#2155)
kbeaugrand Jun 5, 2023
9801c70
CI - Update Staging deployment parameters
kbeaugrand Jun 5, 2023
4e3094c
Hide certificates when getting the magic ommand
kbeaugrand Jun 5, 2023
d137a75
Prefix Azure specific parameters (#2157)
ssgueye2 Jun 5, 2023
e94d835
Fix hide certificates when getting the magic command
kbeaugrand Jun 5, 2023
96b4e75
Add AWS__GreengrassCoreTokenExchangeRoleAliasName parameter to the de…
kbeaugrand Jun 5, 2023
232e9b5
Fix connections string screen for edge
kbeaugrand Jun 5, 2023
fca5d61
#2075 Sync AWS IoT Edge Device (#2148)
delager Jun 5, 2023
bc53093
Fix AWS template deployment
kbeaugrand Jun 5, 2023
d80f86b
Fix Edge connexion screen
kbeaugrand Jun 5, 2023
ca2141e
Fix Greengrass enrollment script token replacements
kbeaugrand Jun 5, 2023
9d7bce1
fix bug to delete secrets when removing edge device (#2165)
ssgueye2 Jun 5, 2023
5c32eb0
CI - Update AWS Stging + make LoraFeature disabled by default
kbeaugrand Jun 5, 2023
25be029
Fix remove policies from certificate when deleting edge device
kbeaugrand Jun 5, 2023
e764bd9
#2168 Refactoring Device & EdgeDevice sync job for AWS
delager Jun 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 29 additions & 17 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,18 +1,30 @@
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.238.0/containers/dotnet/.devcontainer/base.Dockerfile

# [Choice] .NET version: 6.0, 3.1, 6.0-bullseye, 3.1-bullseye, 6.0-focal, 3.1-focal
ARG VARIANT="6.0-bullseye-slim"
FROM mcr.microsoft.com/vscode/devcontainers/dotnet:0-${VARIANT}

# [Choice] Node.js version: none, lts/*, 18, 16, 14
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi

# [Optional] Uncomment this section to install additional OS packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# && apt-get -y install --no-install-recommends moby-engine moby-cli moby-build moby-run

# [Optional] Uncomment this line to install global node packages.
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.238.0/containers/dotnet/.devcontainer/base.Dockerfile

# [Choice] .NET version: 6.0, 3.1, 6.0-bullseye, 3.1-bullseye, 6.0-focal, 3.1-focal
ARG VARIANT="7.0-bullseye-slim"
FROM mcr.microsoft.com/vscode/devcontainers/dotnet:dev-${VARIANT}

# [Choice] Node.js version: none, lts/*, 18, 16, 14
ARG NODE_VERSION="none"
RUN if [ "${NODE_VERSION}" != "none" ]; then su vscode -c "umask 0002 && . /usr/local/share/nvm/nvm.sh && nvm install ${NODE_VERSION} 2>&1"; fi

# [Optional] Uncomment this section to install additional OS packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
# && apt-get -y install --no-install-recommends moby-engine moby-cli moby-build moby-run

# [Optional] Uncomment this line to install global node packages.
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1

RUN sudo apt-get update && \
sudo apt-get -y install ca-certificates curl gnupg && \
sudo install -m 0755 -d /etc/apt/keyrings && \
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \
sudo chmod a+r /etc/apt/keyrings/docker.gpg && \
echo \
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
sudo apt-get update && \
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

RUN dotnet dev-certs https
24 changes: 12 additions & 12 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,17 @@
// https://github.com/microsoft/vscode-dev-containers/tree/v0.238.0/containers/dotnet
{
"name": "C# (.NET)",
"build": {
"dockerfile": "Dockerfile",
"args": {
// Update 'VARIANT' to pick a .NET Core version: 3.1, 6.0
// Append -bullseye or -focal to pin to an OS version.
"VARIANT": "6.0-bullseye",
// Options
"NODE_VERSION": "lts/*"
}
},
// Update the 'dockerComposeFile' list if you have more compose files or use different names.
"dockerComposeFile": "docker-compose.yml",

// The 'service' property is the name of the service for the container that VS Code should
// use. Update this value and .devcontainer/docker-compose.yml to the real service name.
"service": "app",

// The 'workspaceFolder' property is the path VS Code should open by default when
// connected. Corresponds to a volume mount in .devcontainer/docker-compose.yml
"workspaceFolder": "/workspace",

// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
Expand All @@ -21,7 +22,6 @@
"ms-dotnettools.csharp",
"ms-dotnettools.vscode-dotnet-runtime",
"tintoy.msbuild-project-tools",
"jchannon.csharpextensions",
"fernandoescolar.vscode-solution-explorer"
]
}
Expand Down Expand Up @@ -51,7 +51,7 @@
// * If only using Remote - Containers with a local container, uncomment this line instead:
// "mounts": [ "source=${env:HOME}${env:USERPROFILE}/.aspnet/https,target=/home/vscode/.aspnet/https,type=bind" ],
// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "dotnet restore",
"postCreateCommand": "dotnet dev-certs https --trust && dotnet restore src/*.sln",
// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "vscode"
}
40 changes: 40 additions & 0 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
version: '3.4'

services:
database:
image: postgres:14-alpine
restart: always
environment:
POSTGRES_PASSWORD: postgrePassword
POSTGRES_DB: cgigeiotdemo
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "sh -c 'pg_isready -U postgres -d cgigeiotdemo'"]
interval: 10s
timeout: 5s
retries: 5

app:
build:
context: .
dockerfile: Dockerfile
# [Optional] You can use build args to set options. e.g. 'VARIANT' below affects the image in the Dockerfile
args:
NODE_VERSION: lts/*

volumes:
# This is where VS Code should expect to find your project's source code and the value of "workspaceFolder" in .devcontainer/devcontainer.json
- ..:/workspace:cached

# Uncomment the next line to use Docker from inside the container. See https://aka.ms/vscode-remote/samples/docker-from-docker-compose for details.
- /var/run/docker.sock:/var/run/docker.sock

# Overrides default command so things don't shut down after the process ends.
command: /bin/sh -c "while sleep 1000; do :; done"

depends_on:
database:
condition: service_healthy
volumes:
pgdata:
36 changes: 36 additions & 0 deletions .github/workflows/aws-cfn-lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Validate CloudFormation templates

on:
pull_request:
branches: [ main, main-vnext ]
paths:
- 'templates/aws/**'
push:
branches: [ main, main-vnext ]
paths:
- 'templates/aws/**'
workflow_dispatch:

jobs:
validate_cfn_templates:
name: Build and Validate CloudFormation Templates
runs-on: ubuntu-latest

steps:
- name: Apply cfn-templates label on PR
uses: actions/github-script@v6
if: ${{ github.event_name == 'pull_request' }}
with:
script: |
github.rest.issues.addLabels({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
labels: ['cfn-templates']
})
- uses: actions/checkout@v3.4.0

- name: Check CloudFormation Syntax
uses: scottbrenner/cfn-lint-action@v2
with:
command: cfn-lint templates/aws/*.yml
148 changes: 148 additions & 0 deletions .github/workflows/aws_deploy_staging.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
name: AWS Deploy Staging

on:
push:
branches: [ main-vnext ]
workflow_dispatch:

jobs:
build:
name: Build and push to ECR
runs-on: ubuntu-latest
environment:
name: AWS Staging
outputs:
aws_public_registry: ${{ steps.login-ecr-public.outputs.registry }}
steps:
- uses: actions/checkout@v3.4.0

- id: docker-tag
uses: yuya-takeyama/docker-tag-from-github-ref-action@v1

- name: Configure AWS credentials
id: aws-credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
# Should use us-east-1 region to push to ECR public registry
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Get a latest Git tag
uses: actions-ecosystem/action-get-latest-tag@v1
id: get-latest-tag
with:
semver_only: true

- name: Bump the semver version up
uses: actions-ecosystem/action-bump-semver@v1
id: bump-semver
with:
current_version: ${{ steps.get-latest-tag.outputs.tag }}
level: minor

- name: Remove leading v to semver
id: final-version
run: |
echo "::set-output name=new_version::$(echo ${{ steps.bump-semver.outputs.new_version }} | sed 's/^v//')"

- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
# list of Docker images to use as base name for tags
images: |
${{ steps.login-ecr.outputs.registry }}/${{ vars.AWS_ECR_REPOSITORY }}
tags: |
type=raw,enable=true,priority=200,prefix=,suffix=,value=${{ steps.final-version.outputs.new_version }}
type=sha,enable=true,priority=100,prefix=sha-,suffix=,format=short

- name: Build and push
# You may pin to the exact commit or the version.
uses: docker/build-push-action@v4.0.0
with:
# Build's context is the set of files located in the specified PATH or URL
context: src/
# Push is a shorthand for --output=type=registry
push: true
build-args: |
BUILD_VERSION=${{ steps.final-version.outputs.new_version }}
GITHUB_RUN_NUMBER=${{ github.run_number }}
tags:
${{ steps.meta.outputs.tags }}

deploy:
name: Deploy to AWS
runs-on: ubuntu-latest
needs: build
environment:
name: AWS Staging
steps:
- uses: actions/checkout@v3.4.0

- name: Configure AWS credentials
id: aws-credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ secrets.AWS_REGION }}

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1

- name: Set outputs
id: git_sha
run: echo "sha_short=$(git rev-parse --short HEAD)" >> $GITHUB_OUTPUT

- name: Deploy to App Runner Image
id: deploy-apprunner
uses: awslabs/amazon-app-runner-deploy@main
env:
ASPNETCORE_ENVIRONMENT: Development
CLOUDPROVIDER: AWS
PORTALNAME: ${{ vars.PORTALNAME }}
AWS__REGION: ${{ secrets.AWS_REGION }}
AWS__ACCOUNTID: ${{ secrets.AWS_ACCOUNTID }}
AWS__BUCKETNAME: ${{ vars.AWS_BUCKETNAME }}
OIDC__APICLIENTID: ${{ vars.OIDC_APICLIENTID }}
OIDC__CLIENTID: ${{ vars.OIDC_CLIENTID }}
OIDC__AUTHORITY: ${{ vars.OIDC_AUTHORITY }}
OIDC__METADATAURL: ${{ vars.OIDC_METADATAURL }}
OIDC__SCOPE: ${{ vars.OIDC_SCOPE }}
OIDC__VALIDATEAUDIENCE: ${{ vars.OIDC_VALIDATEAUDIENCE }}
AWS__GREENGRASSCORETOKENEXCHANGEROLEALIASNAME: "aurademoiot-GreengrassCoreTokenExchangeRoleAlias"
AWS__GREENGRASSREQUIREDROLES__0: "aurademoiot-GreengrassCoreTokenExchangeRoleAliasPolicy"
AWS__GREENGRASSREQUIREDROLES__1: "aurademoiot-GreengrassV2IoTThingPolicy"
with:
service: ${{ vars.AWS_APP_RUNNER_NAME }}
image: ${{ steps.login-ecr.outputs.registry }}/${{ vars.AWS_ECR_REPOSITORY }}:sha-${{ steps.git_sha.outputs.sha_short }}
access-role-arn: ${{ secrets.AWS_ROLE_ARN }}
region: ${{ secrets.AWS_REGION }}
cpu : 1
memory : 2
wait-for-service-stability-seconds: 1200
copy-env-vars: |
ASPNETCORE_ENVIRONMENT
CLOUDPROVIDER
PORTALNAME
AWS__REGION
AWS__ACCOUNTID
AWS__BUCKETNAME
OIDC__APICLIENTID
OIDC__CLIENTID
OIDC__AUTHORITY
OIDC__METADATAURL
OIDC__SCOPE
OIDC__VALIDATEAUDIENCE
AWS__GREENGRASSCORETOKENEXCHANGEROLEALIASNAME
AWS__GREENGRASSREQUIREDROLES__0
AWS__GREENGRASSREQUIREDROLES__1

- name: App Runner URL
run: echo "App runner URL ${{ steps.deploy-apprunner.outputs.service-url }}"
Loading