Arc support #4

mikkelhegn · 2021-06-10T10:46:31Z

Resolves #2

This PR contains the end-to-end support for deploying web and data services to Arc.

Bicep, install scripts and deployment workflows were updated to support Azure and arc paths. To ensure timing of database availability on Arc when migration scripts and the webapi starts, this also includes retry logic when connecting to the database. This is needed as the Azure deployment succeeds before the DB is guaranteed to be up and running in the K8s cluster. The deploy logic doesn't include checking the status of deployments beyond ARM.

Multiple bicep files for Azure and Arc are needed for two reasons:

For the database, it's different ARM resource types
For the web app, some properties are required for Arc, but will not pass validation if they are empty strings

jongio · 2021-06-24T14:01:00Z

deploy/infra/main.bicep

+    appSettingsPgUser: customLocationId == '' ? dbAzure.outputs.pgUser : dbArc.outputs.pgUser //The ternary operator is needed to ensure deployment dependencies are correct
+    appSettingsPgDb: customLocationId == '' ? dbAzure.outputs.pgDb : dbArc.outputs.pgDb //The ternary operator is needed to ensure deployment dependencies are correct
+    appSettingsNodeEnv: webapiNodeEnv
+    appSettingsPgPassword: postgresAdminPassword


spell out postgres in the name?

I see a combo of pg and postgres. Maybe standardize on one? It's a p3

There is lack of consistensy with the eventual env var being used in code. That one, by convention for postgres uses PGHOST, PGDB etc. Too many cascading changes to make this in this pr.

deploy/infra/webapi/dbArc.bicep

deploy/infra/webapi/webappArc.bicep

jongio · 2021-06-24T14:07:00Z

docs/azure_arc.md

+  - [Regions and resource group support](#regions-and-resource-group-support)
+    - [Azure Arc enabled App Service](#azure-arc-enabled-app-service)
+
+## Prerequisites


Can we not show them the az cli commands to get this setup?

Can we not provision/configure with bicep?

It's a multitude:

Create a K8s cluster (anywhere)

Arc-enable (connect)

Add extensions (data and app)

Create custom location

The linked article describes the concepts and the pointers to docs. I'm usin gthis script for an environment on AKS:https://github.com/mikkelhegn/arc. The Arc model has this concept of custom locations, which are mapped to namespaces in a K8 cluster. My assumption is that a platform team will provide the namespace (custom location) for the app team, same way as a subscription would be provided by someone else for Azure (in an Enterprise setup). Scope-wise, I belive this is out for the template.

docs/azure_arc.md

jongio · 2021-06-24T14:08:58Z

docs/azure_arc.md

+KUBE_ENVIRONMENT_ID: ""             #kubeEnvironmentId to host the webapi on Arc
+CUSTOM_LOCATION_ID: ""              #customLocationId for the kubeEnvironment
+DATA_CONTROLLER_ID: ""              #dataControllerId


Where does the user find this info?

Good callout. They are found on the individual resources (KubeEnvironment, CustomLocation and DataController), these are the resources in ARM, which are tied to the Arc-enabled K8s cluster. I've added some guidance above.

What CLI or script commands can they run to get this info? I'm looking for ways to help them get the info quickly

Think of this similar to finding a subscription Id, if you didn't create it you have to ask someone. At least they need to get the customlocation resource Id from whomever created that. This is really a Kubernetes namespace on a Kubernetes cluster somewhere, if you installed it, you know the customer location and can retrieve it's Id. - or do 'az customlocation show -g -n '.
Based on that they can get the other Ids using the graph (my new favorite thing :-)):
'az graph query -q "where (type =~ 'Microsoft.Web/kubeEnvironments' or type =~ 'microsoft.azurearcdata/datacontrollers') and extendedLocation.name == '' | where subscriptionId == '' | project name, id"'

jongio · 2021-06-24T14:09:49Z

docs/azure_arc.md

+./deploy/scripts/install.sh \
+   --resource-name-prefix <resource group name> \
+   --environment-tag <name tag for all resources> \
+   --resource-group-tag <tag> \
+   --location <location (eastus or westeurope)> \
+   --node-env development \
+   --kube-environment-id <kubeEnvironmentId> \
+   --custom-location-id <customLocationId> \
+   --data-controller-id <dataControlleId>


Do we think deploy.sh is a better name than install?

It might be, but it's already in-use in the existing template. I believe renaming woudl be a new work item, if we want to do that.

jongio · 2021-06-25T14:27:26Z

.github/workflows/build_release.yaml

        run: npm run package --prefix services/webapi

+        # Currently needed as ORYX builds are not working on Arc
+      - name: Package application with node_modules for Arc deployment


Does this run for non-arc builds as well? Any side-effects on running this in normal mode if so?

We're producing two artifacts for the app, one without node_modules, and one with. The one is needed for arc, the other can be used in Azure so that modules will be compiled on the platform where it runs. The impact of always building both is storage taken up in GitHub artifacts. We could look into doing this conditionally if you would like that?

jongio · 2021-06-25T14:28:58Z

deploy/infra/main.bicep

+    appSettingsPgHost: customLocationId == '' ? dbAzure.outputs.pgHost : dbArc.outputs.pgHost //The ternary operator is needed to ensure deployment dependencies are correct
+    appSettingsPgUser: customLocationId == '' ? dbAzure.outputs.pgUser : dbArc.outputs.pgUser //The ternary operator is needed to ensure deployment dependencies are correct
+    appSettingsPgDb: customLocationId == '' ? dbAzure.outputs.pgDb : dbArc.outputs.pgDb //The ternary operator is needed to ensure deployment dependencies are correct


If we have different bicep files for arc vs regular, then why do we need this conditional?

Bicep produces individual deployments for each module. If we don't have this ternary operator, bicep will start referencing deployments, which have not been run and the ARM deployment will fail. According to the Bicep team, this is on the backlog to be fixed. I 'think' this is the issue we're running in to, where the proposed fix is the ternary operator inside the deployment scope: Azure/bicep#1876 (comment)

jongio · 2021-06-25T14:30:18Z

deploy/scripts/install.sh

+  --kube-environment-id <kube environment id>
+    Provide the kube environment id to deploy to Arc
+  --custom-location-id <custom location id>
+    Provide the custome location id to deploy to Arc
+  --data-controller-id <data controller id>
+    Provide the ResourceId of the Data Controller


It would be helpful to mention that these are only needed if deploying to Arc.

jongio · 2021-06-25T14:31:31Z

docs/azure_arc.md

+KUBE_ENVIRONMENT_ID: ""             #kubeEnvironmentId to host the webapi on Arc
+CUSTOM_LOCATION_ID: ""              #customLocationId for the kubeEnvironment
+DATA_CONTROLLER_ID: ""              #dataControllerId


What CLI or script commands can they run to get this info? I'm looking for ways to help them get the info quickly

Mikkel Mork Hegnhoj added 5 commits May 25, 2021 17:18

webapi arc support

999050e

arc doc updates

72eef48

Adding retries to connect to db

9483751

npm audit update

ef710ac

arc with db deployment

33a7a98

mikkelhegn requested review from jongio and philliphoff June 10, 2021 10:46

jongio suggested changes Jun 24, 2021

View reviewed changes

review update

ce216ff

jongio reviewed Jun 25, 2021

View reviewed changes

Respondgin to comments

f5ebbef

mikkelhegn closed this Nov 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arc support #4

Arc support #4

mikkelhegn commented Jun 10, 2021

jongio Jun 24, 2021

jongio Jun 24, 2021

mikkelhegn Jun 25, 2021 •

edited

Loading

jongio Jun 24, 2021

mikkelhegn Jun 25, 2021

jongio Jun 24, 2021

mikkelhegn Jun 25, 2021

jongio Jun 25, 2021

mikkelhegn Jun 28, 2021

jongio Jun 24, 2021

mikkelhegn Jun 25, 2021 •

edited

Loading

jongio Jun 25, 2021

mikkelhegn Jun 28, 2021

jongio Jun 25, 2021

mikkelhegn Jun 28, 2021

jongio Jun 25, 2021

mikkelhegn Jun 28, 2021

jongio Jun 25, 2021

Arc support #4

Arc support #4

Conversation

mikkelhegn commented Jun 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkelhegn Jun 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkelhegn Jun 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikkelhegn Jun 25, 2021 •

edited

Loading

mikkelhegn Jun 25, 2021 •

edited

Loading