Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig #515

chenriksson · 2018-08-03T21:16:28Z

Part of https://github.com/NuGet/Engineering/issues/1560. Depends on #505

Includes the following:

Moves bulk of JsonConfigurationJob into NuGet.Jobs.Common, for use by legacy jobs
Updates validation jobs use db helpers from Jobs.Common helpers for NuGet.Services.Sql #505
- JsonConfigurationJob registers databases when GalleryDbConfiguration or ValidationDbConfiguration is found
Updates ArchivePackages.Job to descend from JsonConfigurationJob
- Requires new "Configuration" and "GalleryDatabaseAddress" Octopus config. Will remove old config only after these changes RI'd to master and deployed to prod.

Testing should include the following jobs:

ArchivePackages (Verified on DEV)
Revalidate
ProcessSignature
RebootSearchInstance
SubscriptionProcessor
RevalidateCertificate

chenriksson · 2018-08-03T21:18:38Z

src/ArchivePackages/ArchivePackages.Job.cs


-            Source = CloudStorageAccount.Parse(
-                        JobConfigurationManager.GetArgument(jobArgsDictionary, JobArgumentNames.Source));
+            Configuration = _serviceProvider.GetRequiredService<InitializationConfiguration>();


All command-line options for legacy jobs will be moved to a custom InitializationConfiguration, from json config.

chenriksson · 2018-08-03T21:19:30Z

src/ArchivePackages/ArchivePackages.Job.cs

-        public string Version { get; set; }
-        public string Hash { get; set; }
-        public DateTime? LastEdited { get; set; }
-        public DateTime? Published { get; set; }


Cleanup - just moving classes into separate source files.

chenriksson · 2018-08-03T21:20:17Z

src/ArchivePackages/ArchivePackages.csproj

@@ -75,16 +82,14 @@
    <PackageReference Include="Newtonsoft.Json">
      <Version>9.0.1</Version>
    </PackageReference>
-    <PackageReference Include="NuGet.Services.Sql">
-      <Version>2.27.0</Version>
-    </PackageReference>


We only need direct dependency from Jobs.Common now.

chenriksson · 2018-08-03T21:21:09Z

src/ArchivePackages/Scripts/ArchivePackages.cmd

@@ -7,7 +7,7 @@ cd bin

 	title #{Jobs.archivepackages.Title}

-    start /w archivepackages.exe -VaultName "#{Deployment.Azure.KeyVault.VaultName}" -ClientId "#{Deployment.Azure.KeyVault.ClientId}" -CertificateThumbprint "#{Deployment.Azure.KeyVault.CertificateThumbprint}" -PackageDatabase "#{Jobs.archivepackages.PackageDatabase}" -Source "#{Jobs.archivepackages.Source}" -PrimaryDestination "#{Jobs.archivepackages.PrimaryDestination}" -SecondaryDestination "#{Jobs.archivepackages.SecondaryDestination}" -Sleep "#{Jobs.archivepackages.Sleep}" -InstrumentationKey "#{Jobs.archivepackages.ApplicationInsightsInstrumentationKey}"
+	start /w archivepackages.exe -Configuration "#{Jobs.archivepackages.Configuration}"


Will add "-Configuration" octopus config now, and remove old config after RI and prod deployments are complete.

chenriksson · 2018-08-03T21:23:29Z

src/NuGet.Jobs.Common/JsonConfigurationJob.cs


-namespace NuGet.Jobs.Validation
+namespace NuGet.Jobs
 {
    public abstract class JsonConfigurationJob : JobBase


Old JsonConfigurationJob is now split into Jobs.Common > JsonConfigurationJob and Validation.Common.Job > ValidationJobBase, based on needs of legacy / validation jobs.

chenriksson · 2018-08-03T21:31:39Z

src/ArchivePackages/Settings/dev.json

+  "KeyVault_ValidateCertificate": true,
+  "KeyVault_StoreName": "My",
+  "KeyVault_StoreLocation": "LocalMachine"
+}


Job config could eventually move to NuGetDeployment, as we do with Orchestrator. For now, I don't want to modify build / deployment infrastructure.

joelverhagen · 2018-08-03T23:55:48Z

src/ArchivePackages/Scripts/ArchivePackages.cmd

@@ -7,7 +7,7 @@ cd bin

 	title #{Jobs.archivepackages.Title}

-    start /w archivepackages.exe -VaultName "#{Deployment.Azure.KeyVault.VaultName}" -ClientId "#{Deployment.Azure.KeyVault.ClientId}" -CertificateThumbprint "#{Deployment.Azure.KeyVault.CertificateThumbprint}" -PackageDatabase "#{Jobs.archivepackages.PackageDatabase}" -Source "#{Jobs.archivepackages.Source}" -PrimaryDestination "#{Jobs.archivepackages.PrimaryDestination}" -SecondaryDestination "#{Jobs.archivepackages.SecondaryDestination}" -Sleep "#{Jobs.archivepackages.Sleep}" -InstrumentationKey "#{Jobs.archivepackages.ApplicationInsightsInstrumentationKey}"
+	start /w archivepackages.exe -Configuration "#{Jobs.archivepackages.Configuration}"


Oh nice, so instrumentation key now comes through JSON configuration? I think some of the validation jobs need this fix right?

Reverted to keep -Sleep and -InstrumentationKey in Octopus... wasn't plumbed through to the JobRunner. Can investigate more once I move onto job deployments.

It does look like the Revalidate job has an "AppInsights" section, which could probably be extracted into Jobs.Common and reused.

chenriksson · 2018-08-04T00:55:21Z

src/ArchivePackages/Settings/dev.json

+  },
+
+  "GalleryDb": {
+    "ConnectionString": "Data Source=tcp:#{Jobs.validation.GalleryDatabaseAddress};Initial Catalog=nuget-dev-0-v2gallery;Integrated Security=False;User ID=$$Dev-GalleryDBReadOnly-UserName$$;Password=$$Dev-GalleryDBReadOnly-Password$$;Connect Timeout=30;Encrypt=True"


Will fix GalleryDatabaseAddress prefix

chenriksson · 2018-08-04T01:00:29Z

src/Validation.Common.Job/ValidationJobBase.cs

+            base.ConfigureDefaultJobServices(services, configurationRoot);
+
+            //services.AddSingleton(new TelemetryClient());
+            //services.AddTransient<ITelemetryClient, TelemetryClientWrapper>();


Will remove and keep in JsonConfigurationJob

agr

Address NuGet/NuGetGallery#6475

… JsonConfig (#515)

Address NuGet/NuGetGallery#6475

chenriksson commented Aug 3, 2018

View reviewed changes

chenriksson requested review from joelverhagen, agr, loic-sharma, shishirx34, ryuyu, cristinamanum, skofman1, scottbommarito and dtivel August 3, 2018 21:24

chenriksson commented Aug 3, 2018

View reviewed changes

chenriksson force-pushed the chenriks-jobscommon-sql branch from d41763d to 10561f4 Compare August 3, 2018 21:58

chenriksson changed the title ~~Move JsonConfigurationJob to Jobs.Common~~ Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig Aug 3, 2018

chenriksson force-pushed the chenriks-jobscommon-config branch from 3eb249c to d40a88f Compare August 3, 2018 22:08

joelverhagen reviewed Aug 3, 2018

View reviewed changes

joelverhagen approved these changes Aug 3, 2018

View reviewed changes

chenriksson commented Aug 4, 2018

View reviewed changes

chenriksson mentioned this pull request Aug 6, 2018

Move initial jobs to SQL AAD on DEV #519

Merged

agr approved these changes Aug 6, 2018

View reviewed changes

chenriksson added 4 commits August 7, 2018 05:48

Move JsonConfigurationJob to Jobs.Common

969e604

PR feedback and cleanup

66c098e

Include *.json in archive

d6712f3

cleanup

1583750

chenriksson force-pushed the chenriks-jobscommon-config branch from c63684d to 1583750 Compare August 7, 2018 12:49

chenriksson changed the base branch from chenriks-jobscommon-sql to dev August 7, 2018 12:49

Use Octopus common config

e0daa1a

chenriksson merged commit 8b54901 into dev Aug 7, 2018

chenriksson deleted the chenriks-jobscommon-config branch August 8, 2018 22:37

joelverhagen added a commit that referenced this pull request Sep 27, 2019

Complete Owners2AzureSearch job and wire-up the entry point (#515)

4a93f62

Address NuGet/NuGetGallery#6475

joelverhagen pushed a commit that referenced this pull request Oct 26, 2020

Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to…

cb2e767

… JsonConfig (#515)

joelverhagen added a commit that referenced this pull request Oct 26, 2020

Complete Owners2AzureSearch job and wire-up the entry point (#515)

847659b

Address NuGet/NuGetGallery#6475

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig #515

Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig #515

chenriksson commented Aug 3, 2018 •

edited

Loading

chenriksson Aug 3, 2018

chenriksson Aug 3, 2018

chenriksson Aug 3, 2018

chenriksson Aug 3, 2018

chenriksson Aug 3, 2018 •

edited

Loading

chenriksson Aug 3, 2018

joelverhagen Aug 3, 2018

chenriksson Aug 4, 2018 •

edited

Loading

chenriksson Aug 4, 2018

chenriksson Aug 4, 2018

agr left a comment

Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig #515

Move JsonConfigurationJob to Jobs.Common / Migrate ArchivePackages to JsonConfig #515

Conversation

chenriksson commented Aug 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenriksson Aug 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenriksson Aug 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agr left a comment

Choose a reason for hiding this comment

chenriksson commented Aug 3, 2018 •

edited

Loading

chenriksson Aug 3, 2018 •

edited

Loading

chenriksson Aug 4, 2018 •

edited

Loading