Merge branch 'dev' into vabachu/update-isolated-traces

.github/ISSUE_TEMPLATE/new-release-template.md

@@ -3,67 +3,58 @@ name: New release template
 about: Template for creating new releases of Durable Functions
 title: ''
 labels: ''
-assignees: davidmrdavid, bachuv, nytiannn
+assignees: bachuv, nytiannn, andystaples
 
 ---
 
 **Prep DTFx Release: (assigned to:)**
 _Due: <2-3-business-days-before-release>_
 - [ ] Check DTFx package versions (either DT-AzureStorage only or if there were Core changes DT-AzureStorage, DT-Core, and DT-ApplicationInsights)
 - [ ] Review the [DTFx Dependabot vulnerability alerts](https://github.com/Azure/durabletask/security/dependabot) and address them. Note: code samples / test projects _may_ be excluded from this check.
-- [ ] Delete DTFx test packages from the [ADO feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask).
-- [ ] Run the [DTFx release pipeline](https://durabletaskframework.visualstudio.com/Durable%20Task%20Framework%20CI/_build?definitionId=21) ([defined here](https://github.com/Azure/durabletask/blob/main/azure-pipelines-release.yml)) to obtain new packages.
-- [ ] Publish DTFx packages to the [ADO feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask) for testing.
-- [ ] Keep branch `azure-storage-v12` updated with branch `main`.
+- [ ] Run the [DTFx release pipeline](https://azfunc.visualstudio.com/internal/_build?definitionId=640) ([defined here](https://github.com/Azure/durabletask/blob/main/eng/ci/official-build.yml)) to obtain new packages.
+- [ ] Publish DTFx packages to the [ADO test feed](https://azfunc.visualstudio.com/internal/_artifacts/feed/durabletask-test) for testing.
 
 **Prep DotNet Isolated SDK Release: (assigned to:)**
 _Due: <2-3-business-days-before-release>_
 - [ ] If there were DTFx.Core changes, check its reference version [here](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/src/Abstractions/Abstractions.csproj#L10). If updates are required, document the changes in [release notes](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/src/Abstractions/RELEASENOTES.md).
 - [ ] Check dotnet isolated SDK versions [here](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/eng/targets/Release.props#L20). If updated, document the changes in the [change logs](https://github.com/microsoft/durabletask-dotnet/blob/c838535adb6aedb6671cf193389ce63a6b4a9b24/CHANGELOG.md).
-- [ ] Run pipeline [Release .Net out-of-proc SDK](https://durabletaskframework.visualstudio.com/Durable%20Task%20Framework%20CI/_build?definitionId=29) to create the new package and publish it to the ADO feed for testing.
+- [ ] Run pipeline [Release .Net out-of-proc SDK](https://azfunc.visualstudio.com/internal/_build?definitionId=657) to create the new package and publish it to the ADO feed for testing.
 
-**Prep Release (assigned to: )**
+**Prep WebJobs and Worker Extensions Release (assigned to: )**
 _Due: <2-business-days-before-release>_
 - [ ] Update DTFx packages and Analyzer versions at WebJobs.Extensions.Durabletask.csproj and check if we need to update the WebJobs.Extensions.Durabletask version.
 - [ ] Locally, run `dotnet list package --vulnerable` to ensure the release is not affected by any vulnerable dependencies.
 - [ ] Review the [Dependabot vulnerability alerts](https://github.com/Azure/azure-functions-durable-extension/security/dependabot) and address them. Note: code samples / test projects _may_ be excluded from this check.
-- [ ] Add the Durable Functions package to the [ADO test feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask-test).
 - [ ] Check for package size, make sure it's not surprisingly heavier than a previous release.
 - [ ] Update .NET Isolated SDK version at Worker.Extensions.Durabletask.csproj and check if we need to update the Worker.Extensions.Durabletask version.
-- [ ] Run pipeline [Release .Net Isolated Worker Extension](https://durabletaskframework.visualstudio.com/Durable%20Task%20Framework%20CI/_build?definitionId=30) to create the new package and add it to the ADO feed for testing.
-- [ ] Merge (**choose create a merge commit, NOT squash merge**) dev into main. Person performing validation must approve PR.
-- [ ] Keep branch `v3.x` updated with branch `dev`. Do not merge PRs that are specific to Durable Functions v2.
+- [ ] Run [the extension release pipeline](https://azfunc.visualstudio.com/internal/_build?definitionId=673) to create the new WebJobs.Extensions.Durabletask and Worker.Extensions.Durabletask packages and add them to the [ADO test feed].(https://azfunc.visualstudio.com/internal/_artifacts/feed/durabletask-internal) for testing.
+- [ ] Cherry-pick any PRs that need to be in the `v2.x` branch
 
 **Validation**
 _Due: <1-business-days-before-release>_
 - [ ] Run private performance tests and ensure no regressions. **(assigned to: )**
-- [ ] Smoke test Functions V1, Functions V2, and Functions V3 .NET apps. **(assigned to: )**
-- [ ] Smoke test .NET apps with backend Netherite, MSSQL. **(assigned to: )**
-- [ ] Smoke test .NET isolated apps. **(assigned to: )**
+- [ ] Smoke test .NET isolated apps. **(assigned to: )** - check that the correct version of the webjobs extension is loaded by going to bin\Debug\net8.0\.azurefunctions\Microsoft.Azure.WebJobs.Extensions.DurableTask.dll, right click on Properties, go to the Details tab and check the version
+- [ ] Merge (**choose create a merge commit, NOT squash merge**) dev into main. Person performing validation must approve PR.
 
 **DTFx Release Completion (assigned to: )**
 _Due: <release-deadline>_
-- [ ] Upload DTFx packages to NuGet (directly to nuget.org).
-- [ ] Delete `Microsoft.DurableTask.Sidecar.Protobuf` from MyGet, and publish it to NuGet _iff_ it was updated as an Extension dependency. 
+- [ ] Add the DTFx packages to the [ADO feed](https://azfunc.visualstudio.com/internal/_artifacts/feed/durabletask-internal)
+- [ ] Upload DTFx packages to NuGet (directly to nuget.org). 
 - [ ] Publish release notes for DTFx.
-- [ ] Patch increment DTFx packages that were released (either DT-AzureStorage only or if there were Core changes DT-AzureStorage, DT-Core, and DT-ApplicationInsights)
+- [ ] Patch increment DTFx packages that were released (either DT-AzureStorage only or if there were Core changes DT-AzureStorage, DT-Core, and DT-ApplicationInsights) in WebJobs.Extensions.DurableTask.csproj
 
 **DotNet Isolated SDK Release Completion: (assigned to:)**
 _Due: <release-deadline>_
+- [ ] Add the .NET isolated SDK packages to the [ADO feed](https://azfunc.visualstudio.com/internal/_artifacts/feed/durabletask-internal)
 - [ ] Upload .NET isolated SDK packages to NuGet (directly to nuget.org).
 - [ ] Publish release notes in the durable-dotnet repo.
 
 **Release Completion (assigned to: )**
 _Due: <release-deadline>_
-- [ ] Delete Durable Functions packages from the [ADO test feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask-test).
-- [ ] Run the [Durable Functions release pipeline](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_build?definitionId=23) and select `main` as the branch.
-- [ ] Add the Durable Functions package to the [ADO feed](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_artifacts/feed/durabletask) using [this pipeline](https://dev.azure.com/durabletaskframework/Durable%20Task%20Framework%20CI/_release?_a=releases&view=mine&definitionId=11).
+- [ ] Run the [Durable Functions release pipeline](https://azfunc.visualstudio.com/internal/_build?definitionId=673) and select `dev` or `v2.x` as the branch. Choose `dev` if you are making a v3.x release, otherwise choose `v2.x` for a v2.x release.
+- [ ] Add the Durable Functions packages to the [ADO feed](https://azfunc.visualstudio.com/internal/_artifacts/feed/durabletask-internal)
 - [ ] Upload the Durable Functions package to NuGet (directly to nuget.org).
 - [ ] Upload .NET Isolated worker extension package to NuGet (directly to nuget.org).
 - [ ] Create a PR in the [Azure Functions templates repo](https://github.com/Azure/azure-functions-templates) targeting branch `dev` to update all references of "Microsoft.Azure.WebJobs.Extensions.DurableTask" (search for this string in the code) to the latest version.
 - [ ] _if and only if this is a new major release_, Create a PR in the [Azure Functions bundles repo](https://github.com/Azure/azure-functions-extension-bundles) to update bundles to the latest version .
-- [ ] Merge all pending PR docs from `pending_docs.md.`
-- [ ] Reset `pending_docs.md` and `release_notes.md` in the `dev` branch. You will want to save `release_notes.md` somewhere for when you publish release notes.
-- [ ] Publish release notes from the pre-reset `release_notes.md.`
-- [ ] Post announcement on Twitter (Chris).
-- [ ] Increment Durable Functions patch version.
+- [ ] Publish release notes.

.github/pull_request_template.md

@@ -21,5 +21,5 @@ resolves #issue_for_this_pr
     * [ ] Otherwise: major or minor version updates are reflected in `/src/Worker.Extensions.DurableTask/AssemblyInfo.cs`
 * [ ] My changes **do not** add EventIds to our EventSource logs
     * [ ] Otherwise: Ensure the EventIds are within the supported range in our existing Windows infrastructure. You may validate this with a deployed app's telemetry. You may also extend the range by completing a PR such as [this one](https://msazure.visualstudio.com/One/_git/AAPT-Antares-Websites/pullrequest/7463263?_a=files).
-* [ ] My changes **should** be added to v3.x branch.
-    * [ ] Otherwise: This change only applies to Durable Functions v2.x and **will not** be merged to branch v3.x.
+* [ ] My changes **should** be added to **v2.x** branch.
+    * [ ] Otherwise: This change applies exclusively to WebJobs.Extensions.DurableTask v3.x. It will be retained only in the `dev` and `main` branches and will not be merged into the `v2.x` branch.

.github/workflows/E2ETest.yml

@@ -0,0 +1,52 @@
+name: End to End Test - .NET Isolated/Func V4
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [ main, dev, andystaples/add-gh-action-for-e2e-tests ]
+    paths:
+      - 'src/**'
+      - 'test/e2e/**'
+  pull_request:
+    branches: [ main, dev, andystaples/add-gh-action-for-e2e-tests ]
+    paths:
+      - 'src/**'
+      - 'test/e2e/**'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup .NET Core
+        uses: actions/setup-dotnet@v3
+        with:
+          dotnet-version: 6.0.x
+
+      - name: Setup .NET Core
+        uses: actions/setup-dotnet@v3
+        with:
+          dotnet-version: 8.0.x
+
+      - name: Setup E2E tests
+        shell: pwsh
+        run: |
+          .\test\e2e\Tests\build-e2e-test.ps1 -SkipStorageEmulator
+
+      - name: Build
+        working-directory: test/e2e/Tests
+        run: dotnet build
+
+      # Install Azurite
+      - name: Set up Node.js (needed for Azurite)
+        uses: actions/setup-node@v3
+        with:
+          node-version: '18.x' # Azurite requires at least Node 18
+
+      - name: Install Azurite
+        run: npm install -g azurite
+
+      - name: Run E2E tests
+        working-directory: test/e2e/Tests
+        run: azurite --silent --blobPort 10000 --queuePort 10001 --tablePort 10002 & dotnet test

.gitignore

@@ -297,3 +297,6 @@ functions-extensions/
 .vscode/
 .ionide/
 /src/WebJobs.Extensions.DurableTask/Microsoft.Azure.WebJobs.Extensions.DurableTask.xml
+
+# E2E Tests build output
+/src/WebJobs.Extensions.DurableTask/out/*

WebJobs.Extensions.DurableTask.sln

@@ -75,8 +75,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "pipelines", "pipelines", "{
 	ProjectSection(SolutionItems) = preProject
 		azure-pipelines.yml = azure-pipelines.yml
 		.github\workflows\smoketest-dotnet-isolated-v4.yml = .github\workflows\smoketest-dotnet-isolated-v4.yml
-		.github\workflows\smoketest-dotnet-v2.yml = .github\workflows\smoketest-dotnet-v2.yml
-		.github\workflows\smoketest-dotnet-v3.yml = .github\workflows\smoketest-dotnet-v3.yml
 		.github\workflows\smoketest-node14-v4.yml = .github\workflows\smoketest-node14-v4.yml
 		.github\workflows\smoketest-python37-v4.yml = .github\workflows\smoketest-python37-v4.yml
 	EndProjectSection
@@ -87,6 +85,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "DFPerfScenariosV4", "test\D
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Worker.Extensions.DurableTask.Tests", "test\Worker.Extensions.DurableTask.Tests\Worker.Extensions.DurableTask.Tests.csproj", "{76DEC17C-BF6A-498A-8E8A-7D6CB2E03284}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "E2ETests", "test\e2e\Tests\E2ETests.csproj", "{63628712-4196-4865-B268-5BA3D8F08DE1}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -155,6 +155,10 @@ Global
 		{76DEC17C-BF6A-498A-8E8A-7D6CB2E03284}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{76DEC17C-BF6A-498A-8E8A-7D6CB2E03284}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{76DEC17C-BF6A-498A-8E8A-7D6CB2E03284}.Release|Any CPU.Build.0 = Release|Any CPU
+		{63628712-4196-4865-B268-5BA3D8F08DE1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{63628712-4196-4865-B268-5BA3D8F08DE1}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{63628712-4196-4865-B268-5BA3D8F08DE1}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{63628712-4196-4865-B268-5BA3D8F08DE1}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -184,6 +188,7 @@ Global
 		{7387E723-E153-4B7A-B105-8C67BFBD48CF} = {78BCF152-C22C-408F-9FB1-0F8C99B154B5}
 		{FC8AD123-F949-4D21-B817-E5A4BBF7F69B} = {7387E723-E153-4B7A-B105-8C67BFBD48CF}
 		{76DEC17C-BF6A-498A-8E8A-7D6CB2E03284} = {78BCF152-C22C-408F-9FB1-0F8C99B154B5}
+		{63628712-4196-4865-B268-5BA3D8F08DE1} = {78BCF152-C22C-408F-9FB1-0F8C99B154B5}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {5E9AC327-DE18-41A5-A55D-E44CB4281943}

eng/ci/code-mirror.yml

@@ -5,7 +5,7 @@ trigger:
     # Keep this set limited as appropriate (don't mirror individual user branches).
     - main
     - dev
-    - v3.x
+    - v2.x
 
 resources:
   repositories:

eng/ci/official-build.yml

@@ -7,7 +7,7 @@ trigger:
         include:
             - main
             - dev
-            - v3.x
+            - v2.x
 
 # CI only, does not trigger on PRs.
 pr: none

src/WebJobs.Extensions.DurableTask/DurableTaskJobHostConfigurationExtensions.cs

@@ -112,7 +112,9 @@ internal static IWebJobsBuilder AddDurableScaleForTrigger(this IWebJobsBuilder b
                 provider = new DurableTaskTriggersScaleProvider(serviceProvider.GetService<IOptions<DurableTaskOptions>>(), serviceProvider.GetService<INameResolver>(), serviceProvider.GetService<ILoggerFactory>(), serviceProvider.GetService<IEnumerable<IDurabilityProviderFactory>>(), triggerMetadata);
                 return provider;
             });
-            builder.Services.AddSingleton<IScaleMonitorProvider>(serviceProvider => serviceProvider.GetServices<DurableTaskTriggersScaleProvider>().Single(x => x == provider));
+
+            // Commenting out incremental scale model for hotfix release 3.0.0-rc.4, SC uses TBS by default
+            // builder.Services.AddSingleton<IScaleMonitorProvider>(serviceProvider => serviceProvider.GetServices<DurableTaskTriggersScaleProvider>().Single(x => x == provider));
             builder.Services.AddSingleton<ITargetScalerProvider>(serviceProvider => serviceProvider.GetServices<DurableTaskTriggersScaleProvider>().Single(x => x == provider));
             return builder;
         }

src/WebJobs.Extensions.DurableTask/Listener/DurableTaskMetricsProvider.cs

@@ -16,6 +16,8 @@ internal class DurableTaskMetricsProvider
         private readonly string hubName;
         private readonly ILogger logger;
         private readonly StorageAccountClientProvider storageAccountClientProvider;
+        private PerformanceHeartbeat heartbeat;
+        private DateTime heartbeatTimeStamp;
 
         private DisconnectedPerformanceMonitor performanceMonitor;
 
@@ -29,33 +31,40 @@ public DurableTaskMetricsProvider(
             this.logger = logger;
             this.performanceMonitor = performanceMonitor;
             this.storageAccountClientProvider = storageAccountClientProvider;
+            this.heartbeat = null;
+            this.heartbeatTimeStamp = DateTime.MinValue;
         }
 
         public virtual async Task<DurableTaskTriggerMetrics> GetMetricsAsync()
         {
             DurableTaskTriggerMetrics metrics = new DurableTaskTriggerMetrics();
 
             // Durable stores its own metrics, so we just collect them here
-            PerformanceHeartbeat heartbeat = null;
             try
             {
                 DisconnectedPerformanceMonitor performanceMonitor = this.GetPerformanceMonitor();
-                heartbeat = await performanceMonitor.PulseAsync();
+
+                // We only want to call PulseAsync every 5 seconds
+                if (this.heartbeat == null || DateTime.UtcNow > this.heartbeatTimeStamp.AddSeconds(5))
+                {
+                    this.heartbeat = await performanceMonitor.PulseAsync();
+                    this.heartbeatTimeStamp = DateTime.UtcNow;
+                }
             }
             catch (Exception e) when (e.InnerException is RequestFailedException)
             {
                 this.logger.LogWarning("{details}. HubName: {hubName}.", e.ToString(), this.hubName);
             }
 
-            if (heartbeat != null)
+            if (this.heartbeat != null)
             {
-                metrics.PartitionCount = heartbeat.PartitionCount;
-                metrics.ControlQueueLengths = JsonConvert.SerializeObject(heartbeat.ControlQueueLengths);
-                metrics.ControlQueueLatencies = JsonConvert.SerializeObject(heartbeat.ControlQueueLatencies);
-                metrics.WorkItemQueueLength = heartbeat.WorkItemQueueLength;
-                if (heartbeat.WorkItemQueueLatency > TimeSpan.Zero)
+                metrics.PartitionCount = this.heartbeat.PartitionCount;
+                metrics.ControlQueueLengths = JsonConvert.SerializeObject(this.heartbeat.ControlQueueLengths);
+                metrics.ControlQueueLatencies = JsonConvert.SerializeObject(this.heartbeat.ControlQueueLatencies);
+                metrics.WorkItemQueueLength = this.heartbeat.WorkItemQueueLength;
+                if (this.heartbeat.WorkItemQueueLatency > TimeSpan.Zero)
                 {
-                    metrics.WorkItemQueueLatency = heartbeat.WorkItemQueueLatency.ToString();
+                    metrics.WorkItemQueueLatency = this.heartbeat.WorkItemQueueLatency.ToString();
                 }
             }