File extraction issue when 2 projects deploy at the same time to the Octopus Server and use the same Large Git Repository as a script source

[Clare-Octopus]

Severity

Blocking customers with large Git Repositories from deploying concurrently using the same script from the same Git repository to the Octopus Server (Default worker pool with no workers in).

Version

Earliest tried 2024.2.9303, can replicate it on 2024.3.12915 but not on Cloud (see below)

Latest Version

I could reproduce the problem in the latest build

What happened?

There seems to be an extraction issue when you have a large Git Repository and you are trying to source a script from that Git Repository with 2 projects that deploy at the same time to the Octopus Server (default worker pool). One project will deploy but the other wont be able to extract the Git file.

Please note this only seems to happen when deploying to the Octopus Server itself (default worker pool with no workers in it), not a deployment target or specific worker.

15:32:01   Info     |       Extracting package 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' to 'C:\Octopus\Work\20241126153158-25-6'
15:32:01   Verbose  |       Timed operation 'Extract package' started.
15:32:01   Verbose  |       Could not collect archive metrics
15:32:01   Error    |       System.IO.IOException: The process cannot access the file 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' because it is being used by another process.

Both deployments will use the same Git hash (see project 2 log below):

Extracting package 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip'

Ideally we need to zip those packages up with different Git hashes so they are not held up by other deployments using the same Git scripts from the same repos.

You will see both Git repositories are cloned, placed in the same zip folder with the same hash and then on extraction, one project will extract the zip folder and the other will error out stating that the file is in use.

I tested this with a small Git Repository and both deployed fine so this looks to only be an issue with large git repositories (the large project I tested with having the issue was 27mb).

I also tested this with a deployment target and it waits for the other project so it handles it better when using the same deployment target, just not when running on the Octopus Server (default worker pool with no workers in):

Reproduction

1. Fork a large git repository (I used this one for testing - https://github.com/linuxkit/linuxkit) to your own Git Repository.
2. Ensure you have a script in there you can use (I added a basic hello world script to that forked repo above).
3. In Octopus ensure you have Git credentials setup for the Git repository you are using above.
4. Create a project with a Run a Script step in it and use Git Repository as your Script Source.
5. Ensure you enter the Git Credentials in the step and point to the script we placed in the Git repository in step 2.
6. Set the deployment to run on the Octopus Server (or the default worker pool as long as there are no workers in there so it will run on the Octopus server).
7. Clone that project so you have two projects with the same step in it.
8. Create releases for both projects and deploy both at the same time, see one error out and one succeed to deploy.

Error and Stacktrace

Executing Run a Script (type Run a Script) on Octopus Server
15:31:56   Verbose  |       Running this script in the Octopus Server security context (NT AUTHORITY\SYSTEM)
15:31:56   Verbose  |       Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-25
15:31:56   Verbose  |       Executable directory is 'C:\Windows\system32\WindowsPowershell\v1.0'
15:31:56   Verbose  |       Executable name or full path: 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe'
15:31:56   Verbose  |       No user context provided. Running as current user.
15:31:56   Verbose  |       Starting 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in working directory 'C:\Octopus\Work\20241126153156-25-5' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
15:31:58   Verbose  |       Process 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in 'C:\Octopus\Work\20241126153156-25-5' exited with code 0
15:31:58   Verbose  |       Using Calamari.win-x64 27.1.16-hotfix0008
15:31:58   Verbose  |       Running this script in the Octopus Server security context (NT AUTHORITY\SYSTEM)
15:31:58   Verbose  |       Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-25
15:31:58   Verbose  |       Executable directory is 'C:\Windows\system32\WindowsPowershell\v1.0'
15:31:58   Verbose  |       Executable name or full path: 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe'
15:31:58   Verbose  |       No user context provided. Running as current user.
15:31:58   Verbose  |       Starting 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in working directory 'C:\Octopus\Work\20241126153158-25-6' using 'OEM United States' encoding running as 'NT AUTHORITY\SYSTEM' with the same environment variables as the launching process
15:31:59   Verbose  |       Calamari Version: 27.1.16-hotfix0008
15:31:59   Verbose  |       Environment Information:
15:31:59   Verbose  |       OperatingSystem: Microsoft Windows NT 10.0.20348.0
15:31:59   Verbose  |       OsBitVersion: x64
15:31:59   Verbose  |       Is64BitProcess: True
15:31:59   Verbose  |       CurrentUser: NT AUTHORITY\SYSTEM
15:31:59   Verbose  |       MachineName: TESTINSTANCE
15:31:59   Verbose  |       ProcessorCount: 8
15:31:59   Verbose  |       CurrentDirectory: C:\Octopus\Work\20241126153158-25-6
15:31:59   Verbose  |       TempDirectory: C:\Windows\TEMP\
15:31:59   Verbose  |       HostProcess: Calamari (3876)
15:32:01   Info     |       Extracting package 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' to 'C:\Octopus\Work\20241126153158-25-6'
15:32:01   Verbose  |       Timed operation 'Extract package' started.
15:32:01   Verbose  |       Could not collect archive metrics
15:32:01   Error    |       System.IO.IOException: The process cannot access the file 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' because it is being used by another process.
15:32:01   Error    |       at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
15:32:01   Error    |       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
15:32:01   Error    |       at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
15:32:01   Error    |       at Calamari.Common.Features.Packages.ZipPackageExtractor.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\ZipPackageExtractor.cs:line 28
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.ArchiveLimits.EnforceDecompressionLimitDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\ArchiveLimits\EnforceDecompressionLimitDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.ArchiveLimits.EnforceCompressionRatioDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\ArchiveLimits\EnforceCompressionRatioDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.Logging.LogArchiveMetricsDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\Logging\LogArchiveMetricsDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.CombinedPackageExtractor.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\CombinedPackageExtractor.cs:line 40
15:32:01   Error    |       at Calamari.Deployment.Conventions.StageScriptPackagesConvention.ExtractPackage(String packageFile, String extractionDirectory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\Conventions\StageScriptPackagesConvention.cs:line 97
15:32:01   Error    |       at Calamari.Deployment.Conventions.StageScriptPackagesConvention.Install(RunningDeployment deployment) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\Conventions\StageScriptPackagesConvention.cs:line 31
15:32:01   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions(IEnumerable`1 installConventions) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 80
15:32:01   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions(Boolean logExceptions) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 31
15:32:01   Error    |       The process cannot access the file 'C:\Octopus\Packages\git-resources\1bf205cb404934a8a5861a4ce636336604ca1404@L2a7a26970d35049faa59ab14845883f0ca6c4385@da39a3ee5e6b4b0d3255bfef95601890afd80709.zip' because it is being used by another process.
15:32:01   Error    |       System.IO.IOException
15:32:01   Error    |       at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
15:32:01   Error    |       at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
15:32:01   Error    |       at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
15:32:01   Error    |       at Calamari.Common.Features.Packages.ZipPackageExtractor.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\ZipPackageExtractor.cs:line 28
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.ArchiveLimits.EnforceDecompressionLimitDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\ArchiveLimits\EnforceDecompressionLimitDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.ArchiveLimits.EnforceCompressionRatioDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\ArchiveLimits\EnforceCompressionRatioDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.PackageExtractorDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\PackageExtractorDecorator.cs:line 18
15:32:01   Error    |       at Calamari.Common.Features.Packages.Decorators.Logging.LogArchiveMetricsDecorator.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\Decorators\Logging\LogArchiveMetricsDecorator.cs:line 25
15:32:01   Error    |       at Calamari.Common.Features.Packages.CombinedPackageExtractor.Extract(String packageFile, String directory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\Features\Packages\CombinedPackageExtractor.cs:line 40
15:32:01   Error    |       at Calamari.Deployment.Conventions.StageScriptPackagesConvention.ExtractPackage(String packageFile, String extractionDirectory) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\Conventions\StageScriptPackagesConvention.cs:line 97
15:32:01   Error    |       at Calamari.Deployment.Conventions.StageScriptPackagesConvention.Install(RunningDeployment deployment) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\Conventions\StageScriptPackagesConvention.cs:line 31
15:32:01   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions(IEnumerable`1 installConventions) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 80
15:32:01   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions(Boolean logExceptions) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Shared\Deployment\ConventionProcessor.cs:line 31
15:32:01   Error    |       at Calamari.Commands.RunScriptCommand.Execute(String[] commandLineArguments) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari\Commands\RunScriptCommand.cs:line 93
15:32:01   Error    |       at Calamari.Program.ResolveAndExecuteCommand(IContainer container, CommonOptions options) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari\Program.cs:line 60
15:32:01   Error    |       at Calamari.Common.CalamariFlavourProgram.Run(String[] args) in C:\BuildAgent\work\e61a42f6adc5dcb6\source\Calamari.Common\CalamariFlavourProgram.cs:line 80
15:32:01   Verbose  |       Process 'C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe' in 'C:\Octopus\Work\20241126153158-25-6' exited with code 100
15:32:01   Verbose  |       Updating manifest with output variables
15:32:01   Verbose  |       Updating manifest with action evaluated variables
15:32:01   Verbose  |       Updating manifest with output variables
15:32:01   Verbose  |       Updating manifest with action evaluated variables
15:32:01   Fatal    |       The remote script failed with exit code 100
15:32:01   Fatal    |       The action Run a Script on the Octopus Server failed

More Information

Zendesk Ticket (internal) - https://octopus.zendesk.com/agent/tickets/210051
R and D discussion (internal) - https://octopusdeploy.slack.com/archives/CNHBHV2BX/p1730212646751319

Workaround

Since I could not replicate this using smaller repositories you will need to either split your scripts out into smaller repositories so when Octopus zips them up its not having to copy your whole repository over. This will mean a smaller zip file size to extract and so will take less time, meaning both deployments should be free to use the same scripts from the same repository.

Either that or because Octopus handles this better with deployment targets (one waits for the other) you could setup a tentacle on the Octopus server and use that to deploy to instead as a deployment target.

[rain-on]

Release Note: Git Resources will only be downloaded if missing (rather than once for each task), this allows multiple operations to safely access the resource simultaneously.