Skip to content

Commit

Permalink
Threading fixes (#2618)
Browse files Browse the repository at this point in the history
* Resolve task invocation issue that prevented effective parallelism.

* Update release notes.
  • Loading branch information
michaelcfanning authored Feb 16, 2023
1 parent 254ab10 commit 120fae3
Show file tree
Hide file tree
Showing 3 changed files with 7 additions and 5 deletions.
2 changes: 2 additions & 0 deletions src/ReleaseHistory.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@
* BRK: For `Guid` properties defined in SARIF spec, updated Json schema to use `uuid`, and updated C# object model to use `Guid?` instead of `string`. [#2555](https://github.com/microsoft/sarif-sdk/pull/2555)
* BRK: Mark `AnalyzeCommandBase` as obsolete. This type will be removed in the next significant update. [#2599](https://github.com/microsoft/sarif-sdk/pull/2599)
* BRK: `LogUnhandledEngineException` no longer has a return value (and updates the `RuntimeErrors` context property directly as other helpers do). [#2599](https://github.com/microsoft/sarif-sdk/pull/2599)
* BUG: Populate missing context region data for small, single-line scan targets. [#2616](https://github.com/microsoft/sarif-sdk/pull/2616)
* BUG: Increase parallelism in `MultithreadedAnalyzeCommandBase` by correcting task creation. []#2618](https://github.com/microsoft/sarif-sdk/pull/2618)
* BUG: Resolve hangs due to unhandled exceptions during multithreaded analysis file enumeration phase. [#2599](https://github.com/microsoft/sarif-sdk/pull/2599)
* BUG: Resolve hangs due to unhandled exceptions during multithreaded analysis file hashing phase. [#2600](https://github.com/microsoft/sarif-sdk/pull/2600)
* BUG: Another attempt to resolve 'InvalidOperationException' with message `Collection was modified; enumeration operation may not execute` in `MultithreadedAnalyzeCommandBase`, raised when analyzing with the `--hashes` switch. [#2459](https://github.com/microsoft/sarif-sdk/pull/2549). There was a previous attempt to fix this in [#2447](https://github.com/microsoft/sarif-sdk/pull/2447).
Expand Down
8 changes: 4 additions & 4 deletions src/Sarif.Driver/Sdk/MultithreadedAnalyzeCommandBase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -218,20 +218,20 @@ private void MultithreadedAnalyzeTargets(TOptions options,

// 1: First we initiate an asynchronous operation to locate disk files for
// analysis, as specified in analysis configuration (file names, wildcards).
Task<bool> enumerateFilesOnDisk = EnumerateFilesOnDiskAsync(options);
Task<bool> enumerateFilesOnDisk = Task.Run(() => EnumerateFilesOnDiskAsync(options));

// 2: Files found on disk are put in a specific sort order, after which a
// reference to each scan target is put into a channel for hashing,
// if hashing is enabled.
Task<bool> hashFilesAndPutInAnalysisQueue = HashFilesAndPutInAnalysisQueueAsnc();
Task<bool> hashFilesAndPutInAnalysisQueue = Task.Run(() => HashFilesAndPutInAnalysisQueueAsnc());

// 3: A dedicated set of threads pull scan targets and analyze them.
// On completing a scan, the thread writes the index of the
// scanned item to a channel that drives logging.
var workers = new Task<bool>[options.Threads];
for (int i = 0; i < options.Threads; i++)
{
workers[i] = ScanTargetsAsync(skimmers, disabledSkimmers);
workers[i] = Task.Run(() => ScanTargetsAsync(skimmers, disabledSkimmers));
}

// 4: A single-threaded consumer watches for completed scans
Expand All @@ -240,7 +240,7 @@ private void MultithreadedAnalyzeTargets(TOptions options,
// scan of the same targets using the same production code
// should produce a log file that is byte-for-byte identical
// to the previous output.
Task<bool> logScanResults = LogScanResultsAsync(rootContext);
Task<bool> logScanResults = Task.Run(() => LogScanResultsAsync(rootContext));

Task.WhenAll(workers)
.ContinueWith(_ => _resultsWritingChannel.Writer.Complete())
Expand Down
2 changes: 1 addition & 1 deletion src/Sarif/Writers/SarifLogger.cs
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ private void EnhanceRun(IEnumerable<string> analysisTargets,

public Func<Uri, HashData> ComputeHashData { get; set; }

public IDictionary<string, HashData> AnalysisTargetToHashDataMap { get; }
public IDictionary<string, HashData> AnalysisTargetToHashDataMap { get; set; }

public IDictionary<ReportingDescriptor, ReportingDescriptorReference> RuleToReportingDescriptorReferenceMap { get; }

Expand Down

0 comments on commit 120fae3

Please sign in to comment.