Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync project level info in parallel. #72976

Merged
merged 58 commits into from
Apr 12, 2024
Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
da8d7e0
Add perf logging
CyrusNajmabadi Apr 10, 2024
9829da1
sync in strips
CyrusNajmabadi Apr 10, 2024
3552d4b
in progress'
CyrusNajmabadi Apr 10, 2024
6377536
fix
CyrusNajmabadi Apr 10, 2024
d150afd
downstream
CyrusNajmabadi Apr 10, 2024
a255fa8
Fetch attributes up front'
CyrusNajmabadi Apr 10, 2024
6d82a4b
get info and text together
CyrusNajmabadi Apr 10, 2024
87de924
Merge remote-tracking branch 'upstream/main' into syncStripes
CyrusNajmabadi Apr 10, 2024
046d435
sync in parallel
CyrusNajmabadi Apr 10, 2024
c8b9a0f
fix
CyrusNajmabadi Apr 10, 2024
d140c85
delete
CyrusNajmabadi Apr 10, 2024
f82f5ad
Apply suggestions from code review
CyrusNajmabadi Apr 10, 2024
58ae336
revert
CyrusNajmabadi Apr 10, 2024
ee6ce0a
Merge branch 'syncStripes' of https://github.com/CyrusNajmabadi/rosly…
CyrusNajmabadi Apr 10, 2024
1033273
Merge branch 'syncStripes' into syncStripesParallel
CyrusNajmabadi Apr 10, 2024
c63c074
remove log helper
CyrusNajmabadi Apr 10, 2024
cdc412a
rename file
CyrusNajmabadi Apr 11, 2024
2439f04
Search less
CyrusNajmabadi Apr 11, 2024
939b9c4
rename
CyrusNajmabadi Apr 11, 2024
357bfeb
rename file
CyrusNajmabadi Apr 11, 2024
c808509
Search less
CyrusNajmabadi Apr 11, 2024
542020e
rename
CyrusNajmabadi Apr 11, 2024
170a0d3
Merge branch 'syncStripes' into syncStripesParallel
CyrusNajmabadi Apr 11, 2024
4c8be11
Merge remote-tracking branch 'upstream/main' into syncStripesParallel2
CyrusNajmabadi Apr 11, 2024
bd55071
Hybrid mode
CyrusNajmabadi Apr 11, 2024
bc626e2
Docs
CyrusNajmabadi Apr 11, 2024
6cbd1b5
Share code
CyrusNajmabadi Apr 11, 2024
e15a73c
optimize
CyrusNajmabadi Apr 11, 2024
69528e5
Add docs
CyrusNajmabadi Apr 11, 2024
21ae618
Docs
CyrusNajmabadi Apr 11, 2024
510331c
Share code
CyrusNajmabadi Apr 11, 2024
d8eb43a
Docs
CyrusNajmabadi Apr 11, 2024
92654f4
Docs
CyrusNajmabadi Apr 11, 2024
afb12f6
fine grained kinds
CyrusNajmabadi Apr 11, 2024
d49c484
Simplify pattern
CyrusNajmabadi Apr 11, 2024
dc8ed7b
Only search solutions and project checksums
CyrusNajmabadi Apr 11, 2024
5b3db93
Add asset hints
CyrusNajmabadi Apr 11, 2024
8929756
REmove unused
CyrusNajmabadi Apr 11, 2024
38902cf
Simplify more
CyrusNajmabadi Apr 11, 2024
2a39c5c
Add implicit
CyrusNajmabadi Apr 11, 2024
4304094
Use kinds uniformly
CyrusNajmabadi Apr 11, 2024
b4efb7f
Updates
CyrusNajmabadi Apr 11, 2024
e91b985
speeling
CyrusNajmabadi Apr 11, 2024
6142d02
Simplify
CyrusNajmabadi Apr 11, 2024
9cdb0bc
Do not capture
CyrusNajmabadi Apr 11, 2024
c9193fd
Finer grained kinds
CyrusNajmabadi Apr 11, 2024
84e2b7a
Finer grained kinds
CyrusNajmabadi Apr 11, 2024
633f604
in progress
CyrusNajmabadi Apr 11, 2024
b044039
in progress
CyrusNajmabadi Apr 11, 2024
5569fb5
in progress
CyrusNajmabadi Apr 11, 2024
2ff97df
in progress
CyrusNajmabadi Apr 11, 2024
10f5b5e
in progress
CyrusNajmabadi Apr 11, 2024
9c9f17b
Only doc text
CyrusNajmabadi Apr 11, 2024
1229821
in progress
CyrusNajmabadi Apr 11, 2024
06dd3e5
in progress
CyrusNajmabadi Apr 11, 2024
4639794
Flesh out
CyrusNajmabadi Apr 11, 2024
b7638f6
tests
CyrusNajmabadi Apr 11, 2024
ec5ad04
docs
CyrusNajmabadi Apr 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
using System.Threading.Tasks;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Editor.UnitTests.Workspaces;
using Microsoft.CodeAnalysis.PooledObjects;
using Microsoft.CodeAnalysis.Remote;
using Microsoft.CodeAnalysis.Remote.Testing;
using Microsoft.CodeAnalysis.Serialization;
Expand Down Expand Up @@ -137,7 +138,11 @@ public async Task TestProjectSynchronization()
var assetSource = new SimpleAssetSource(workspace.Services.GetService<ISerializerService>(), map);

var service = new AssetProvider(sessionId, storage, assetSource, remoteWorkspace.Services.GetService<ISerializerService>());
await service.SynchronizeProjectAssetsAsync(await project.State.GetStateChecksumsAsync(CancellationToken.None), CancellationToken.None);

using var _ = ArrayBuilder<ProjectStateChecksums>.GetInstance(out var allProjectChecksums);
allProjectChecksums.Add(await project.State.GetStateChecksumsAsync(CancellationToken.None));

await service.SynchronizeProjectAssetsAsync(allProjectChecksums, CancellationToken.None);

TestUtils.VerifyAssetStorage(map, storage);
}
Expand Down
112 changes: 0 additions & 112 deletions src/Workspaces/Core/Portable/Workspace/Solution/AssetHint.cs

This file was deleted.

126 changes: 126 additions & 0 deletions src/Workspaces/Core/Portable/Workspace/Solution/AssetPath.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.
// See the LICENSE file in the project root for more information.

using System;
using System.Runtime.Serialization;
using Roslyn.Utilities;

namespace Microsoft.CodeAnalysis.Serialization;

/// <summary>
/// Required information passed with an asset synchronization request to tell the host where to scope the request to. In
/// particular, this is often used to scope to a particular <see cref="Project"/> or <see cref="Document"/> to avoid
/// having to search the entire solution.
/// </summary>
[DataContract]
internal readonly struct AssetPath
{
/// <summary>
/// Instance that will only look up solution-level data when searching for checksums.
/// </summary>
public static readonly AssetPath SolutionOnly = AssetPathKind.Solution;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SolutionOnly

nit: doesn't seem necessary. Almost all other callers just specify the AssetPathKind explicitly.


/// <summary>
/// Special instance, allowed only in tests/debug-asserts, that can do a full lookup across the entire checksum
/// tree. Should not be used in normal release-mode product code.
/// </summary>
public static readonly AssetPath FullLookupForTesting = AssetPathKind.Solution | AssetPathKind.Projects | AssetPathKind.Documents;

[DataMember(Order = 0)]
private readonly AssetPathKind _kind;

/// <summary>
/// If not null, the search should only descend into the single project with this id.
/// </summary>
[DataMember(Order = 1)]
public readonly ProjectId? ProjectId;

/// <summary>
/// If not null, the search should only descend into the single document with this id.
/// </summary>
[DataMember(Order = 2)]
public readonly DocumentId? DocumentId;

public AssetPath(AssetPathKind kind, ProjectId? projectId = null, DocumentId? documentId = null)
{
_kind = kind;
ProjectId = projectId;
DocumentId = documentId;
}

public bool IncludeSolution => (_kind & AssetPathKind.Solution) != 0;
public bool IncludeProjects => (_kind & AssetPathKind.Projects) != 0;
public bool IncludeDocuments => (_kind & AssetPathKind.Documents) != 0;

public bool IncludeProjectStateChecksums => (_kind & AssetPathKind.ProjectStateChecksums) != 0;
public bool IncludeProjectAttributes => (_kind & AssetPathKind.ProjectAttributes) != 0;
public bool IncludeProjectCompilationOptions => (_kind & AssetPathKind.ProjectCompilationOptions) != 0;
public bool IncludeProjectParseOptions => (_kind & AssetPathKind.ProjectParseOptions) != 0;
public bool IncludeProjectProjectReferences => (_kind & AssetPathKind.ProjectProjectReferences) != 0;
public bool IncludeProjectMetadataReferences => (_kind & AssetPathKind.ProjectMetadataReferences) != 0;
public bool IncludeProjectAnalyzerReferences => (_kind & AssetPathKind.ProjectAnalyzerReferences) != 0;

public bool IncludeDocumentStateChecksums => (_kind & AssetPathKind.DocumentStateChecksums) != 0;
public bool IncludeDocumentAttributes => (_kind & AssetPathKind.DocumentAttributes) != 0;
public bool IncludeDocumentText => (_kind & AssetPathKind.DocumentText) != 0;

public static implicit operator AssetPath(AssetPathKind kind) => new(kind);

/// <summary>
/// Searches only for information about this project.
/// </summary>
public static implicit operator AssetPath(ProjectId projectId) => new(AssetPathKind.Projects, projectId, documentId: null);

/// <summary>
/// Searches only for information about this document.
/// </summary>
public static implicit operator AssetPath(DocumentId documentId) => new(AssetPathKind.Documents, documentId.ProjectId, documentId);

/// <summary>
/// Searches the requested project, and all documents underneath it. Used only in tests.
/// </summary>
/// <param name="projectId"></param>
/// <returns></returns>
public static AssetPath SolutionAndProjectForTesting(ProjectId projectId)
CyrusNajmabadi marked this conversation as resolved.
Show resolved Hide resolved
=> new(AssetPathKind.Solution | AssetPathKind.Projects, projectId);

/// <summary>
/// Searches all documents within the specified project.
/// </summary>
/// <param name="projectId"></param>
/// <returns></returns>
public static AssetPath DocumentsInProject(ProjectId projectId)
=> new(AssetPathKind.Documents, projectId);
}

[Flags]
internal enum AssetPathKind
{
/// <summary>
/// Search solution-level information.
/// </summary>
Solution = 1 << 0,

ProjectStateChecksums = 1 << 1,
ProjectAttributes = 1 << 2,
ProjectCompilationOptions = 1 << 3,
ProjectParseOptions = 1 << 4,
ProjectProjectReferences = 1 << 5,
ProjectMetadataReferences = 1 << 6,
ProjectAnalyzerReferences = 1 << 7,

/// <summary>
/// Search projects for results. All project-level information will be searched.
/// </summary>
Projects = ProjectStateChecksums | ProjectAttributes | ProjectCompilationOptions | ProjectParseOptions | ProjectProjectReferences | ProjectMetadataReferences | ProjectAnalyzerReferences,

DocumentStateChecksums = 1 << 8,
DocumentAttributes = 1 << 9,
DocumentText = 1 << 10,

/// <summary>
/// Search documents for results.
/// </summary>
Documents = DocumentStateChecksums | DocumentAttributes | DocumentText,
}
3 changes: 0 additions & 3 deletions src/Workspaces/Core/Portable/Workspace/Solution/Checksum.cs
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,6 @@ public static Checksum ReadFrom(ObjectReader reader)
public static Func<IEnumerable<Checksum>, string> GetChecksumsLogInfo { get; }
= checksums => string.Join("|", checksums.Select(c => c.ToString()));
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, i hope we neve actually enable this one. it's insane.


public static Func<ProjectStateChecksums, string> GetProjectChecksumsLogInfo { get; }
= checksums => checksums.Checksum.ToString();

// Explicitly implement this method as default jit for records on netfx doesn't properly devirtualize the
// standard calls to EqualityComparer<long>.Default.Equals
public bool Equals(Checksum other)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,19 +55,20 @@ public void AddAllTo(HashSet<Checksum> checksums)

[PerformanceSensitive("https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1333566", AllowGenericEnumeration = false)]
internal static async Task FindAsync<TState>(
AssetPath assetPath,
TextDocumentStates<TState> documentStates,
DocumentId? hintDocument,
HashSet<Checksum> searchingChecksumsLeft,
Dictionary<Checksum, object> result,
CancellationToken cancellationToken) where TState : TextDocumentState
{
var hintDocument = assetPath.DocumentId;
if (hintDocument != null)
{
var state = documentStates.GetState(hintDocument);
if (state != null)
{
Contract.ThrowIfFalse(state.TryGetStateChecksums(out var stateChecksums));
await stateChecksums.FindAsync(state, searchingChecksumsLeft, result, cancellationToken).ConfigureAwait(false);
await stateChecksums.FindAsync(assetPath, state, searchingChecksumsLeft, result, cancellationToken).ConfigureAwait(false);
}
}
else
Expand All @@ -80,7 +81,7 @@ internal static async Task FindAsync<TState>(

Contract.ThrowIfFalse(state.TryGetStateChecksums(out var stateChecksums));

await stateChecksums.FindAsync(state, searchingChecksumsLeft, result, cancellationToken).ConfigureAwait(false);
await stateChecksums.FindAsync(assetPath, state, searchingChecksumsLeft, result, cancellationToken).ConfigureAwait(false);
}
}
}
Expand Down
Loading
Loading