Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove file count limitation in server build #54992

Merged
merged 1 commit into from
Jul 22, 2021
Merged

Conversation

jaredpar
Copy link
Member

The server protocol used to enforce a max of 65,535 files passed to the
compilation. This isn't due to any limitation in the compiler but
instead a guard against a bug in one of our internal build systems. It
provided graceful failure when a build authoring bug caused the contents
of the file system to be passed to the compiler.

That is no longer an issue and the existing limit is blocking real
customer scenarios. Should it become an issue again there is code we
could insert into the customer targets to add a similar guard.

I thought for a bit on what a new limit should be and eventually decided
there wasn't a reasonable one. The compiler server already has a
protocol guard on the size of the message payload. The batch compiler
generally doesn't enforce limits of this nature, insead we attempt to
scale to the build requested by the user. Hence just removed the
limitation entirely

closes #54775

The server protocol used to enforce a max of 65,535 files passed to the
compilation. This isn't due to any limitation in the compiler but
instead a guard against a bug in one of our internal build systems. It
provided graceful failure when a build authoring bug caused the contents
of the file system to be passed to the compiler.

That is no longer an issue and the existing limit is blocking real
customer scenarios. Should it become an issue again there is code we
could insert into the customer targets to add a similar guard.

I thought for a bit on what a new limit should be and eventually decided
there wasn't a reasonable one. The compiler server already has a
protocol guard on the size of the message payload. The batch compiler
generally doesn't enforce limits of this nature, insead we attempt to
scale to the build requested by the user. Hence just removed the
limitation entirely

closes dotnet#54775
@jaredpar jaredpar requested a review from a team as a code owner July 20, 2021 20:51
@jaredpar
Copy link
Member Author

@roslyn-compiler PTAL small bug fix

Copy link
Contributor

@RikkiGibson RikkiGibson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try running a build with more than 2^16 source files?

@jaredpar jaredpar enabled auto-merge (squash) July 20, 2021 21:35
@jaredpar
Copy link
Member Author

Did you try running a build with more than 2^16 source files?

Yep. It actually works better than I would expect

@jaredpar jaredpar disabled auto-merge July 22, 2021 01:46
@jaredpar jaredpar merged commit 8551591 into dotnet:main Jul 22, 2021
@ghost ghost added this to the Next milestone Jul 22, 2021
@jaredpar jaredpar deleted the so-big branch July 22, 2021 01:47
tombuildsstuff added a commit to hashicorp/pandora that referenced this pull request Jul 22, 2021
…5 source files in a project

This is necessary since there's sufficient files when generating all versions of all the things
that we exceed this limit. This is tracked in the upstream issue dotnet/roslyn#54775
and has been fixed in dotnet/roslyn#54992 (but not yet released).

Ultimately this fixes #129
tombuildsstuff added a commit to hashicorp/pandora that referenced this pull request Jul 22, 2021
…5 source files in a project

This is necessary since there's sufficient files when generating all versions of all the things
that we exceed this limit. This is tracked in the upstream issue dotnet/roslyn#54775
and has been fixed in dotnet/roslyn#54992 (but not yet released).

Ultimately this fixes #129
333fred added a commit to 333fred/roslyn that referenced this pull request Jul 23, 2021
* upstream/main: (249 commits)
  Switch back queue name to default (dotnet#55064)
  Fix 'code model' with file scoped namespaces
  Map documents to be reanalyzed back from compile-time to design-time documents (dotnet#55054)
  Update MSBuild Workspace test projects target framework
  Enable CA1069 for ErrorCode and MessageID (dotnet#54695)
  Dev16->17 updates
  Update global.json
  Record completion of "parameterless struct constructor" feature (dotnet#55049)
  Generalize rude edit messages to be applicable to both Hot Reload and EnC (dotnet#55012)
  Update azure-pipelines-official.yml
  Update azure-pipelines-integration.yml
  Merge pull request dotnet#54992 from jaredpar/so-big
  Parameterless struct constructors: Remaining work items (dotnet#54811)
  Update docs/wiki/Diagnosing-Project-System-Build-Errors.md
  update queue name
  Dev16->17 changes
  Fix test
  Fix 'move type' with file scoped namespaces
  Fix 'match folder and namespace' with file scoped namespaces
  Log NFW
  ...
@allisonchou allisonchou modified the milestones: Next, 17.0.P3 Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can't compile more than 65,535 source files via msbuild
4 participants