autorest could not read file that exists and is accessible

[pakrym]

We are seeing the following error on and off in CI builds

EXEC : error : Could not read 'https://raw.githubusercontent.com/Azure/azure-rest-api-specs/53574efb3c9199b9dcec21bd8e7ca5b9e8d4f9b9/specification/deviceupdate/data-plane/readme.md'. [/mnt/vss/_work/1/s/sdk/deviceupdate/Azure.Iot.DeviceUpdate/src/Azure.Iot.DeviceUpdate.csproj]

But the file is there and accessible.

Do we need retries?

[pakrym]

Another failure: https://dev.azure.com/azure-sdk/public/_build/results?buildId=779168&view=logs&jobId=b70e5e73-bbb6-5567-0939-8415943fadb9&j=b70e5e73-bbb6-5567-0939-8415943fadb9&t=bdeefc16-b669-5ebd-ad94-a2c19ade53b0

  FATAL: Failed to acquire 'require'd configuration 'https://github.com/Azure/azure-rest-api-specs/blob/bd75cbc7ae9c997f39362ac9d19d557219720bbd/specification/eventgrid/data-plane/readme.md'
EXEC : error : Could not read 'https://github.com/Azure/azure-rest-api-specs/blob/bd75cbc7ae9c997f39362ac9d19d557219720bbd/specification/eventgrid/data-plane/readme.md'. [/mnt/vss/_work/1/s/sdk/eventgrid/Azure.Messaging.EventGrid/src/Azure.Messaging.EventGrid.csproj]
/home/cloudtest/.nuget/packages/microsoft.azure.autorest.csharp/3.0.0-beta.20210311.1/buildMultiTargeting/Microsoft.Azure.AutoRest.CSharp.targets(32,5): error MSB3073: The command "npx autorest@3.0.6339 --skip-csproj --skip-upgrade-check --version=3.1.3 /mnt/vss/_work/1/s/sdk/eventgrid/Azure.Messaging.EventGrid/src/autorest.md  --use=/home/cloudtest/.nuget/packages/microsoft.azure.autorest.csharp/3.0.0-beta.20210311.1/buildMultiTargeting/../tools/netcoreapp3.0/any/ --output-folder=/mnt/vss/_work/1/s/sdk/eventgrid/Azure.Messaging.EventGrid/src/Generated --clear-output-folder=true --namespace=Azure.Messaging.EventGrid --shared-source-folders="/mnt/vss/_work/1/s/eng//../sdk/core/Azure.Core/src/Shared/;/home/cloudtest/.nuget/packages/microsoft.azure.autorest.csharp/3.0.0-beta.20210311.1/buildMultiTargeting/../content/Generator.Shared/"" exited with code 1. [/mnt/vss/_work/1/s/sdk/eventgrid/Azure.Messaging.EventGrid/src/Azure.Messaging.EventGrid.csproj]

[timotheeguerin]

So it is a little tricky to figure out if the problem, I have a PR which adds a little more logging to see if this is just a connection issue or some other internal parsing problem(Seems like there is a few place where I could see it failing) #3980.

Guessing you need an actual publish version of autorest to set on the repo builds so we can try having the error again with additional logging?

Could you also try adding --verbose, --debug to the autorest cli in the build?

[pakrym]

Guessing you need an actual publish version of autorest to set on the repo builds so we can try having the error again with additional logging?

Yeah, we would need to go through the entire update process to get it in the repo.

Could you also try adding --verbose, --debug to the autorest cli in the build?

I can try if it doesn't produce too much output.

[pakrym]

Can we proactively add retries to places that download files?

[timotheeguerin]

Can we proactively add retries to places that download files?

Yeah, looking into this as well

[pakrym]

  AutoRest code generation utility [cli version: 3.0.6339; node: v14.16.1, max-memory: 2048 MB]
  (C) 2018 Microsoft Corporation.
  https://aka.ms/autorest
  AutoRest code generation utility [cli version: 3.1.5; node: v14.16.1, max-memory: 2048 MB]
  (C) 2018 Microsoft Corporation.
  https://aka.ms/autorest
     Loading AutoRest core      '/home/cloudtest/.autorest/@autorest_core@3.1.3/node_modules/@autorest/core/dist' (3.1.3)
     Loading AutoRest core      '/home/cloudtest/.autorest/@autorest_core@3.3.1/node_modules/@autorest/core/dist' (3.3.1)
  INFORMATION: > Loading local AutoRest extension '@autorest/csharp' (/home/cloudtest/.nuget/packages/microsoft.azure.autorest.csharp/3.0.0-beta.20210414.5/buildMultiTargeting/../tools/netcoreapp3.1/any/)
  INFORMATION: > Loading local AutoRest extension '@autorest/csharp' (/home/cloudtest/.nuget/packages/microsoft.azure.autorest.csharp/3.0.0-beta.20210319.1/buildMultiTargeting/../tools/netcoreapp3.1/any/)
  INFORMATION: > Loading AutoRest extension '@autorest/modelerfour' (4.18.3->4.18.3)
  INFORMATION: > Loading AutoRest extension '@autorest/modelerfour' (4.17.2->4.17.2)
EXEC : error : Could not read 'https://raw.githubusercontent.com/Azure/azure-rest-api-specs/c81039d056cc5aa0a0025f5b9e5446a7c194bd1f/specification/eventhub/resource-manager/Microsoft.EventHub/stable/2017-04-01/AuthorizationRules.json'. [/mnt/vss/_work/1/s/sdk/eventhub/Azure.ResourceManager.EventHubs/src/Azure.ResourceManager.EventHubs.csproj]

[pakrym]

Ah, we've pinned the management generator to the older version.