Secret file corruption

[mathewc]

Seeing cases where the host.json (and possibly other) secret files can become corrupted. Example Kusto query (eas region):

FunctionsLogs
| where PreciseTimeStamp >= datetime(2018-06-22 03:00) and PreciseTimeStamp <= datetime(2018-06-22 03:10)
| where RoleInstance == "SmallDedicatedWebWorkerRole_IN_239"
| where Pid == 9340
| where Level < 3
| take 10

We need to do a few things:

1. improve the error message in such cases so the user can self service. In this instance, it required a team member to look into system logs. e.g. "Failure to parse D:\home\data\functions\secrets\host.json. Additional text encountered after finished reading JSON content: r. Path '', line 18, position 1"

2. find out how the file got corrupted. User states that he did not modify the file at all.

For #2, we don’t have any locking or other synchronization around the gen/regen of secrets files. Two concurrent requests could come in and start generating/persisting secrets at the same time. However, for the file based implementation, we’re just using a StreamWriter with error handling/retries. The worst I’d expect would be multiple writes, last one wins, each writing the same content. We should try to repro such corruption by doing some local concurrency tests on this codepath to see if we can repro.

[mathewc]

The share options we're using in the file helper here are causing this. We shouldn't be sharing access to the file when we write it.

Also we should consider recovery here, similar to what we do for secret files that don't decrypt.