Ensure that status lockfile is closed before trying to release work

[shanemcd]

For context, see:

• Obtain status file lock when updating full status #319
• AWX RuntimeError resulting in loss of worker task logs and true outcome awx#9961

[chrismeyersfsu]

I can't put my finger on it. But this feels like the wrong way to solve this.

[ghjm]

If we're right here, having already passed the != nil check, and some other goroutine or thread gains control and sets bwu.status.LockFile to nil, then the following call to Close() will panic because it is trying to call a function on a nil pointer.

[fosterseth]

eek good point. My first instinct is that we ought to make more use of the statusLock mutex. There are some methods that are reading and writing to status file without getting a lock. monitorLocalStatus calls Load, which grabs status file lock, without getting the mutex lock, for example.

[ghjm]

I used way too many mutex locks in Receptor. A better and more Go-idiomatic way to do this would be to have a single goroutine that does all the status file changes, with channels for other processes to send updates or receive current data.

[ghjm]

I remember there was a lot of subtlety to this, that I spent a lot of time on it, and that the absence of mutex locks in the status file updates is not just an accident - I had something in mind. But looking at it now, it seems pretty reasonable that you should hold a read lock on BaseWorkUnit.statusLock while performing a file update, or a full lock if you're reading back from the file to memory (ie, changing the in-memory data).

[shanemcd]

Replaced by #321