Nscale analize is missing some container

[LucaLanziani]

nscale           0.16.1
nscale-kernel    0.16.16
nscale-client    0.16.15

• All environment was composed with docker containers
• There was a previous version running
• There was a new version ready to be deployed

While deploying the new version we had an error stating that it couldn't start a container because the port was already in use.
After some digging we noticed that nscale analize was missing that container.

I couldn't debug it in that moment and I couldn't reproduce it.
If I run into the problem again I'll update this issue with more details.

[AdrianRossouw]

one of the things i've noticed is that nscale can't differentiate between containers that have the same commit id. So if the settings were related to files added outside of a git commit, or if you just changed some env variables, nscale analyze has trouble picking it up.

i've also seen it happen in situations where there are multiple instances of the same service, ie: exposed on port ranges 8000-8003. It picks up that one of them is missing, but it can only start the last one again.
IE: if the port 8000 service is down, nscale will try to start the 8003 one again instead of the 8000 one.

@pelger mentioned this was improved in 0.17, but am not sure it was properly resolved.

I ended up having to log into the remote server, and docker kill each of the containers manually, and then running 'nscale system fix' to get them up and running again.

[pelger]

@AdrianRossouw this should be fixed in 0.17 - currently in pre-release