feat(csi driver): make mount and unmounts non-blocking

[Abhinandan-Purkait]

• Adds the mount utils to frontend and backend sys mount calls via a child process
• Makes the currently mount and umount calls async

[Abhinandan-Purkait]

bors try

[bors-openebs-mayastor]

try

Build failed:

• continuous-integration/jenkins/branch

[Abhinandan-Purkait]

bors try

[bors-openebs-mayastor]

try

Build failed:

• continuous-integration/jenkins/branch

[Abhinandan-Purkait]

bors try

[Abhinandan-Purkait]

Testing:

1. 150 Volumes creation followed deletion all at once.
2. Randomly killed(force) the csi node pods, while operations were in flight.

Observations:

1. All operations completed eventually.
2. No stale entries found eventually.
3. Did not see zombie processes :-;

[bors-openebs-mayastor]

try

Build failed:

• continuous-integration/jenkins/branch

[tiagolobocastro]

wonder if we need this at all anymore, if it's not safe to call ioctl directly, maybe we can simply call the mount binary? Though it might be safer to do this first until we figure out how to migrate from this to mount/unmount arguments properly.

[Abhinandan-Purkait]

With using binaries directly, the errors become a bit vague, but anyhow yeah we can do that. For now kept it as it was to keep the same behaviour.

[tiagolobocastro]

If the mount/unmount calls get stuck due to the device being in a bad state, how are you ensuring you don't keep stacking up new processes trying to the same operation?

[Abhinandan-Purkait]

In my testing I did not see any process stacking up.

[tiagolobocastro]

Don't the ioctl get stuck? Which hangs the server. I thought this was the issue causing the the lockups.

[dsharma-dc]

When the problem happens(repro), have you checked the host's dmesg if it shows any softlockup or oops message from kernel? In that case the host is already unreliable.

[Abhinandan-Purkait]

bors try

[bors-openebs-mayastor]

try

Build succeeded:

• continuous-integration/jenkins/branch