Bugs during recursive DELETE

[adavydow]

I tried to remove a directory, but the following error occurred:

 neuro rm storage:results/synthetic-reverse/synth_scheduler/lr02_gamma05step30
Using path 'storage://adavydow/results/synthetic-reverse/synth_scheduler/lr02_gamma05step30'

ERROR: Illegal argument(s) ({"error": "Unexpected exception: [Errno 39] Directory not empty: 'logs'. Path with query: /api/v1/storage/adavydow/results/synthetic-reverse/synth_scheduler/lr02_gamma05step30?op=DELETE."})

All future attempts to remove the directory result in:

neuro rm storage:results/synthetic-reverse/synth_scheduler/lr02_gamma05step30
Using path 'storage://adavydow/results/synthetic-reverse/synth_scheduler/lr02_gamma05step30'

ERROR: Illegal argument(s) ({"error": "Unexpected exception: [Errno 16] Device or resource busy: '.nfs0000000006fa64e000000001'. Path with query: /api/v1/storage/adavydow/results/synthetic-reverse/synth_scheduler/lr02_gamma05step30?op=DELETE."})
)

[adavydow]

Same bug with another directory.

(synthetic-reverse) adavydow-mac:synthetic-reverse adavydow$ neuro rm storage:results/synthetic-reverse/albu/basic
Using path 'storage://adavydow/results/synthetic-reverse/albu/basic'

ERROR: Illegal argument(s) ({"error": "Unexpected exception: [Errno 39] Directory not empty: 'logs'. Path with query: /api/v1/storage/adavydow/results/synthetic-reverse/albu/basic?op=DELETE."})

(synthetic-reverse) adavydow-mac:synthetic-reverse adavydow$ neuro rm storage:results/synthetic-reverse/albu/basic
Using path 'storage://adavydow/results/synthetic-reverse/albu/basic'

ERROR: Illegal argument(s) ({"error": "Unexpected exception: [Errno 16] Device or resource busy: '.nfs0000000006fa64f700000002'. Path with query: /api/v1/storage/adavydow/results/synthetic-reverse/albu/basic?op=DELETE."})

[dalazx]

thanks @adavydow. we'll take a look shortly.

[anayden]

@adavydow how reproducible is that for you? every time or very occasionally?

I have two theories about the issue:

• Some files within the directory are marked read-only
• Files are being added/modified/synchronized via NFS while deletion is happening (i.e., there's a race between list+rm operations vs. changes in the filesystem).

[anayden]

I discussed the issue with Alexey Davydow and he's unable to reproduce it now. I added extra logging for the exception and we'll be able to troubleshoot it better when/if it happens again.

[anayden]

@adavydow could you please update the issue with the problem you encountered last week?

[adavydow]

Encountered the problem again.
Additional information: this problem occurs more often under a high load.
It’s not a neuro rm problem. Deletion from inside the container doesn’t work too.

[asvetlov]

Note, the latest client allows to delete directories with --recursive flag only.
It doesn't close the problem, though. The deletion code is the same still

[dalazx]

the sticky sessions functionality should've resolved the issue. closing for now.

cc @asvetlov @anayden

[serhiy-storchaka]

I have encountered the same issue in #918. It was fixed by repeating the removing. It would be better to repeat it in platform-storage instead of client.