Renovate cleanRepository fails when repo contains files with special characters

[svenbs]

What Renovate type are you using?
Renovate CLI

Describe the bug
If a repo contains files with special encoded characters, the cleanupRepository() or remove() calls will fail with ENOTEMPTY error, as it could not delete the files with special characters in it.

e.g some file like this: ''$'\334''dokument_V1.3_Java7.doc'

Did you see anything helpful in debug logs?

 INFO: Repository is disabled - skipping (repository=redacted)
FATAL: Fatal error: ENOTEMPTY: directory not empty, rmdir '/tmp/renovate/repos/gitlab/redacted/data/' (repository=xyz)
       "err": {
         "errno": -39,
         "code": "ENOTEMPTY",
         "syscall": "rmdir",
         "path": "/tmp/renovate/repos/gitlab/redacted/data",
         "message": "ENOTEMPTY: directory not empty, rmdir '/tmp/renovate/repos/gitlab/redacted/data'",
         "stack": "Error: ENOTEMPTY: directory not empty, rmdir '/tmp/renovate/repos/gitlab/redacted/data'"
       }

To Reproduce
Steps to reproduce the behavior:

1. Create repository and touch a file like this: touch '''$'\334''testfile''
2. Configure { "enabled": false } in the renovate.json, so the repo will be skipped and renovate tries to delete it.

Expected behavior
The repo should get deleted and renovate should move on to the next repo.

[rakeshtembhurne]

@svenbs Could you please share your system details, like OS, node, npm/yarn, git versions where you faced these issues? I am trying to replicate this but seems to be working on my system.

[rakeshtembhurne]

The issue appears to be with fs-extra npm package.

const fs = require('fs-extra');
// ...
await fs.remove(config.localDir);

It appears that there is a known issue (jprichardson/node-fs-extra#575) related to non utf8 characters in files names for linux platforms.

[rarkins]

To gauge how important this is, I'd like to know:

• Are these types of filenames OK with git?
• Are these types of filenames OK with github.com?

And if the answer is yes to both then we need to look if there's any other solution for removing directories with such contents.

[rarkins]

I'm seeing this issue more frequently in the app. Wondering if rimraf handles it better

[rarkins]

When attempting to test one such repo locally, I found that git failed to clone it to my mac. So perhaps you need a Linux fs to start with.

[rarkins]

Example repo/file: https://github.com/moul/jquery-rocket/tree/master/rocket-3d/img

[olegkrivtsov]

I'd like to take this one.

[olegkrivtsov]

My plan for fixing this issue:

1. Edit lib/workers/repository/index.ts:

renovate/lib/workers/repository/index.ts

Lines 71 to 77 in 6732fce

	if (localDir && !config.persistRepoData) {
	try {
	await deleteLocalFile('.');
	} catch (err) /* istanbul ignore if */ {
	logger.warn({ err }, 'localDir deletion error');
	}
	}

This block deletes the repo directory after repo was processed. I think I need to replace the call to deleteLocalFile() with rimraf(): https://www.npmjs.com/package/rimraf From the docs it looks like rimraf works on any OS, so I'd just call it without any OS checks.

Could you please tell me if that look like a good plan?

[rarkins]

Can you reproduce the problem?

[olegkrivtsov]

Hi @rarkins I tried to reproduce on this repo: https://github.com/olegkrivtsov/renovate-special-chars

I created several test files with special chars and tried to run Renovate in Linux Ubuntu 20.04 with {enabled: true} and {enabled:false}. But it worked as expected, I see no errors.

What OS should I use for reproducing? I guess Linux?

[rarkins]

Let's close until we can get a working reproduction