Implement automatic cleanup of .godot/ folder items (.import/ in 3.x)

[chanon]

Godot version:
Godot 3.0.2.stable.official

Issue description:
I really like Godot 3's new resource workflow with the .import files and .import folder.
I like that all the source files and settings can be kept in source control.

However, I don't like how Godot never deletes the .stex/.sample/.md5 files for deleted resources in the .import folder. It just feels like the .import folder is accumulating trash more and more until eventually I can't stand it anymore and decide to delete the whole folder and wait for a full re-import. It just bugs me and is something that kind of needs to be managed, when it's supposed to be something you can ignore.

Sometimes I'm also not sure if it affects the editor's knowledge of whether the actual file still exists. (Eg. it is using the .import cache but actually there is no resource file anymore) making me unsure if the image I see in the editor still actually exists. (But that is kind of a separate issue ... because currently even if I delete the resource file and the related .import folder files, the image in the editor is still there. I would expect it to disappear since the editor should know that the file doesn't exist anymore ... it even deleted the .import import settings file for it already! EDIT added as #17734)

It looks like the editor knows how to cleanup the .import import settings files that are in the res folder when the source resource is deleted. I would suggest also cleaning up the .stex/.sample/.md5 files in the .import folder at the same time.

Steps to reproduce:

• Delete or rename a resource file in the res folder using an external file explorer.
• Activate Godot
• Go back to external file explorer to see that the .import file next to the resource file has been deleted (this is good)
• Check inside the .import folder to see that the files related to that resource are still in there. (why not delete them too?)

Using the internal file manager actually results in the same thing when renaming. When deleting it does remove the imported resource but leaves the .md5 file.

[ghost]

pretty sure this is a regression. godot used to delete the old referenced files in .import, if deleted through the FileSystem tab.

[studioschade]

This has been happening to me as well and after moving some music I spent this weekend bashing my head on the wall trying to figure out why it was trying to load music and saying that the resource didn't exist :)

[ZodmanPerth]

I just moved up to Godot 3.1 and am doing some reorganisation (moving and renaming folders and scenes). I just emptied my .import folder in Windows Explorer, and when I clicked back in the Godot editor it did a scan and repopulated the .imports folder with all the assets it currently knows about.

I agree the editor should do this cleanup automatically and keep .imports in sync, but at least there is this workaround in the meantime.

[Anutrix]

This is should be easy to implement. My simple logic:

1. Just make a String List called To_Be_Deleted_List of all filenames present in .import folder.
2. Compare each of the names to [deps] section of all the *.import files.
3. Remove from To_Be_Deleted_List, if match is found.
4. Only names of Artifacts that can be purged/cleaned will left be in To_Be_Deleted_List.

Is this bad approach to solve this issue?
I would implement it myself, but I don't know much about file systems yet.

[ZodmanPerth]

I think the way assets handle imports is changing, so this may become a non issue in the near future.

[Anutrix]

Didn't it undergo a major change in 3.0? So another big change for 4.0?

[akien-mga]

There's no big change to the import system planned for 4.0. (There are changes to texture flags, but AFAIK textures will still be imported to StreamTextures in the .import/ folder.)

[ZodmanPerth]

Ah cool. Thanks for clarifying.

[snoopdouglas]

The following Python script seems to work for cleaning up .import/. Note this is quick and dirty; it doesn't properly parse .import files, though it does ensure it's reading deps.dest_files specifically.

I've tested this thoroughly but please back up your .import/ or use version control before running this.

#!/usr/bin/python3

import re
from glob import iglob
import json
import os

used = set()
unused = set()

re_dest_files = re.compile(r'^dest_files=(\[.*\])$')

# for all *.import files in project:
for path in iglob("**/*.import", recursive=True):
    with open(path, 'r') as f:
        # find the dest_files line in the [deps] section
        deps = False
        found = False
        for line in f:
            if not deps:
                deps = (line == '[deps]\n')
                continue
            else:
                if line[0] == '[':
                    # new section encountered, somehow there's no dest_files
                    break

                # extract the imported file path(s)
                match = re_dest_files.match(line)
                if match:
                    paths_str = match.group(1)
                    paths = json.loads(paths_str)
                    md5s = set()
                    for i, p in enumerate(paths):
                        # remove 'res://'
                        paths[i] = p[6:]
                        md5s.add(os.path.splitext(paths[i])[0] + '.md5')

                    used |= set(paths)
                    used |= md5s
                    found = True
                    break

        if not found:
            print(f"warning: {path}: no deps.dest_files")

# find every file in .import/ that isn't referenced in a *.import file
for path in iglob('.import/*'):
    if path not in used:
        unused.add(path)

# do this alphabetically for nicer terminal output
unused = list(unused)
unused.sort()
n = 0

for path in unused:
    print(path)
    os.remove(path)
    n += 1

print('---')
print(f'removed {n} files from .import/')

[Daniel-da-Silva]

I have the same problem and it is really weird - I deleted the .import folder and everything reimported and seems to work fine.
The big thing is: The project directory is now 31.5 mb large and before deleting the .import folder it was 127 mb large which is quite a bit on my small SSD. Are there plans to change that system?

Couldn't there be a button in Godot like: Clean-Up or something?

[Calinou]

The big thing is: The project directory is now 31.5 mb large and before deleting the .import folder it was 127 mb large which is quite a bit on my small SSD. Are there plans to change that system?

Imported textures (for 3D usage) will always take a fair amount of space because VRAM compression is designed to have a fixed compression factor. Using a size-optimized compression algorithm on top would slow down compression and decompression noticeably, which makes it a bad tradeoff in most scenarios. Iteration speeds are paramount to a smooth development experience 🙂

Couldn't there be a button in Godot like: Clean-Up or something?

This could be added, but it would likely require the editor to be closed at the same time you press it (unless we can implement something like this in core).

[Daniel-da-Silva]

I tried VRAM compression out once but also got back to lossless (the game is 2D) although I see it a lot in my 3D game (the assets in the project alone are only around 1 mb and the project folder is now 42.8 mb). Just out of curiosity: Why doesn't this have impact on the exported files?

Honestly, I think both solutions would be great/a big improvement :D Especially for people who don't know how to access hidden folders etc.

[Calinou]

Just out of curiosity: Why doesn't this have impact on the exported files?

The exported project contains only the imported resources, not the source assets. You can check this by exporting a ZIP archive instead of a PCK file using the Export dialog's Export PCK/ZIP... button.

[Daniel-da-Silva]

thank you very much!

[rambda]

#17733 (comment)

Thanks @snoopdouglas for the code.

Also:
If you are on Windows, use this version. File path on Windows has the backslash problem, which can lead to file deletion by mistake. Also it solved an exception on non-utf-8 systems.

Edit: fix a bug that if texture compression mode is VRAM, the .md5 will be identified as unused file.

#!/usr/bin/python3

import re
from glob import iglob
import json
import os

used = set()
unused = set()

re_dest_files = re.compile(r'^dest_files=(\[.*\])$')

# for all *.import files in project:
for path in iglob("**/*.import", recursive=True):
    with open(path, 'r', encoding='UTF-8') as f:
        # find the dest_files line in the [deps] section
        deps = False
        found = False
        for line in f:
            if not deps:
                deps = (line == '[deps]\n')
                continue
            else:
                if line[0] == '[':
                    # new section encountered, somehow there's no dest_files
                    break

                # extract the imported file path(s)
                match = re_dest_files.match(line)
                if match:
                    paths_str = match.group(1)
                    paths = json.loads(paths_str)
                    md5s = set()
                    
                    vram_md5 = os.path.splitext(os.path.splitext(paths[0])[0])[0] + ".md5"
                    vram_md5 = vram_md5[6:]
                    md5s.add(vram_md5)
                    
                    for i, p in enumerate(paths):
                        # remove 'res://'
                        paths[i] = p[6:]
                        md5s.add(os.path.splitext(paths[i])[0] + '.md5')

                    used |= set(paths)
                    used |= md5s
                    found = True
                    break

        if not found:
            print(f"warning: {path}: no deps.dest_files")

# find every file in .import/ that isn't referenced in a *.import file
for path in iglob('.import/*'):
    path = path.replace('\\', '/')
    if path not in used:
        unused.add(path)

# do this alphabetically for nicer terminal output
unused = list(unused)
unused.sort()
n = 0

for path in unused:
    print(path)
    n += 1

def confirm():
    answer = input()
    if answer == "Y":
        for path in unused:
            os.remove(path)
        print(f'removed {n} files from .import/')
        input()
    elif answer == "N":
        return
    else:
        confirm()

print(f"There are {n} unused files. Remove? Y/N")
confirm()

[jitspoe]

I noticed that not only do the .import files not delete when deleting from the FileSystem tab in godot, but it seems lots to keep lots of iterations around for different versions of the same texture. This might actually be because I was trying to import textures that were too large and caused the engine to become unstable, but I had a lot of junk accumulated in the .import folder. Would be really nice to clean this up automatically when a texture changes or is deleted!

[jordanlis]

Hi everyone.

I think this is a very important topic : I agree that these *.import files should be removed once you delete its original file.

Until now, I was using this extension : https://godotengine.org/asset-library/asset/813
I think this extension shouldn't exist and that the file shouldn't be kept at all as I said.

Now, this extension is not available for 4.x, so I don't have so many options rather than checking myself for the duplicates of old files, and delete them, but it's a bit tedious. Maybe I'll try the script you mentioned above.

Anyway, if at anypoint a dev could make something for this, it could be great !

[Poobslag]

Here is a bash shell script which reports orphaned .import files in the specified $PROJECT_DIR.

#!/bin/bash

# report orphaned .import files; make sure to assign PROJECT_DIR
PROJECT_DIR=project
while IFS= read -r import_file; do
    filename="${import_file%.import}"
    if [ ! -f "$filename" ]; then
        echo "Found orphaned import file: $import_file"
    fi
done < <(find "$PROJECT_DIR" -type f -name "*.import")