Regression: Assets are getting reimported for no reason at open project time

[kubecz3k]

Operating system or device, Godot version, GPU Model and driver (if graphics related):
c846e49, Ubuntu 17.04

Issue description:
It seems once for some time, upon opening project all assets are getting reimported for no explicit reason. This was already present in d413f37.

Steps to reproduce:
This happens to me quite randomly to tell truth, but I'm able to recreate it by closely following those steps:

1. Download godot demos repo
2. Ensure there is no .import folder inside material demo
3. Open materials demo project
4. Save main scene
5. Exit to project manager
6. Open project again
7. Experience asset reimport

For the second and third time reopening project is usually not enough. In such case it sometimes is enough to open couple random scenes, modify them a little (add line script, change node name) and save very frequently. Also I'm closing scenes one by one before exit in such scenario.
In such case one can also try to just remove .import folder and go through the former reproduction steps

Link to minimal example project:

[kubecz3k]

I assume some recent reimport complains like #12939 are happening because of this issue.
(I don't think that infrequent 3.0 testers, are aware of the fact that by design such slow reimport should be one time thing)

[ghost]

I can confirm this

[djrm]

this is really bad when you have lot of high res assets, i had to wait like 1 hour or so.

[reduz]

This is probably the fault of @akien-mga changing godot configuration locations, which will most likely force all projects to reimport once.

That said, I am planning on making all the reimport info local and stored in the .import, including an md5, to avoid random reimports like in such cases..

[djrm]

@reduz i dont think so i've been having this problem for months. and it actually repeats quite often

[djrm]

also yeah the md5 would be great, the current reimport code is super convoluted, and sometimes it happens twice, making things worse (i noticed this when reworking the obj importer)

[reduz]

i will change the reimport logic then, since it's a good chance. please keep this issue open

…

On Tue, Nov 21, 2017 at 3:54 PM, Daniel Ramirez ***@***.***> wrote: @reduz <https://github.com/reduz> i dont think so i've been having this problem for months. and it actualy repeats quite often — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13135 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF-Z238EYNHo-nrb4FzP1kBpEltRTx_xks5s4xxtgaJpZM4Qllck> .

[kubecz3k]

@reduz what is happening recently is really abnormal. On those sessions where I did any real work on the project, I can be practically sure, next time when I will open an editor all the assets will be fully re-imported.
In my use case it never worked like that. I'm convinced there must be some new regression not older than 10 days.

[reduz]

will do after beta though, want to avoid breaking this logic too much accidentally

[akien-mga]

This is probably the fault of @akien-mga changing godot configuration locations, which will most likely force all projects to reimport once.

That would only trigger the issue once though, and this issue describes the reimport as happening many times when reloading the scene again. So either I broke also the writing to the project metadata dir (which is possible), or the problem would be somewhere else.

[akien-mga]

It should definitely be fixed before beta though, it makes the editor unusable from what I understand.

[rminderhoud]

Yes, I think there is definitely something funky with the re-import behavior and it has been happening for a while. I have the same issue of "always re-import on startup" but I also have issues with my custom EditorImportPlugin. On startup the resources supported by my EditorImportPlugin are not re-imported and are not even detected (don't show up in Filesystem pane). I have to touch <parent_dir_of_asset> which then triggers a re-import of the entire directory even if everything had previously been imported.

I think this may be related?

[Ace-Dragon]

I agree that the reimport code is not very smart right now.

I've also seen cases where a full reimport gets done when Godot would've only needed to import a small handful of new or updated assets. It also will generally occur if you download a new Godot build and open your project for the first time (in that build).

This is on the Win64 platform, I also think the logic should be fixed before the beta in order to get more people testing it (as users will more likely keep using the beta and make reports if it's fully usable).

[NathanWarden]

Strange, I haven't noticed this at all. I've noticed it needs to scan the assets to make sure they haven't changed, but I haven't seen them actually re-import.

@reduz How does the importer know an asset has been updated?

I would think it would:

A) Check the modified date of the individual asset against a db or related file in the .import folder. If that has changed then B) hash the source file and check it against a hash of the last imported version. So, if the modified date has changed and the hash is different, then re-import. But, if the hash is the same it will skip the re-import. That way it will never re-import anything that hasn't changed. And if A is true, but regardless if B is false it should still update the last modified date so it will skip that asset next time.

Or psuedo code:

if ( modified_date_changed ) {
    if ( hash_changed ) {
        reimport();
    }
    update_time_stamp();
}

[nylnook]

I can also confirm it also happen for me each time I start Godot since I compiled on last Monday : Archlinux 64 bits (Antergos distrib): it takes like 2 minutes each time for my current project (small, but with a lot of AnimatedSprites).

[HummusSamurai]

Also happens on Fedora Linux 27, it's very odd.

[spookyboo]

I never used Godot before (Ogre3D die-hard user) and I wanted to give Godot_v3.0-beta2_win64.exe a go. So, I created an empty project and its scanning file 'ftbitmap.obj' forever and displays 'unicode error: invalid skip' in the log (looping forever). I cannot get it to work again even after multiple restarts. The editor is unusable and although I think Godot seems very good, this chases newbies (like me) away.

[HummusSamurai]

@spookyboo I am sorry that this has happened to you.

I suggest that you open a separate issue and copy-paste your comment so more devs can see it, and attach the problematic project as a zip file.

Have you tried removing the 'ftbitmap.obj' file and then restarting? That could make the engine work again.

And also regarding "this chases newbies (like me) away", I think it is better for newcomers to use the official stable 2.1 version that's on the website under Downloads, as the beta has a disclaimer on the news page that says "IMPORTANT: This is a beta build, which means that it is not suitable for use in production, nor for press reviews of what Godot 3.0 would be on its release."

[spookyboo]

Godot just scans my entire c: drive, which contains a large amount of textures and 3d models. Why does it do that? I know that 3.0 is a beta. That is why I tried it. To perform a monkey test and determine whether a complete newbie is able to find its way ;-)

[Ace-Dragon]

To Spookyboo; I just place the .dae files in a special assets folder within the project directory (Godot definitely isn't scanning my entire drive, so I'm not sure what you're doing).

[HummusSamurai]

@spookyboo Godot definitely shouldn't be scanning the entire c: drive, by design it only scans the project folder.

If it's scanning the c: drive then it's definitely a bug.

[spookyboo]

Ok, I think it was my fault. It was scanning 'my documents' for some reason (the project was in a subdir of 'my documents'). I deleted the project and tried to reproduce the scenario, without any luck. Thanks for the help. Now start with some tutorials