Houdini: Cleanup after publishing

[MustafaJafar]

Changelog Description

This PR adds a collector to add the parent of the exported files to the clean up list.

Additional info

Cautions, This feature can be dangerous.

Testing notes:

1. Publish different product types from Houdini
2. All exported files should be gone after the integration finishes.

[MustafaJafar]

As far as I tested, this PR works with local publishing, I'm not sure yet about farm publishing.
Let me add my thoughts as I 'm not sure if this PR addresses the issue correctly.

---

From digging around discord and the code I found out that there are plugins that do the cleaning up:

• ayon-core/plugins/cleanup.py which remove the staging dir if it were in temp path
• ayon_core/plugins/publish/cleanup_explicit.py which removes in path in this list context.data["cleanupFullPaths"]
• ayon_core/plugins/publish/cleanup_farm.py which I'm not sure how to use it.

---

tbh, I find this feature dangerous and requires pipeline admins to be aware of how to use it.
I think these settings are related, I don't know if they are working with Houdini 😅 :

• ayon+settings://core/tools/publish/custom_staging_dir_profiles
• ayon+settings://core/publish/CleanUp
• ayon+settings://core/publish/CleanUpFarm

[MustafaJafar]

Mentioning @BigRoy as I couldn't add him as a reviewer in this PR.

[MustafaJafar]

Here are the current test results on my side.
I tested these product types and it deletes files as expected

        "camera",
        "pointcache",
        "imagesequence",
        "mantraifd",
        "review",
        "staticMesh",
        "usd",
        "vdbcache"

FYI, I found out a bug in mantra ifd, I'll fix it in a separate PR, here #75

From publisher log:

Removed files:
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\S_BigRoboMain\S_BigRoboMain.fbx
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\cameraMain\cameraMain.abc
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\imagesequenceMain\imagesequenceMain.1001.exr
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\imagesequenceMain\imagesequenceMain.1002.exr
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\mantraifdMain\mantraifdMain.1001.ifd
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\mantraifdMain\mantraifdMain.1002.ifd
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\pointcacheABC\pointcacheABC.abc
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\pointcacheBgeo\pointcacheBgeo.1001.bgeo
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\pointcacheBgeo\pointcacheBgeo.1002.bgeo
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\reviewMain\reviewMain.1001.png
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\reviewMain\reviewMain.1002.png
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\reviewMain\reviewMain_h264focal_length_burnin.mp4
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\usdMain\usdMain.usd
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\vdbcacheMain\vdbcacheMain.1001.vdb
\\storage\work\ayon_projects\Robo\Assets\Character\BigRobo\work\cfx\ayon\vdbcacheMain\vdbcacheMain.1002.vdb

[BigRoy]

The Mantra IFD and .rs files being deleted doesn't seem like what you'd want right? Or are those IFD files actually being published? Aren't those just intermediate rendering files?

[moonyuet]

I encountered this issue during installing addon

Traceback (most recent call last):
      File "/backend/ayon_server/addons/definition.py", line 84, in versions
        self.init_addon(version_dir)
      File "/backend/ayon_server/addons/definition.py", line 148, in init_addon
        module = import_module(vname, server_module_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/backend/ayon_server/addons/utils.py", line 18, in import_module
        spec.loader.exec_module(module)
      File "<frozen importlib._bootstrap_external>", line 940, in exec_module
      File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
      File "/addons/houdini/0.3.7/server/__init__.py", line 5, in <module>
        from .settings import HoudiniSettings, DEFAULT_VALUES
      File "/addons/houdini/0.3.7/server/settings/__init__.py", line 1, in <module>
        from .main import (
      File "/addons/houdini/0.3.7/server/settings/main.py", line 15, in <module>
        from .publish import (
      File "/addons/houdini/0.3.7/server/settings/publish.py", line 100, in <module>
        class PublishPluginsModel(BaseSettingsModel):
      File "/addons/houdini/0.3.7/server/settings/publish.py", line 111, in PublishPluginsModel
        default_factory=BasicValidateModel,
                        ^^^^^^^^^^^^^^^^^^
    NameError: name 'BasicValidateModel' is not defined

We need to change BasicValidateModel to BasicEnabledStatesModel after this

[BigRoy]

Tested out the PR for pointcache and vdb sequences - works for me:

And if the folder was empty before publishing (or didn't exist) it also removes it - which is a debug log:

[moonyuet]

Tested with Review, Camera and Mantra ROP(both local and farm)
Works smoothly.

[BigRoy]

I pushed a version bump - only thing to double check @MustafaJafar is Kayla's comment. If you think that's all resolved / fine as it is feel free to mark resolved and merge the PR.

Did another pointcache + vdb test run just now. Works fine.

[moonyuet]

I pushed a version bump - only thing to double check @MustafaJafar is Kayla's comment. If you think that's all resolved / fine as it is feel free to mark resolved and merge the PR.

Did another pointcache + vdb test run just now. Works fine.

we need to dimiss Ondrej's review to merge it btw.

The request has been implemented, but Ondrej is not around to review now.