Maya: Load published workfile as template

[BigRoy]

Changelog Description

Allows to load a published maya workfile and use it as a workfile template.

This way, someone can create a workfile template using the Create Template Placeholders:

Then publish their workfile:

And another artist can use that published workfile as a starting template from the loader:

Additional info

This is basically a 'simple implementation' of this OpenPype issue: #742

Includes a fix for incorrect usage of placeholder.data["family"] that existed in the code which should have been placeholder.data["product_type"]. There are some other bugs with the placeholder logic - but those should be fixed by ynput/OpenPype#327

Anyway, please report anything you find during testing.

Testing notes:

1. Follow above steps to create a published workfile that can act as workfile template
2. Load it as template via loader

---

1. Usual workfile templates (not done via loader) should also continue to work, like AYON > Template Builder > Build Workfile from Template

[MustafaJafar]

When I checked out this PR. I can no longer see AYON menu when launching Maya (I have 2022.4). Not sure if it's related to this PR but it doesn't happen in develop branch.

file -f -new;
// untitled // 
commandPort -securityWarning -name commandportDefault;
onSetCurrentLayout "General";
// Loading Bifrost version 2.3.1.0-202110140143-ddfa15b
// Bifrost: Loading library: Amino, from: Autodesk.
// Bifrost: Loading library: AminoMayaTranslation, from: Autodesk.
// Bifrost: Loading library: bif, from: Autodesk.
// Bifrost: Loading library: bifrostObjectMayaTranslations, from: Autodesk.
// Bifrost: Loading library: geometries, from: Autodesk.
// Bifrost: Loading library: fluids, from: Autodesk.
// Bifrost: Loading library: particles, from: Autodesk.
// Bifrost: Loading library: file, from: Autodesk.
// Bifrost: Loading library: mpm, from: Autodesk.
// Bifrost: Loading library: modeling, from: Autodesk.
// Bifrost: Loading library: nucleus, from: Autodesk.
// Bifrost: Loading library: simulation, from: Autodesk.
// Bifrost: Loading library: riv_types, from: Autodesk.
// Bifrost: Loading library: riv, from: Autodesk.
// Bifrost: Loading library: scatter_pack, from: Autodesk.
// Bifrost: Loading library: graphs, from: Autodesk.
// Warning: file: C:\Users\Mustafa Taher\Documents\maya\2022\prefs\filePathEditorRegistryPrefs.mel line 4: filePathEditor: Attribute 'aiImage.filename' is invalid or is not designated 'usedAsFilename'. // 
// Warning: file: C:\Users\Mustafa Taher\Documents\maya\2022\prefs\filePathEditorRegistryPrefs.mel line 5: filePathEditor: Attribute 'aiPhotometricLight.aiFilename' is invalid or is not designated 'usedAsFilename'. // 
// Warning: file: C:\Users\Mustafa Taher\Documents\maya\2022\prefs\filePathEditorRegistryPrefs.mel line 6: filePathEditor: Attribute 'aiStandIn.dso' is invalid or is not designated 'usedAsFilename'. // 
// Warning: file: C:\Users\Mustafa Taher\Documents\maya\2022\prefs\filePathEditorRegistryPrefs.mel line 7: filePathEditor: Attribute 'aiVolume.filename' is invalid or is not designated 'usedAsFilename'. // 
// AbcExport v1.0 using Alembic 1.7.5 (built Nov 30 2020 18:40:46)
evalDeferred "shaderBallRendererMenuUpdate";
// Warning: line 1: filePathEditor: Attribute 'aiStandIn.dso' and label 'Standin' have been saved already. // 
// Warning: line 1: filePathEditor: Attribute 'aiVolume.filename' and label 'VDB' have been saved already. // 
// Warning: line 1: filePathEditor: Attribute 'aiImage.filename' and label 'Image' have been saved already. // 
// Warning: line 1: filePathEditor: Attribute 'aiPhotometricLight.aiFilename' and label 'IES' have been saved already. // 
# Warning: [mtoa] Registering UI for translator "polymesh" for Maya node mvUsdCompoundShape, but node has no translators. Did you mean to call registerAETemplate? # 
# Warning: [mtoa] Registering UI for translator "curves" for Maya node mvUsdCompoundShape, but node has no translators. Did you mean to call registerAETemplate? # 
# Warning: [mtoa] Registering UI for translator "points" for Maya node mvUsdCompoundShape, but node has no translators. Did you mean to call registerAETemplate? # 
# Warning: [mtoa] Registering UI for translator "mesh_light" for Maya node mvUsdCompoundShape, but node has no translators. Did you mean to call registerAETemplate? # 
Error loading Extension UI C:/Program Files/Multiverse/8.2.0/Maya/2022/arnold/extensions\mvRenderUsdArnoldExt421.py
# Traceback (most recent call last):
# #   File "C:\Program Files\Autodesk\Arnold\maya2022\scripts\mtoa\api\extensions.py", line 30, in loadExtensionUI
    return imp.load_module(name, fp, pathname, description)
# #   File "C:\Program Files\Autodesk\Maya2022\Python37\lib\imp.py", line 234, in load_module
    return load_source(name, filename, file)
# #   File "C:\Program Files\Autodesk\Maya2022\Python37\lib\imp.py", line 171, in load_source
    module = _load(spec)
#   File "<frozen importlib._bootstrap>", line 696, in _load
#   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
#   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
#   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
# #   File "C:/Program Files/Multiverse/8.2.0/Maya/2022/arnold/extensions\mvRenderUsdArnoldExt421.py", line 181, in <module>
    callbacks.addNodeAddedCallback(mvUsdCompoundShapeNodeAddedCallback, kShapeNodeName, False)
# #   File "C:\Program Files\Autodesk\Arnold\maya2022\scripts\mtoa\callbacks.py", line 106, in addNodeAddedCallback
    manageCallback(om.MDGMessage.addNodeAddedCallback(cb, nodeType))
# RuntimeError: (kInvalidParameter): Invalid parameter
import arnold
import mtoa
import mtoa.cmds.registerArnoldRenderer;mtoa.cmds.registerArnoldRenderer.registerArnoldRenderer()
updateRenderOverride;
// AbcImport v1.0 using Alembic 1.7.5 (built Nov 30 2020 18:40:46)
updateRendererUI;
// Warning: Panel size cannot accommodate all requested Heads Up Display elements. // 

[BigRoy]

When I checked out this PR. I can no longer see AYON menu when launching Maya (I have 2022.4). Not sure if it's related to this PR but it doesn't happen in develop branch.

Thanks, it seemed TypedDict is not present yet in that Python 3 version. So I'll remove the type hints.

# Error: cannot import name 'TypedDict' from 'typing' (C:\Program Files\Autodesk\Maya2022\Python37\lib\typing.py)
# Traceback (most recent call last):
#   File "<maya console>", line 1, in <module>
# ImportError: cannot import name 'TypedDict' from 'typing' (C:\Program Files\Autodesk\Maya2022\Python37\lib\typing.py) # 

Just so you're aware. Note that the startup issues for these usually are reported in the "Output Window" of Maya since it's the console output before the Maya UI is initialized.

As such, I suspect you should've seen an error somewhat similar to the error I posted here in the Maya Output Window:

For completeness sake, Maya 2022 is Python 3.7.7 and TypedDict seems implemented since Py 3.8

[LiborBatek]

@BigRoy seems like nice feature! quick question...so in theory if using that published workfile via action Load as Template it just opens the workfile with placeholders or actually build up the work file from those placeholders as when triggering "Build from workfile" ? Just to clarify it for me beforehand :) thx

[BigRoy]

@BigRoy seems like nice feature! quick question...so in theory if using that published workfile via action Load as Template it just opens the workfile with placeholders or actually build up the work file from those placeholders as when triggering "Build from workfile" ? Just to clarify it for me beforehand :) thx

It opens + builds similar to regular workfile templates but without the need to having it set up in settings as a workfile template for a particular task. Basically we have a project where we have a few templates the artists can pick from, they go to the "templates" asset on the project and then pick the workfile template product that fits their need and use that as their starting point.

So it's basically exactly like workfile templates- but building from a published product instead of a hardcoded filepath in the project settings.

[LiborBatek]

Works for me... tested in maya 2024

It does what it should...

[BigRoy]

Thanks @iLLiCiTiT - submitted your review suggestions.

[LiborBatek]

@iLLiCiTiT I redid the testing after those commits and all working fine, can be merged...

[MustafaJafar]

It works on my side.

I'm wondering:

1. I found out that the name space of loaded template follows my scene/asset/task. so, is this the expected behavior ?
2. I can only load one template per scene, is this the expected behavior ? (what about combining multiple templates into one big template ?)

[LiborBatek]

I'm wondering:

1. I found out that the name space of loaded template follows my scene/asset/task. so, is this the expected behavior ?
2. I can only load one template per scene, is this the expected behavior ? (what about combining multiple templates into one big template ?)

@MustafaJafar 1) yes its correct...it resolves the asset name by the namespace (each asset have its own)

2. Yes, it pretty much follows the Template Builder Workfile tool which allows to use one file at the time, so also ok.

The point 2) and its functionality as it is now, if any changes should be made then in separate PR, not related to this one IMHO

[BigRoy]

• I found out that the name space of loaded template follows my scene/asset/task. so, is this the expected behavior ?

• I can only load one template per scene, is this the expected behavior ? (what about combining multiple templates into one big template ?)

As @LiborBatek correctly pointed out. That is indeed the correct behavior - if any of those should change it's outside of the scope of this PR - it's the way the Workfile Templates functionality was designed.

---

Should be ready to merge?

[tokejepsen]

Couple of errors when building the template, which is technically not related to this PR but hard to test without fixing it.

# ==================
# Maya command error
# ==================
# Traceback (most recent call last):
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 722, in populate_scene_placeholders
#     placeholder_plugin.populate_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 246, in populate_placeholder
#     self.populate_load_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1650, in populate_load_placeholder
#     self.load_succeed(placeholder, container)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 275, in load_succeed
#     self._parent_in_hierarchy(placeholder, container)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 334, in _parent_in_hierarchy
#     cmds.parent(node, world=True)
RuntimeError: Maya command error
# # *** WRN: >>> { <ayon_core.hosts.maya.api.workfile_template_builder.MayaTemplateBuilder object at 0x0000020A1BA750A0> }: [ Failed to process placeholder |Context_model_folder with plugin MayaPlaceholderLoadPlugin ] 
# ========
# 'subset'
# ========
# Traceback (most recent call last):
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 722, in populate_scene_placeholders
#     placeholder_plugin.populate_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 246, in populate_placeholder
#     self.populate_load_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1601, in populate_load_placeholder
#     for representation in self._reduce_last_version_repre_entities(
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1553, in _reduce_last_version_repre_entities
#     product_name = repre_context["subset"]
KeyError: 'subset'

[BigRoy]

_reduce_last_version_repre_entities

@tokejepsen Would you like me to include a fix for this in this PR? It does look to be a bug that's present outside of this PR so might be best suited for another PR?

[tokejepsen]

Would you like me to include a fix for this in this PR? It does look to be a bug that's present outside of this PR so might be best suited for another PR?

Yeah, lets get it in here so we can test in context.

[BigRoy]

# ==================
# Maya command error
# ==================
# Traceback (most recent call last):
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 722, in populate_scene_placeholders
#     placeholder_plugin.populate_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 246, in populate_placeholder
#     self.populate_load_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1650, in populate_load_placeholder
#     self.load_succeed(placeholder, container)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 275, in load_succeed
#     self._parent_in_hierarchy(placeholder, container)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 334, in _parent_in_hierarchy
#     cmds.parent(node, world=True)

So this error is resolved by parenting your placeholders to a group (making sure they are not top level locators). This is a limitation of the workfile templates in Maya which should be solved in a separate PR (and I have a fix for as well which is in ynput/OpenPype#327 which I still need to separate in smaller PRs.

---

# ========
# 'subset'
# ========
# Traceback (most recent call last):
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 722, in populate_scene_placeholders
#     placeholder_plugin.populate_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\hosts\maya\api\workfile_template_builder.py", line 246, in populate_placeholder
#     self.populate_load_placeholder(placeholder)
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1601, in populate_load_placeholder
#     for representation in self._reduce_last_version_repre_entities(
#   File "C:\Users\tokejepsen\ayon-core\client\ayon_core\pipeline\workfile\workfile_template_builder.py", line 1553, in _reduce_last_version_repre_entities
#     product_name = repre_context["subset"]
KeyError: 'subset'

This one I can't seem to reproduce. When printing the representation entity, I get for example this:

{
    "attrib": {
        "fps": null,
        "template": "{root[work]}/{project[name]}/{hierarchy}/{folder[name]}/publish/{product[type]}/{product[name]}/v{version:0>3}/{project[code]}_{folder[name]}_{product[name]}_v{version:0>3}<_{output}><.{frame:0>4}><_{udim}>.{ext}",
        "resolutionWidth": null,
        "pixelAspect": null,
        "frameEnd": null,
        "clipOut": null,
        "clipIn": null,
        "frameStart": null,
        "extension": null,
        "path": "C:\\projects\\ayontest\\asset\\char_hero\\publish\\pointcache\\pointcacheMain\\v005\\ynts_char_hero_pointcacheMain_v005.abc",
        "description": null,
        "resolutionHeight": null,
        "handleStart": null,
        "handleEnd": null
    },
    "versionId": "73bfe327f76211eebb65b4969159e74e",
    "active": true,
    "createdAt": "2024-04-10T17:47:51.815979+00:00",
    "files": [
        {
            "hash": "ynts_char_hero_pointcacheMain_v005,abc|1712771271,0777626|3302",
            "id": "73d84350f76211ee95ccb4969159e74e",
            "name": "ynts_char_hero_pointcacheMain_v005.abc",
            "size": 3302,
            "path": "{root[work]}/ayontest/asset/char_hero/publish/pointcache/pointcacheMain/v005/ynts_char_hero_pointcacheMain_v005.abc"
        }
    ],
    "id": "73cb720ef76211ee8c12b4969159e74e",
    "context": {
        "ext": "abc",
        "root": {
            "work": "C:/projects"
        },
        "task": {
            "name": "modeling",
            "type": "Modeling",
            "short": "mdl"
        },
        "user": "admin",
        "asset": "char_hero",
        "family": "pointcache",
        "folder": {
            "name": "char_hero"
        },
        "subset": "pointcacheMain",
        "product": {
            "name": "pointcacheMain",
            "type": "pointcache"
        },
        "project": {
            "code": "ynts",
            "name": "ayontest"
        },
        "version": 5,
        "username": "admin",
        "hierarchy": "asset",
        "representation": "abc"
    },
    "data": {
        "context": {
            "ext": "abc",
            "root": {
                "work": "C:/projects"
            },
            "task": {
                "name": "modeling",
                "type": "Modeling",
                "short": "mdl"
            },
            "user": "admin",
            "asset": "char_hero",
            "family": "pointcache",
            "folder": {
                "name": "char_hero"
            },
            "subset": "pointcacheMain",
            "product": {
                "name": "pointcacheMain",
                "type": "pointcache"
            },
            "project": {
                "code": "ynts",
                "name": "ayontest"
            },
            "version": 5,
            "username": "admin",
            "hierarchy": "asset",
            "representation": "abc"
        }
    },
    "name": "abc"
}

Which interestingly enough also contains the backwards compatible keys in context - I can update it to {folder[name]} and {product[name]}. I just wonder whether for now we're safer to be 'backwards compatible' instead of forward compatible? Since I have a feeling existing publishes will not contain the folder[name] keys for example?

The question is. Why does it fail for you @tokejepsen ?

@iLLiCiTiT any preference?

[tokejepsen]

So this error is resolved by parenting your placeholders to a group (making sure they are not top level locators). This is a limitation of the workfile templates in Maya which should be solved in a separate PR (and I have a fix for as well which is in #327 which I still need to separate in smaller PRs.

@BigRoy are you suggesting to wait for ynput/OpenPype#327 (or a smaller PR that fixes this issue) to be merged?

[BigRoy]

are you suggesting to wait for ynput/OpenPype#327 (or a smaller PR that fixes this issue) to be merged?

I'm suggesting to test now with a parent group, because that's how it currently only works in all maya workfile templates. The bug is - afaik - unrelated to this PR. So, if that's the only issue - then merge this PR and solve that in another; no need to wait.

[LiborBatek]

speaking of future enhancements... how about populating the template file with publish instances and pre-occupy it with groups containing the placeholder(s) ? is it something reasonable and possible event.? would make huge impact on automation...

[tokejepsen]

how about populating the template file with publish instances and pre-occupy it with groups containing the placeholder(s) ? is it something reasonable and possible event.?

Think its better to support not being parent to anything. Its small bugfix which will come in future PRs eventually.

[tokejepsen]

The question is. Why does it fail for you @tokejepsen ?

@BigRoy not sure why its not failing for anyone else. This is a freshly published workfile. Here is the the representation;

{
    "asset": "shots",
    "ext": "abc",
    "folder": {
        "name": "shots"
    },
    "hierarchy": "",
    "product": {
        "name": "modelUnique",
        "type": "model"
    },
    "project": {
        "code": "com",
        "name": "demo_Commercial"
    },
    "representation": "abc",
    "root": {
        "work": "P:/PROJECTS"
    },
    "task": {
        "name": "modeling",
        "short": "mdl",
        "type": "Modeling"
    },
    "user": "admin",
    "username": "admin",
    "version": 7
}

[LiborBatek]

how about populating the template file with publish instances and pre-occupy it with groups containing the placeholder(s) ? is it something reasonable and possible event.?

Think its better to support not being parent to anything. Its small bugfix which will come in future PRs eventually.

Yeah I get your point and fully agree...I was more refering to pre create the Publish instances too...in Template file.

as I did test it now, it seems working sort of. I have test it with Layout and Render instances for publishing, there always need to be something put into their maya sets otherwise got lost during building phase.

One more issue being with Render instance... as long as no render layer present (in new empty scene for applying the template file in ) the publish instance is not created at all.

The good thing is if any publish instances present in the template file they get proper context when builded...

So its almost functional in terms of prepopulating the template file wtih publishes....just need some tweaking imho

Here is my scene builded from the template containing even publish instances (builded automatically when using particular template file)

@BigRoy do you have something to add to that?

[BigRoy]

Think its better to support not being parent to anything. Its small bugfix which will come in future PRs eventually.

I understand your reasoning @tokejepsen - I was just hoping to avoid now also implementing that fix 'separately' that's already implemented in the other PR ynput/OpenPype#327 currently since it comes with a bit more refactoring/fixes. Fixing that now separately just means that that fix would still get superseded by those changes down the line.

Anyway, will take a look at how much code is involved to incorporate in this PR.

---

The question is. Why does it fail for you @tokejepsen ?

@BigRoy not sure why its not failing for anyone else. This is a freshly published workfile. Here is the the representation;

{
    "asset": "shots",
    "ext": "abc",
    "folder": {
        "name": "shots"
    },
    "hierarchy": "",
    "product": {
        "name": "modelUnique",
        "type": "model"
    },
    "project": {
        "code": "com",
        "name": "demo_Commercial"
    },
    "representation": "abc",
    "root": {
        "work": "P:/PROJECTS"
    },
    "task": {
        "name": "modeling",
        "short": "mdl",
        "type": "Modeling"
    },
    "user": "admin",
    "username": "admin",
    "version": 7
}

That doesn't seem like a published workfile, if it's extension abc, but I suppose you mean a published model from a newly created workfile. I guess I'll just need to retry creating a publish again and try again.

---

@LiborBatek there's definitely good merit in having a workfile template come with good publish instances by default, etc. - however I'd say that's maya workfile template system issue if currently hard to do or not possible and should be unrelated to this particular PR. As such, I think your question merits an issue of its own if it's currently incapable of working the way you want it to work.

I know that e.g. Nuke also has "Creator" placeholders or whatever to create placeholders for publishing instances. I suppose it'd be someting similar on the maya side then.

[BigRoy]

Think its better to support not being parent to anything. Its small bugfix which will come in future PRs eventually.

Fixed with e75f44f

---

Unfortunately I can't reproduce the subset error on this branch myself. Not with existing products, nor new ones generated from new workfiles. Also recreated a workfile template, again worked with success.

@iLLiCiTiT preference on a fix for that? I suppose we should somehow support existing workfile templates and publishes that may not have folder name, etc. as context keys?

The issue also isn't necessarily related to this PR since I'd expect it also exists outside of this PR. So we might be best fixing that in a separate PR anyway.

[iLLiCiTiT]

The issue also isn't necessarily related to this PR since I'd expect it also exists outside of this PR. So we might be best fixing that in a separate PR anyway.

Agree. It is bug in workfiles template builder.

Created PR #416

[tokejepsen]

Tested successfully in Maya 2023.