Use product name templates for render products on farm publish #447

antirotor · 2024-04-24T17:14:04Z

Changelog Description

Product names for render instance were hardcoded in the Deadline logic, not following product name template use in other places. This PR adds ability to use product name templates with all necessary keys to influence how render products are named. It also collects information about renderlayer (ROP node name) to anatomy data along with AOV name so it can be used for template formatting.

Additional info

Product names for instances to be published were hardcoded directly in the script here:

ayon-core/client/ayon_core/pipeline/farm/pyblish_functions.py

Lines 573 to 597 in 8f42c50

 product_type = skeleton["productType"] 

 if not s_product_name.startswith(product_type): 

 group_name = '{}{}{}{}{}'.format( 

 product_type, 

 task[0].upper(), task[1:], 

 s_product_name[0].upper(), s_product_name[1:]) 

 else: 

 group_name = s_product_name 

 # if there are multiple cameras, we need to add camera name 

 expected_filepath = col[0] if isinstance(col, (list, tuple)) else col 

 cams = [cam for cam in cameras if cam in expected_filepath] 

 if cams: 

 for cam in cams: 

 if not aov: 

 product_name = '{}_{}'.format(group_name, cam) 

 elif not aov.startswith(cam): 

 product_name = '{}_{}_{}'.format(group_name, cam, aov) 

 else: 

 product_name = "{}_{}".format(group_name, aov) 

 else: 

 if aov: 

 product_name = '{}_{}'.format(group_name, aov) 

 else: 

 product_name = '{}'.format(group_name)

This was bypassing any template set in ayon+settings://core/tools/creator/product_name_profiles. With this change, one can set template like this:

{product[type]}<{Renderlayer}><_{Aov}>

resulting in:

renderMainlayer_normal

It is up to TD to set correctly such template to make sure, product name will be stable enough. As a side-effect, optional keys can be used now in product name templates adding flexibility (and perhaps some danger of misuse).

renderlayer is now set on instance along with aov and is collected to anatomy data, so it can be used in template formatting so you can use template for renders like:

{root[work]}/{project[name]}/{hierarchy}/{asset}/publish/{family}/{subset}/{@version}/<{renderlayer}/>

Note

Already collected anatomy data (when available) could be passed to get_product_name() to allow even more keys to be used, but is there any real use case for it?

Also, as a point for discussion - there is variant that is now ignored in this case. It could be reasonable to use it exactly for this and move templating to "variant templates" but maybe it is breaking settings to even smaller unmanagable bits.

Testing notes:

Set product name template to use renderlayer or aov keys (can be optional using<{key}> syntax)
Publish renders from DCC supporting setting of these keys (like Maya and Houdini)
Use {renderlayer} or {aov} in anatomy templates

collect rop name to instance to align with the `renderlayer` in maya

if `aov` and `renderlayer` is set on instance, use it in anatomy data for formatting templates

allow using optional keys in formatting of product name templates

remove hardcoded instance names in favor of product name templates

moonyuet · 2024-04-25T06:22:43Z

Should we also test to see if it works with 3dsmax multicam renders?

antirotor · 2024-04-25T09:26:41Z

Should we also test to see if it works with 3dsmax multicam renders?

definitely - there is issue with cameras that was even in original code and I've kept the logic to be fixed in some other PR. Problem with that logic is that it is trying to find camera name from the camera list in the file name and first camera found is used. But that is very fragile because it cannot deal with ambiguous names (what if your camera is named the same as AOV or other part) and so on.

LiborBatek

I have setup product template using the renderlayer token but constantly getting errors when trying publish

Traceback (most recent call last):
  File "C:\Users\lbate\AppData\Local\Ynput\AYON\dependency_packages\ayon_2402141620_windows.zip\dependencies\pyblish\plugin.py", line 527, in __explicit_process
    runner(*args)
  File "C:\Work\REPO\ayon-core\client\ayon_core\plugins\publish\collect_anatomy_instance_data.py", line 56, in process
    self.fill_latest_versions(context, project_name)
  File "C:\Work\REPO\ayon-core\client\ayon_core\plugins\publish\collect_anatomy_instance_data.py", line 263, in fill_latest_versions
    product_entities = list(ayon_api.get_products(
  File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\server_api.py", line 4248, in get_products
    parsed_data = query.query(self)
  File "C:\Program Files\Ynput\AYON 1.0.2\dependencies\ayon_api\graphql.py", line 343, in query
    raise GraphQlQueryFailed(response.errors, query_str, variables)
ayon_api.exceptions.GraphQlQueryFailed: GraphQl query Failed: Name 'LightingMain_{renderlayer}' does not match regex '^[a-zA-Z0-9_]([a-zA-Z0-9_\.\-]*[a-zA-Z0-9_])?$' on item 'project/products' (Line 3 Column 5)

BigRoy · 2024-05-02T15:24:38Z

client/ayon_core/hosts/houdini/plugins/publish/collect_karma_rop.py

+ # to align with maya render layers
+ instance.data["renderlayer"] = rop.name()


Sorry, what? Why is the ROP node's name used as the renderlayer? That seems odd?

Isn't it much easier if a studio wants this for them to us $OS as part of the productName or subset attributes on the node itself so that it's editable by the studio, requires no additional changes?

Yes, but then you can't use it in anatomy data. One thing is product name, the other is to use it somewhere else in path. I agree that render layer is somewhat arbitrary name in Houdini context, but most of the DCCs are using render layer is one way or the other and I wouldn't spam the anatomy data with every DCC specific term.

BigRoy · 2024-05-02T15:26:36Z

client/ayon_core/plugins/publish/collect_anatomy_instance_data.py

+ # make render layer name (or ROP name) available in the
+ # templates
+ render_layer = instance.data.get("renderlayer")
+ if render_layer:
+ anatomy_data["renderlayer"] = render_layer
+
+ # make AOV name if present available for templates
+ aov = instance.data.get("aov")
+ if aov:
+ anatomy_data["aov"] = aov


What's "renderlayer" if not part of the product name already? This seems like an odd decision to allow renderlayer (which as far as I know doesn't really have a valid connotation in Houdini?) to be part of the anatomy template? Or?

you might want to have render layer as a part of the path (not the whole product name).

you might want to have render layer as a part of the path (not the whole product name).

When doing so - what would you anticipate the product name be instead?

Coming back to this from your PR description:

This was bypassing any template set in ayon+settings://core/tools/creator/product_name_profiles. With this change, one can set template like this:

{product[type]}<{Renderlayer}><_{Aov}>

resulting in:

renderMainlayer_normal

It is up to TD to set correctly such template to make sure, product name will be stable enough. As a side-effect, optional keys can be used now in product name templates adding flexibility (and perhaps some danger of misuse).

renderlayer is now set on instance along with aov and is collected to anatomy data, so it can be used in template formatting so you can use template for renders like:

In practice a user should usually never see this. They see the product name or representation in the loader or the manager. Which raises the question - how come its this crucial in the published filepath which usually isn't something that is displayed to the artists anyway?

I think it is just an option to categorize things when you need to have access to them outside loaders/pipeline. We get those requests from time to time when you have part that is not integrated and you need to access the files directly. Then dropping the whole render layer folder to DCC with all of it's AOV sequences can speed things up.

Totally agree that's valuable - but doesn't that mean we should just expose these as data on the representations that can be picked up by the Delivery action instead? Because I believe this currently requires the template to actually use {aov} in the published filepaths just so one can use that also in the delivery template since I think it stores only the used context on publish, right?

If not and this is solely for "delivery" templates I think it could be nice.

You mean to "force" it to context on representation even without it being used in the template so it can be picked up later on by delivery action? Like we are storing udim here

ayon-core/client/ayon_core/plugins/publish/integrate.py

Lines 856 to 859 in 4d84ed8

# Explicitly store the full list even though template data might

# have a different value because it uses just a single udim tile

if repre.get("udim"):

repre_context["udim"] = repre.get("udim") # store list

(and also frame somewhere)

Yes, similar to that. Not sure if that's "good design" - but that was indeed what I meant.

I'd expect this to way more valuable for delivery templates than production templates since in production templates it may be likely you will have duplicity anyway since usually the AOV is part of the product name as well.

client/ayon_core/pipeline/farm/pyblish_functions.py

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

…-product-names-templated

…om:ynput/ayon-core into enhancement/render-product-names-templated

MustafaJafar · 2024-05-22T12:41:03Z

Hey, I was trying to test this PR in Houdini and I'm confused.

which one should I change ? and what is the expected behavior ?

ayon+settings://core/tools/creator/product_name_profiles
ayon+anatomy://{my-project}/templates/publish/1 render template.

As far as I know updating product_name_profiles affect the product name in the creator tab. and how does that support renderLayer (rop name) before actually creating the rop ?

Also, I wonder how does that work with run-time instances that are created in https://github.com/ynput/ayon-core/blob/develop/client/ayon_core/hosts/houdini/plugins/publish/collect_local_render_instances.py

LiborBatek · 2024-05-22T13:00:05Z

@MustafaJafar I guess you need to change both...one alters the product name (aka rendered file name) and the other alter or sets the folder path for the rendered sequence (again using tokens like <aov>) which wasnt possible before I guess.

MustafaJafar · 2024-05-22T18:27:42Z

I tested this PR furtherly and Here my results in Houdini:
On farm it works as expected.

Render job	Render after Integration

On local rendering, I think some work needs to be added to https://github.com/ynput/ayon-core/blob/develop/client/ayon_core/hosts/houdini/plugins/publish/collect_local_render_instances.py
I'll add my local edits from my testing as a PR to this PR.
Edit: Here it's #543

LiborBatek

Finally I could properly test it and thanks to @antirotor setup correctly my templates so it successfully finnished my DL job.

All seems working well.

BigRoy · 2024-05-23T11:43:00Z

client/ayon_core/pipeline/farm/pyblish_functions.py

+ host_name=instance.context.data["hostName"],
+ product_type=skeleton["productType"],
+ dynamic_data=dynamic_data,
+ variant=instance.data.get('variant', ''),


Do we really want to fall back to an empty variant?

BigRoy · 2024-05-23T11:47:22Z

client/ayon_core/pipeline/farm/pyblish_functions.py

+ product_name = get_product_name(
+ project_name=instance.context.data["projectName"],
+ task_name=task_name,
+ task_type=instance.data["taskEntity"]["taskType"],
+ host_name=instance.context.data["hostName"],
+ product_type=skeleton["productType"],
+ dynamic_data=dynamic_data,
+ variant=instance.data.get('variant', ''),
+ project_settings=instance.context.data.get("project_settings"),
+ )


I'm a bit worried about the fact that this now seems to require templates to use aov to ensure multiple AOVs become a unique product? Does this mean that with 'older templates' this would now break? Or what is ensuring that they become separate products?

I'm a bit worried about the fact that this now seems to require templates to use aov to ensure multiple AOVs become a unique product? Does this mean that with 'older templates' this would now break? Or what is ensuring that they become separate products?

older templates were ignored anyway - product name was hardcoded here so no matter what you had in the settings, it wasn't used. Now this ofc rises interesting question about transition - because with this, the existing templates will be suddenly used and without change, it will probably break. 🤔

…emplated-local-houdini-render Use product name templates for render products on local Houdini render

MustafaJafar · 2024-06-21T13:51:05Z

client/ayon_core/hosts/houdini/plugins/publish/collect_local_render_instances.py

@@ -64,6 +64,8 @@ def process(self, instance):
 expectedFiles = next(iter(instance.data["expectedFiles"]), {})

 product_type = "render" # is always render
+ # TODO: Match the value of product_group with the value of
+ # group_name in farm pyblish functions.
 product_group = get_product_name(
 context.data["projectName"],
 context.data["taskEntity"]["name"],


for reference,
I gave this PR a test. and was going to mention an issue with the product group name in local Houdini render.
Then, I figured out there was a comment about it in the code.

Anyways, it doesn't work because it depends on the profile template name and the function call doesn't pass the necessary dynamic data.
Also, group name shouldn't include the {aov} token, therefore should it use a different template name ?

how come it was working for @LiborBatek ?

how come it was working for @LiborBatek ?

because tested for maya I guess...

antirotor · 2024-07-01T15:08:29Z

This need some transitional option in the settings so default template can be configured.

antirotor · 2024-07-02T14:03:56Z

This will be reopened in Houdini addon new repository.

antirotor added 5 commits April 24, 2024 18:49

🔥 remove interface file

2d403b3

🎨 collect rop name

43f2fed

collect rop name to instance to align with the `renderlayer` in maya

🎨 add aov and renderlayer to anatomy data

cd2d5f4

if `aov` and `renderlayer` is set on instance, use it in anatomy data for formatting templates

♻️ use StringTemplate for template formatting in product names

cfe8f08

allow using optional keys in formatting of product name templates

♻️ use product name templates for render instances

8fa1317

remove hardcoded instance names in favor of product name templates

antirotor added enhancement sponsored This is directly sponsored by a client or community member labels Apr 24, 2024

antirotor requested a review from iLLiCiTiT April 24, 2024 17:14

antirotor self-assigned this Apr 24, 2024

ynbot added type: enhancement Improvement of existing functionality or minor addition host: Houdini size/S labels Apr 24, 2024

Merge branch 'develop' into enhancement/render-product-names-templated

bf46634

antirotor marked this pull request as ready for review April 30, 2024 15:22

antirotor requested a review from LiborBatek April 30, 2024 15:22

antirotor assigned iLLiCiTiT and LiborBatek and unassigned antirotor Apr 30, 2024

LiborBatek mentioned this pull request May 2, 2024

Maya: Rendering multilayer EXR images always resulting into names with "_beauty" suffix #439

Closed

2 tasks

LiborBatek reviewed May 2, 2024

View reviewed changes

BigRoy reviewed May 2, 2024

View reviewed changes

MustafaJafar self-requested a review May 21, 2024 09:02

iLLiCiTiT reviewed May 22, 2024

View reviewed changes

client/ayon_core/pipeline/farm/pyblish_functions.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed May 22, 2024

View reviewed changes

client/ayon_core/pipeline/farm/pyblish_functions.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed May 22, 2024

View reviewed changes

client/ayon_core/pipeline/farm/pyblish_functions.py Outdated Show resolved Hide resolved

MustafaJafar and others added 2 commits May 22, 2024 15:05

Merge branch 'develop' into enhancement/render-product-names-templated

ad61ced

Update client/ayon_core/pipeline/farm/pyblish_functions.py

1ae71bb

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

antirotor and others added 4 commits May 22, 2024 14:31

Update client/ayon_core/pipeline/farm/pyblish_functions.py

6541dbe

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Update client/ayon_core/pipeline/farm/pyblish_functions.py

f7aada3

Co-authored-by: Jakub Trllo <43494761+iLLiCiTiT@users.noreply.github.com>

Merge remote-tracking branch 'origin/develop' into enhancement/render…

b37b0ec

…-product-names-templated

Merge branch 'enhancement/render-product-names-templated' of github.c…

eb83310

…om:ynput/ayon-core into enhancement/render-product-names-templated

antirotor and others added 3 commits May 22, 2024 16:48

♻️ remove debug print

61147d2

🎨 pass render layer to representation context

19c7c04

Merge branch 'develop' into enhancement/render-product-names-templated

0bcf3d3

MustafaJafar added 3 commits May 22, 2024 21:49

add renderlayer and aov_name to dynamic_data of get_product_name

9012e52

add renderlayer and aov keys to created runtime render instances

cba100a

add TODO about product_group value

62870f2

MustafaJafar mentioned this pull request May 22, 2024

Use product name templates for render products on local Houdini render #543

Merged

LiborBatek self-requested a review May 23, 2024 11:10

LiborBatek approved these changes May 23, 2024

View reviewed changes

BigRoy reviewed May 23, 2024

View reviewed changes

Merge pull request #543 from ynput/enhancement/render-product-names-t…

871f6a7

…emplated-local-houdini-render Use product name templates for render products on local Houdini render

MustafaJafar reviewed Jun 21, 2024

View reviewed changes

antirotor closed this Jul 2, 2024

This was referenced Aug 8, 2024

Use product name for render templates #841

Merged

Render layer name collector ynput/ayon-houdini#67

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use product name templates for render products on farm publish #447

Use product name templates for render products on farm publish #447

antirotor commented Apr 24, 2024

moonyuet commented Apr 25, 2024

antirotor commented Apr 25, 2024

LiborBatek left a comment

BigRoy May 2, 2024

antirotor May 6, 2024

BigRoy May 2, 2024

antirotor May 6, 2024

BigRoy May 6, 2024

antirotor May 7, 2024

BigRoy May 7, 2024

antirotor May 13, 2024

BigRoy May 13, 2024 •

edited

Loading

MustafaJafar commented May 22, 2024

LiborBatek commented May 22, 2024 •

edited

Loading

MustafaJafar commented May 22, 2024 •

edited

Loading

LiborBatek left a comment

BigRoy May 23, 2024

BigRoy May 23, 2024

antirotor May 23, 2024

MustafaJafar Jun 21, 2024 •

edited

Loading

m-u-r-p-h-y Jun 26, 2024

LiborBatek Jun 26, 2024

antirotor commented Jul 1, 2024

antirotor commented Jul 2, 2024

	product_type = skeleton["productType"]
	if not s_product_name.startswith(product_type):
	group_name = '{}{}{}{}{}'.format(
	product_type,
	task[0].upper(), task[1:],
	s_product_name[0].upper(), s_product_name[1:])
	else:
	group_name = s_product_name

	# if there are multiple cameras, we need to add camera name
	expected_filepath = col[0] if isinstance(col, (list, tuple)) else col
	cams = [cam for cam in cameras if cam in expected_filepath]
	if cams:
	for cam in cams:
	if not aov:
	product_name = '{}_{}'.format(group_name, cam)
	elif not aov.startswith(cam):
	product_name = '{}_{}_{}'.format(group_name, cam, aov)
	else:
	product_name = "{}_{}".format(group_name, aov)
	else:
	if aov:
	product_name = '{}_{}'.format(group_name, aov)
	else:
	product_name = '{}'.format(group_name)

		# to align with maya render layers
		instance.data["renderlayer"] = rop.name()

	# Explicitly store the full list even though template data might
	# have a different value because it uses just a single udim tile
	if repre.get("udim"):
	repre_context["udim"] = repre.get("udim") # store list

Use product name templates for render products on farm publish #447

Use product name templates for render products on farm publish #447

Conversation

antirotor commented Apr 24, 2024

Changelog Description

Additional info

Testing notes:

moonyuet commented Apr 25, 2024

antirotor commented Apr 25, 2024

LiborBatek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BigRoy May 13, 2024 • edited Loading

Choose a reason for hiding this comment

MustafaJafar commented May 22, 2024

LiborBatek commented May 22, 2024 • edited Loading

MustafaJafar commented May 22, 2024 • edited Loading

LiborBatek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MustafaJafar Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antirotor commented Jul 1, 2024

antirotor commented Jul 2, 2024

BigRoy May 13, 2024 •

edited

Loading

LiborBatek commented May 22, 2024 •

edited

Loading

MustafaJafar commented May 22, 2024 •

edited

Loading

MustafaJafar Jun 21, 2024 •

edited

Loading