Farm publishing: check if published items do exist

[antirotor]

Problem

When job is submitted to farm and its frame range is changed there (and not in submitting host), we try to integrate non-existent files and that will fail.

Unfortunately it will not fail very gracefully and because of the nature of farm jobs, it will be attempted multple times, producing possibly hundreds of folders by ntegrate_new before job will be terminated and marked as failure.

Solution

Solution is two-fold:

1. Provide validator that will check actual files against expected files
2. Make integrate_new more robust by checking existence of its inputs before any changes in database or filesystem happens.

[mkolar]

The key question here is what is intended by the user. There are 2 worlds clashing here 1. supervisors that want validations and 2. artist that want to hack and slash through the shots.

This is particularly happening when artist sends a shot to deadline and then tweaks the frame range there to only run a bunch of frames. Unforunatelly if validator fails on the farm it pretty much resulting in the same thing as if it just crashes. The artist simply sees that nothing has been published even though the render went through correctly.

I'm really not sure how to tackle this considering that pype inherently wants all the data consistent and a rogue shot with less frames is a bit of a no-no. Realistically though. production often calls for these kinds of hacks.

Maybe let's figure out a way to allow for partial publish, but only if the artist explicitly requests it. The question is how to do that gracefully.

[antirotor]

Maybe let's figure out a way to allow for partial publish, but only if the artist explicitly requests it. The question is how to do that gracefully.

What we can do is create Deadline post job script (not as standalone job, rather as this). This script could check rendered frames, requested frames (those mentioned in metadata json) and frame range set in job.

If rendered frames and set frame range matches, it should modify metadata json and allow publish.

All other cases (except when everything equals) should be considered as failure.

[mkolar]

That's not bad. explicit enough, but permissive at the same time.

[create-issue-branch]

Branch bugfix/170-farm-publishing-check-if-published-items-do-exist created!