Refactor Deadline Submission JobInfo #49

kalisp · 2024-10-18T14:01:28Z

Changelog Description

This PR tries to make DL submissions easier, mostly handling of JobInfo segment. Collection of values is moved out of submission phase much earlier where values could be filled from default values in Settings or overriden by selected list of field by artist in Publisher UI.

Move to universal collector should help with potential validation of filled values.
Profiles in Settings should allow more control of values (like priority, limit_groups) based on task_types or hosts etc.

Additional info

Should be used with ynput/ayon-core#958 !!

All DCC need to be tested:

Testing notes:

Should be used with ynput/ayon-core#958 !!

do regular DL publishing
test that nothing breaks
update ayon+settings://deadline/publish/CollectJobInfo with new defaults and check if they propagete to Publisher
update values in Publisher if they propagate to DL

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

It is handling EnvironmentKey* type of fields

BigRoy

I like where this is going! :)

client/ayon_deadline/lib.py

client/ayon_deadline/abstract_submit_deadline.py

BigRoy · 2024-10-18T15:44:58Z

server/settings/publish_plugins.py

+    chunk_size: int = SettingsField(999, title="Frames per Task")
+    priority: int = SettingsField(50, title="Priority")
+    group: str = SettingsField("", title="Group")
+    limit_groups: list[LimitGroupsSubmodel] = SettingsField(
+        default_factory=list,
+        title="Limit Groups",
+    )
+    concurrent_tasks: int = SettingsField(
+        1, title="Number of concurrent tasks")
+    department: str = SettingsField("", title="Department")
+    use_gpu: bool = SettingsField("", title="Use GPU")
+    job_delay: str = SettingsField(
+        "", title="Delay job",
+        placeholder="dd:hh:mm:ss"
+    )
+    use_published: bool = SettingsField(True, title="Use Published scene")
+    asset_dependencies: bool = SettingsField(True, title="Use Asset dependencies")
+    workfile_dependency: bool = SettingsField(True, title="Workfile Dependency")
+    multiprocess: bool = SettingsField(False, title="Multiprocess")


Would be very nice to get description argument set for each of these.

BigRoy · 2024-10-18T15:50:23Z

server/settings/publish_plugins.py

    enabled: bool = SettingsField(True)
    optional: bool = SettingsField(title="Optional")
    active: bool = SettingsField(title="Active")


With all the other attributes removed from these models we can now make one EnableStateModel class instead that we use for all of these making this file much smaller.

The question is if these jobs should be actually optional at all as if they would be disabled, submit publish job will fail.

The question is if these jobs should be actually optional at all as if they would be disabled, submit publish job will fail.

I guess the idea would have been that it maybe would allow to disable just the farm publishing support for an instance - but there creators definitely are not configured that way. So I agree - may make no sense to be optional, or even be disabled, currently?

BigRoy · 2024-10-18T15:52:56Z

client/ayon_deadline/plugins/publish/aftereffects/submit_aftereffects_deadline.py

-        dln_job_info.add_render_job_env_var()
+
+        # already collected explicit values for rendered Frames
+        if not dln_job_info.Frames:


I actually think this is a place where we should not be setting this in a Deadline specific way but have it accessible on the instance even outside of the job info. So Frames would be build up from instance.data["frames"] instead so that it could turn out to be non-Deadline specific. However, not critical for now and we can do that later. Might be worth a TODO in code or an issue after this PR.

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

client/ayon_deadline/plugins/publish/global/submit_publish_job.py

server/settings/publish_plugins.py

Not yet working on AOVs.

Deadline portion must be there for ValidateExpectedFiles

client/ayon_deadline/lib.py

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py

productName might not change, this condition is clear

client/ayon_deadline/plugins/publish/houdini/submit_houdini_cache_deadline.py

Co-authored-by: Jakub Trllo <[email protected]>

antirotor

Tested in Unreal and Maya

LiborBatek · 2024-11-21T08:39:07Z

@kalisp I dont see any addon settings when built the addon package....it used to work with older commits (when first time tested)

??

however renders went through the DL fine and been integrated on AYON...LGTM

LiborBatek

Been tested again in blender and maya and both works well...

note: I had experienced DL addon settings were not available to me (package built issue?) so this should be checked if an issue or not.

…support-of-Frame-List' into feature/AY-6789_Render-instance-support-of-Frame-List

Doesnt make much sense on ContextPlugin and it is not working on Pyblish

…f-Frame-List

MustafaJafar

I think these are no longer needed in fusion submitter settings.

server/settings/publish_plugins.py

LiborBatek · 2024-11-21T12:48:56Z

@kalisp now addon settings been visible...so all good on this front!

MustafaJafar

@iLLiCiTiT, Should we add a minor bump to his PR?

MustafaJafar

For information, This PR is now documented in ynput/ayon-documentation#301

Should be bumped to release when available

…support-of-Frame-List' into feature/AY-6789_Render-instance-support-of-Frame-List

kalisp added 13 commits October 16, 2024 18:50

New Settings for CollectJobInfoItem

d1a883b

These settings should be base of generic JobInfo values for Deadline submission. They should contain variables previously contained in Submit* Settings. Some of them should be exposed to Publisher UI as artist overrides.

Matched implemented artist overrides definitions

f72e87f

Added collected jobInfo object

2b29403

Use DeadlineJobInfo as collector object

cb0d24d

It was decided that dataclasses should be used instead of attrs. This moves DeadlineJobInfo which is full mapping of JobInfo from abstract submitters to collectors to limit need of new class and necessary remapping later.

Removed six dependency

37c0a0d

Implemented new generic method applicable for all DCC

3358838

Added new argument for old get_job_info (which should be probabaly renamed) to pass base of prepared object to be enhanced with DCC specific fields

Removed unneeded imports

60da8b9

Do not pass empty string to DeadlineJobInfo

ae3c739

Empty strings overrides None defaults which might cause issue (it definitely does for job_delay).

Removed unneded imports

1c8990a

Reimplemented get_job_info, moved to generic impl

57860ce

Removed storign deadline info to instance

10f566f

'deadline' dictionary wasnt used at all, it contained large DeadlineJobInfo which just enlarged metadata json unnecessary.

Updates to DeadlineJobInfo to match previous implementation

bbb12b7

It is handling EnvironmentKey* type of fields

Added Frames field for explicit rendering

c26c997

kalisp requested review from BigRoy, antirotor and iLLiCiTiT October 18, 2024 14:01

kalisp self-assigned this Oct 18, 2024

BigRoy reviewed Oct 18, 2024

View reviewed changes

kalisp added 8 commits October 18, 2024 18:15

Run CollectJobInfo only locally

e1eba7f

Implemented explicit frames filtering on simple files

8062462

Not yet working on AOVs.

Reworked removal of deadline in submit job

ec908e0

Deadline portion must be there for ValidateExpectedFiles

Removed empty line

64b22ff

Added empty line at the end

0d3125d

Formatting

c206b24

Removed forgotten dev code

803c25c

Refactor batchname

201c5b8

BigRoy added the type: enhancement Improvement of existing functionality or minor addition label Oct 21, 2024

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/lib.py Show resolved Hide resolved

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/lib.py Outdated Show resolved Hide resolved

iLLiCiTiT reviewed Oct 22, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py Outdated Show resolved Hide resolved

Fix checking for default value

13d7cf5

iLLiCiTiT reviewed Nov 19, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py Outdated Show resolved Hide resolved

kalisp added 2 commits November 19, 2024 11:59

Fix extending host specific definitions

25c1e7f

Fix farm check optionality

1445df7

iLLiCiTiT reviewed Nov 19, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/global/collect_jobinfo.py Outdated Show resolved Hide resolved

kalisp added 2 commits November 19, 2024 12:11

Fix default value

5287ab5

Made condition clearer

564d993

productName might not change, this condition is clear

iLLiCiTiT reviewed Nov 19, 2024

View reviewed changes

client/ayon_deadline/plugins/publish/houdini/submit_houdini_cache_deadline.py Outdated Show resolved Hide resolved

kalisp and others added 2 commits November 20, 2024 12:17

Formatting change

c1d31f3

Co-authored-by: Jakub Trllo <[email protected]>

Formatting change

305bf13

Co-authored-by: Jakub Trllo <[email protected]>

kalisp requested a review from LiborBatek November 20, 2024 11:19

antirotor approved these changes Nov 20, 2024

View reviewed changes

LiborBatek approved these changes Nov 21, 2024

View reviewed changes

kalisp added 2 commits November 21, 2024 10:53

Fix wrong values in Settings

ac82c13

Merge remote-tracking branch 'origin/feature/AY-6789_Render-instance-…

7bb87a2

…support-of-Frame-List' into feature/AY-6789_Render-instance-support-of-Frame-List

moonyuet mentioned this pull request Nov 21, 2024

Farm rendering changes ynput/ayon-unreal#165

Open

kalisp added 2 commits November 21, 2024 12:26

Fix filtering on family for getting DL env vars

84eafba

Doesnt make much sense on ContextPlugin and it is not working on Pyblish

Merge branch 'develop' into feature/AY-6789_Render-instance-support-o…

c49abba

…f-Frame-List

MustafaJafar reviewed Nov 21, 2024

View reviewed changes

server/settings/publish_plugins.py Outdated Show resolved Hide resolved

server/settings/publish_plugins.py Outdated Show resolved Hide resolved

MustafaJafar reviewed Nov 21, 2024

View reviewed changes

kalisp added 2 commits November 21, 2024 15:05

Bump ayon dependency

4b3b327

Should be bumped to release when available

Merge remote-tracking branch 'origin/feature/AY-6789_Render-instance-…

e4763e7

…support-of-Frame-List' into feature/AY-6789_Render-instance-support-of-Frame-List

kalisp added the bump minor label Nov 21, 2024

kalisp added 2 commits November 22, 2024 11:12

concurrent_tasks moved to generic settings

49a7fe4

Removed unused import

0defb2f

kalisp merged commit 59b8fa3 into develop Nov 22, 2024
1 check passed

MustafaJafar mentioned this pull request Nov 22, 2024

Submit along Extra Info with jobs #53

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Deadline Submission JobInfo #49

Refactor Deadline Submission JobInfo #49

kalisp commented Oct 18, 2024 •

edited

Loading

BigRoy left a comment

BigRoy Oct 18, 2024

BigRoy Oct 18, 2024

kalisp Nov 11, 2024 •

edited

Loading

BigRoy Nov 11, 2024

BigRoy Oct 18, 2024

antirotor left a comment

LiborBatek commented Nov 21, 2024 •

edited

Loading

LiborBatek left a comment

MustafaJafar left a comment •

edited

Loading

LiborBatek commented Nov 21, 2024

MustafaJafar left a comment

MustafaJafar left a comment

Refactor Deadline Submission JobInfo #49

Refactor Deadline Submission JobInfo #49

Conversation

kalisp commented Oct 18, 2024 • edited Loading

Changelog Description

Additional info

Testing notes:

BigRoy left a comment

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

kalisp Nov 11, 2024 • edited Loading

Choose a reason for hiding this comment

BigRoy Nov 11, 2024

Choose a reason for hiding this comment

BigRoy Oct 18, 2024

Choose a reason for hiding this comment

antirotor left a comment

Choose a reason for hiding this comment

LiborBatek commented Nov 21, 2024 • edited Loading

LiborBatek left a comment

Choose a reason for hiding this comment

MustafaJafar left a comment • edited Loading

Choose a reason for hiding this comment

LiborBatek commented Nov 21, 2024

MustafaJafar left a comment

Choose a reason for hiding this comment

MustafaJafar left a comment

Choose a reason for hiding this comment

kalisp commented Oct 18, 2024 •

edited

Loading

kalisp Nov 11, 2024 •

edited

Loading

LiborBatek commented Nov 21, 2024 •

edited

Loading

MustafaJafar left a comment •

edited

Loading