Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. #5506

mzient · 2024-06-06T15:09:08Z

Category:

New feature (non-breaking change which adds functionality)
Refactoring (Redesign of existing code that doesn't affect functionality)

Description:

Prior to this change constructing exactly the same pipeline (e.g. by calling a function decorated with @pipeline_def) multiple times produced pipelines with different operator instance names and differently named operator instances and DataNodes. This PR changes that so that pipelines with the same structure have the same node names.
This is achieved by:

Using a separate operator counter in each pipeline. When all nodes all instantiated within pipeline scope, no further action is required. This happens for the vast majority of cases.
If a name collision occurs (only possible when "stealing" a subgraph from another pipeline), an error is raised.

Pipelines that are defined without a "current" pipeline have distinct operator instance names.

Additionally, there were some problems with operator discovery. I rewrote it to a much simpler DFS.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

pipeline_test.py: test_dangling_subgraph

Checklist

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: N/A

JIRA TASK: N/A

mzient · 2024-06-06T15:12:13Z

dali/python/nvidia/dali/pipeline.py

-    def get_op_outputs_num():
-        # BSF traverse the graph first to learn, for each reachable operator in the graph,
-        # how many data-nodes/edges the operator contributes to
-        # (i.e. the number of outputs of the operator instance)
-        op_outputs_num = {}
-        edges = deque(output_nodes)
-        while edges:
-            current_edge = edges.popleft()
-            source_op = get_source_op(current_edge)
-            if source_op.id in op_outputs_num:
-                op_outputs_num[source_op.id] += 1
-            else:
-                op_outputs_num[source_op.id] = 1
-                source_op.check_args()
-                edges.extend(get_op_input_edges(source_op))
-        return op_outputs_num
-
+    visited = set()
    ops = []
-    edges = deque(output_nodes)
-    op_total_outputs_num = get_op_outputs_num()
-    op_visited_outputs_num = {op_id: 0 for op_id in op_total_outputs_num}
-    while edges:
-        current_edge = edges.popleft()
-        source_op = get_source_op(current_edge)
-        op_visited_outputs_num[source_op.id] += 1
-        # Actually visit the operator only when all the nodes it contributes to
-        # were already processed
-        if op_visited_outputs_num[source_op.id] == op_total_outputs_num[source_op.id]:
-            ops.append(source_op)
-            edges.extend(get_op_input_edges(source_op))
-    ops.reverse()


This was unnecessarily complex. DFS is much better suited for discovering topology.

mzient · 2024-06-06T15:13:29Z

dali/python/nvidia/dali/ops/__init__.py

@@ -57,7 +57,8 @@
 class _OpCounter(object):
    # pylint: disable=too-few-public-methods
    _lock = threading.Lock()
-    _op_count = count(0)
+    # start from something large to avoid confusion with (more common) per-pipeline numbering


Note for reviewers: "avoid confusion" means only confusion while debugging. The renaming can deal with duplicate names.

dali-automaton · 2024-06-06T15:15:21Z

CI MESSAGE: [15624008]: BUILD STARTED

szkarpinski · 2024-06-06T15:21:50Z

dali/python/nvidia/dali/ops/__init__.py

@@ -57,7 +57,8 @@
 class _OpCounter(object):
    # pylint: disable=too-few-public-methods
    _lock = threading.Lock()
-    _op_count = count(0)
+    # start from something large to avoid confusion with (more common) per-pipeline numbering


Suggested change

# start from something large to avoid confusion with (more common) per-pipeline numbering

# start from something large to avoid confusion while debugging with (more common) per-pipeline numbering

dali/python/nvidia/dali/pipeline.py

dali/test/python/test_pipeline.py

dali-automaton · 2024-06-06T15:32:05Z

CI MESSAGE: [15624354]: BUILD STARTED

dali-automaton · 2024-06-06T15:58:25Z

CI MESSAGE: [15624997]: BUILD STARTED

dali-automaton · 2024-06-06T18:00:30Z

CI MESSAGE: [15624997]: BUILD FAILED

dali-automaton · 2024-06-06T21:57:17Z

CI MESSAGE: [15633718]: BUILD STARTED

dali-automaton · 2024-06-06T23:35:30Z

CI MESSAGE: [15633718]: BUILD FAILED

dali-automaton · 2024-06-07T16:47:28Z

CI MESSAGE: [15654084]: BUILD STARTED

dali/test/python/test_pipeline.py

dali-automaton · 2024-06-07T16:55:23Z

CI MESSAGE: [15654084]: BUILD FAILED

dali-automaton · 2024-06-07T17:06:12Z

CI MESSAGE: [15654586]: BUILD STARTED

dali-automaton · 2024-06-10T12:29:12Z

CI MESSAGE: [15709123]: BUILD FAILED

…es from other pipelines (including None). Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

- handle renaming of duplicate nodes - handle renaming of pipeline inputs that are not direct outputs of any operator - improve error message Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

…ends. Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

dali-automaton · 2024-06-11T08:51:44Z

CI MESSAGE: [15734267]: BUILD STARTED

dali/test/python/auto_aug/test_auto_augment.py

dali/test/python/test_pipeline.py

dali-automaton · 2024-06-11T12:28:20Z

CI MESSAGE: [15734267]: BUILD PASSED

mzient commented Jun 6, 2024

View reviewed changes

szkarpinski self-assigned this Jun 6, 2024

szkarpinski reviewed Jun 6, 2024

View reviewed changes

mzient marked this pull request as draft June 7, 2024 06:53

github-advanced-security bot found potential problems Jun 7, 2024

View reviewed changes

dali/test/python/test_pipeline.py Fixed Show fixed Hide fixed

dali/test/python/test_pipeline.py Fixed Show fixed Hide fixed

NVIDIA deleted a comment from dali-automaton Jun 9, 2024

mzient marked this pull request as ready for review June 9, 2024 21:42

mzient changed the title ~~Keep separate per-pipeline operator counters. Rename operator instances from other pipelines (including None).~~ Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines. Jun 9, 2024

mzient changed the title ~~Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines.~~ Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. Jun 9, 2024

dali-automaton assigned banasraf Jun 10, 2024

mzient force-pushed the per_pipeline_name_generation branch from 6ac2b99 to a291562 Compare June 10, 2024 12:28

mzient added 8 commits June 10, 2024 14:29

Keep separate per-pipeline operator counters. Rename operator instanc…

e069330

…es from other pipelines (including None). Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Comments + formatting.

1431ea1

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Use weakref for pipeline reference in operator instance.

70e06a9

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Review: improved comments, extended test.

d627d7d

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Bugfixes:

aa820f9

- handle renaming of duplicate nodes - handle renaming of pipeline inputs that are not direct outputs of any operator - improve error message Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

[WIP]

808b819

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Working!

f753acb

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Fix unused import. Adjust test.

cf103f1

Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

Add missing schema check. Adjust test criteria in autoaugment and fri…

999c1cc

…ends. Signed-off-by: Michal Zientkiewicz <michalz@nvidia.com>

mzient force-pushed the per_pipeline_name_generation branch from a291562 to 999c1cc Compare June 10, 2024 12:30

szkarpinski reviewed Jun 11, 2024

View reviewed changes

dali/test/python/auto_aug/test_auto_augment.py Show resolved Hide resolved

szkarpinski reviewed Jun 11, 2024

View reviewed changes

dali/test/python/test_pipeline.py Show resolved Hide resolved

szkarpinski approved these changes Jun 11, 2024

View reviewed changes

banasraf approved these changes Jun 11, 2024

View reviewed changes

mzient merged commit 9dfdeac into NVIDIA:main Jun 11, 2024
6 checks passed

mzient mentioned this pull request Jun 11, 2024

Add failure tests for stealing subgraphs. Minor fix in pipeline. #5518

Merged

18 tasks

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. #5506

Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. #5506

mzient commented Jun 6, 2024 •

edited

Loading

mzient Jun 6, 2024

mzient Jun 6, 2024

dali-automaton commented Jun 6, 2024

szkarpinski Jun 6, 2024

mzient Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 10, 2024

dali-automaton commented Jun 11, 2024

dali-automaton commented Jun 11, 2024

	# start from something large to avoid confusion with (more common) per-pipeline numbering
	# start from something large to avoid confusion while debugging with (more common) per-pipeline numbering

Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. #5506

Keep separate per-pipeline operator counters. Error out when "stealing" subgraphs from other pipelines results in duplicate names. #5506

Conversation

mzient commented Jun 6, 2024 • edited Loading

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

Checklist

Documentation

DALI team only

Requirements

mzient Jun 6, 2024

Choose a reason for hiding this comment

mzient Jun 6, 2024

Choose a reason for hiding this comment

dali-automaton commented Jun 6, 2024

szkarpinski Jun 6, 2024

Choose a reason for hiding this comment

mzient Jun 6, 2024

Choose a reason for hiding this comment

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 6, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 7, 2024

dali-automaton commented Jun 10, 2024

dali-automaton commented Jun 11, 2024

dali-automaton commented Jun 11, 2024

mzient commented Jun 6, 2024 •

edited

Loading