Constants computed on-the-fly when model is serialized to IR #27705
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ization to IR. Helps to not materialize big computed constants in memory for the entire model. Instead make such Constants one by one during serialization to IR. It consists of two parts: - Serialize pass modification to listen for 'postponed_constant' runtime attribute in a node and on-the-fly replace such a node by a Constant node with constant_fold method. - Python helper `make_postponed_constant` that wrap an arbitrary callback that creates a tensor into a custom Python operation that has runtime attribute `postponed_constant`. This operation will constant folded into a Constant instance with data obtained by calling that callback.
github-actions
bot
added
category: Core
slyalin
commented
Nov 22, 2024
Comment on lines
+139
to
+143
| // TODO: As it was mentioned above that there was a chance to have a collision easily, should we limit the number of iterations in the following loop? | ||
| // FIXME: Loop over items that have matching hash keys only, not entire range [found, m_hash_to_file_positions.end()). | ||
| // FIXME: According to https://en.cppreference.com/w/cpp/container/multimap/find, std::multimap::find returns any item with the matching key, not necessary | ||
| // the first one. But the loop below checking items starting with `found` item only even if there may be other items with matching hash before that position. | ||
| // So we are not checking all possible matches. |
There was a problem hiding this comment.
@pavel-esir, please confirm that we have an issue here.
slyalin
commented
Nov 22, 2024
| return PostponedConstant.class_type_info | ||
|
|
||
| def evaluate(self, outputs, _): | ||
| maker().copy_to(outputs[0]) |
There was a problem hiding this comment.
If we allow the assignment of new ov.Tensor instances in outputs, then this copying won't be required. Another (probably better) option is to pass outputs[0] to maker function to build the tensor in place.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A way to avoid the materialization of big constants by computing them on-the-fly during IR serialization. Going to be used in NNCF when compressing big models.