Update Polygraphy to v0.35.1

Signed-off-by: Rajeev Rao <[email protected]>

tools/Polygraphy/.gitignore

@@ -67,3 +67,6 @@ venv/
 ENV/
 env.bak/
 venv.bak/
+
+# Crash dumps
+core

tools/Polygraphy/CHANGELOG.md

@@ -3,6 +3,62 @@
 Dates are in YYYY-MM-DD format.
 
 
+## v0.35.1 (2022-01-14)
+### Added
+- Added a `providers` parameter to `SessionFromOnnx` to specify execution providers for ONNX-Runtime and a
+    corresponding `--providers` argument to CLI tools.
+
+### Changed
+- `CompareFunc.simple()` will now correctly display the minimum required tolerances when using `elemwise` mode.
+    Note that in elementwise comparison mode, each element of the output is compared against both tolerances, and
+    only counted as a mismatch if both are exceeded. Hence, the minimum required tolerances apply if only one type
+    of tolerance is being used. When both absolute/relative tolerance are set, the requirements may be lower.
+
+
+## v0.35.0 (2022-01-06)
+### Added
+- Added a `memory_pool_limits` parameter to `CreateConfig`.
+- Added a `--pool-limit`/`--memory-pool-limit` argument to command-line tools.
+
+### Changed
+- Changed the default base calibrator class to `IInt8EntropyCalibrator2` since it works across both GPU and DLA.
+    To preserve the old behavior, specify `--calibration-base-class=IInt8MinMaxCalibrator` on the command-line
+    or specify the `BaseClass` argument in `Calibrator` in the Python API.
+- Deprecated `--workspace` command-line option and `max_workspace_size` parameter in `CreateConfig`.
+    Use `--pool-limit` and `memory_pool_limits` respectively instead.
+
+### Removed
+- Removed deprecated module `polygraphy.util.serde`. Use `polygraphy.json` instead.
+- Removed `--tactic-replay` command-line option. Use `--load-tactics`/`--save-tactics` instead.
+
+
+## v0.34.2 (2021-11-29)
+### Added
+- Added support for `MeanVarianceNormalization` to `PluginRefRunner`.
+
+
+## v0.34.1 (2021-11-24)
+### Added
+- Added a `profiling_verbosity` parameter to `CreateConfig()`.
+- Added support for displaying layer-level engine information in `inspect model` for newer versions of TensorRT.
+
+
+## v0.34.0 (2021-11-22)
+### Added
+- Added a new `add()` API to `RunResults` to make it easier to create custom output data.
+    Added [a new example](./examples/cli/run/06_comparing_with_custom_output_data/) to demonstrate how to use this API.
+
+### Changed
+- Deprecated `--mode` option in `inspect model`; a new `--show` option has been introduced
+    which can be used to individually control what is displayed.
+- Command-line tools will now use `IInt8EntropyCalibrator2` for calbration if DLA and int8 mode are enabled
+    since the default does not work with DLA.
+
+### Removed
+- Removed several deprecated submodules of `polygraphy.common`: `constants`, `cuda`, `exception`, `func`.
+    These can now be found under the top-level `polygraphy` module instead.
+
+
 ## v0.33.2 (2021-10-21)
 ### Changed
 - Improved the help messages of various subtools, including `run`, `debug build`, and `debug reduce`.

tools/Polygraphy/Makefile

@@ -19,6 +19,7 @@ leak_check:
 	export PYTHONPATH=$(CURDIR):$${PYTHONPATH} && \
 	    export PATH=$(CURDIR)/bin:$${PATH} && \
 	    export POLYGRAPHY_INTERNAL_CORRECTNESS_CHECKS=1 && \
+		export PYTHONMALLOC=malloc && \
 		valgrind --leak-check=full python3 -m pytest tests/ -v --durations=5 2>&1 | tee leak-check.log
 
 clean:

tools/Polygraphy/README.md

@@ -117,7 +117,7 @@ To enable this behavior, set the `POLYGRAPHY_AUTOINSTALL_DEPS` environment varia
 
 **NOTE**: *By default, dependencies will be installed using the current interpreter, and may overwrite existing*
     *packages. The default installation command, which is `python -m pip install`, can be overriden by setting*
-    *the `POLYGRAPHY_INSTALL_CMD` environment variable, or setting `polygraphy.config.INSTALL_CMD = "..."` using the Python API.*
+    *the `POLYGRAPHY_INSTALL_CMD` environment variable, or setting `polygraphy.config.INSTALL_CMD` using the Python API.*
 
 #### Installing Manually
 

tools/Polygraphy/examples/api/02_validating_on_a_dataset/README.md

@@ -10,7 +10,7 @@ with a real dataset that includes labels or golden values - especially if the da
 In such cases, it is recommended to use a runner directly instead.
 
 *NOTE: It is possible to provide custom input data to `Comparator.run()` using the `data_loader`*
-    *parameter. This is may be a viable option when using a smaller dataset.*
+    *parameter. This may be a viable option when using a smaller dataset.*
 
 In this example, we use a `TrtRunner` directly to validate an identity model on
 a trivial dataset. Unlike using the `Comparator`, using a runner gives you complete

tools/Polygraphy/examples/api/03_interoperating_with_tensorrt/README.md

@@ -31,7 +31,7 @@ we will focus on cases where you may want to:
     generated TensorRT network, which should be named `"MyIdentity"`:
 
     ```bash
-    polygraphy inspect model example.py --trt-network-func load_network --mode=full
+    polygraphy inspect model example.py --trt-network-func load_network --show layers attrs weights
     ```
 
 3. Run the example:

tools/Polygraphy/examples/api/05_using_tensorrt_network_api/README.md

@@ -21,7 +21,7 @@ loader to seamlessly integrate a network defined using TensorRT APIs with Polygr
     This will invoke `create_network()` from within the script and display the generated TensorRT network:
 
     ```bash
-    polygraphy inspect model example.py --trt-network-func create_network --mode=full
+    polygraphy inspect model example.py --trt-network-func create_network --show layers attrs weights
     ```
 
 3. Run the example:

tools/Polygraphy/examples/cli/debug/02_reducing_failing_onnx_models/README.md

@@ -114,13 +114,13 @@ Hence, the final reduced model should contain just the `Mul` node (since the oth
     ```bash
     polygraphy debug reduce folded.onnx -o initial_reduced.onnx --mode=bisect \
         --fail-regex "Op: Mul" \
-        --check polygraphy inspect model polygraphy_debug.onnx --mode=basic
+        --check polygraphy inspect model polygraphy_debug.onnx --show layers
     ```
 
 4. **[Optional]** As a sanity check, we can inspect our reduced model to ensure that it does contain the `Mul` node:
 
     ```bash
-    polygraphy inspect model initial_reduced.onnx --mode=basic
+    polygraphy inspect model initial_reduced.onnx --show layers
     ```
 
 5. Since we used `bisect` mode in the previous step, the model may not be as minimal as it could be.
@@ -137,12 +137,12 @@ Hence, the final reduced model should contain just the `Mul` node (since the oth
     ```bash
     polygraphy debug reduce initial_reduced.onnx -o final_reduced.onnx --mode=linear \
         --fail-regex "Op: Mul" \
-        --check polygraphy inspect model polygraphy_debug.onnx --mode=basic
+        --check polygraphy inspect model polygraphy_debug.onnx --show layers
     ```
 
 6. **[Optional]** At this stage, `final_reduced.onnx` should contain just the failing node - the `Mul`.
     We can verify this with `inspect model`:
 
     ```bash
-    polygraphy inspect model final_reduced.onnx --mode=basic
+    polygraphy inspect model final_reduced.onnx --show layers
     ```

tools/Polygraphy/examples/cli/inspect/01_inspecting_a_tensorrt_network/README.md

@@ -13,7 +13,7 @@ into TensorRT networks, and then display them.
 
     ```bash
     polygraphy inspect model identity.onnx \
-        --mode=basic --display-as=trt
+        --show layers --display-as=trt
     ```
 
     This will display something like:
@@ -34,4 +34,4 @@ into TensorRT networks, and then display them.
              -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
     ```
 
-    It is also possible to show detailed layer information, including layer attributes, using `--mode=full`.
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.

tools/Polygraphy/examples/cli/inspect/02_inspecting_a_tensorrt_engine/README.md

@@ -21,14 +21,18 @@ about TensorRT engines, i.e. plan files:
 2. Inspect the engine:
 
     ```bash
-    polygraphy inspect model dynamic_identity.engine
+    polygraphy inspect model dynamic_identity.engine \
+        --show layers
     ```
 
+    NOTE: `--show layers` only works if the engine was built with a `profiling_verbosity` other than `NONE`.
+        Higher verbosities make more per-layer information available.
+
     This will display something like:
 
     ```
     [I] ==== TensorRT Engine ====
-        Name: Unnamed Network 0 | Explicit Batch Engine (2 layers)
+        Name: Unnamed Network 0 | Explicit Batch Engine
 
         ---- 1 Engine Input(s) ----
         {X [dtype=float32, shape=(1, 2, -1, -1)]}
@@ -47,4 +51,17 @@ about TensorRT engines, i.e. plan files:
         - Profile: 1
             Binding Index: 2 (Input)  [Name: X [profile 1]] | Shapes: min=(1, 2, 2, 2), opt=(1, 2, 4, 4), max=(1, 2, 6, 6)
             Binding Index: 3 (Output) [Name: Y [profile 1]] | Shape: (1, 2, -1, -1)
+
+        ---- 1 Layer(s) Per Profile ----
+        - Profile: 0
+            Layer 0    | node_of_Y [Op: Reformat]
+                {X [shape=(1, 2, -1, -1)]}
+                 -> {Y [shape=(1, 2, -1, -1)]}
+
+        - Profile: 1
+            Layer 0    | node_of_Y [profile 1] [Op: Reformat]
+                {X [profile 1] [shape=(1, 2, -1, -1)]}
+                 -> {Y [profile 1] [shape=(1, 2, -1, -1)]}
     ```
+
+    It is also possible to show more detailed layer information using `--show layers attrs`.

tools/Polygraphy/examples/cli/inspect/03_inspecting_an_onnx_model/README.md

@@ -11,7 +11,7 @@ The `inspect model` subtool can display ONNX models.
 1. Inspect the ONNX model:
 
     ```bash
-    polygraphy inspect model identity.onnx --mode=basic
+    polygraphy inspect model identity.onnx --show layers
     ```
 
     This will display something like:
@@ -35,4 +35,4 @@ The `inspect model` subtool can display ONNX models.
              -> {y [dtype=float32, shape=(1, 1, 2, 2)]}
     ```
 
-    It is also possible to show detailed layer information, including layer attributes, using `--mode=full`.
+    It is also possible to show detailed layer information, including layer attributes, using `--show layers attrs weights`.

tools/Polygraphy/examples/cli/run/01_comparing_frameworks/README.md

@@ -52,4 +52,4 @@ For more details on working with dynamic shapes in TensorRT, refer to
 [`convert` example 03](../../convert/03_dynamic_shapes_in_tensorrt/) or
 [API example 07](../../../api/07_tensorrt_and_dynamic_shapes/).
 
-For details on how to supply real input data, see [`run` example 05](../05_comparing_with_custom_data/).
+For details on how to supply real input data, see [`run` example 05](../05_comparing_with_custom_input_data/).

tools/Polygraphy/examples/cli/run/05_comparing_with_custom_input_data/README.md

@@ -0,0 +1,56 @@
+# Comparing With Custom Input Data
+
+## Introduction
+
+In some cases, we may want to run comparisons using custom input data.
+Polygraphy provides multiple ways to do so, which are detailed [here](../../../../polygraphy/tools/README.md#using-custom-input-data).
+
+In this example, we'll demonstrate 2 different approaches:
+
+1. Using a data loader script by defining a `load_data()` function in a Python script (`data_loader.py`).
+    Polygraphy will use `load_data()` to generate inputs at runtime.
+
+2. Using a JSON file containing pre-generated inputs.
+    For convenience, we'll use our script from above (`data_loader.py`) to save the inputs
+    generated by `load_data()` to a file called `custom_inputs.json`.
+
+*TIP: Generally, a data loader script is preferrable when working with large amounts of input data*
+    *as it avoids the need to write to the disk.*
+    *On the other hand, JSON files may be more portable and can help ensure reproducibility.*
+
+Finally, we'll supply our custom input data to `polygraphy run` and compare outputs between
+ONNX-Runtime and TensorRT.
+
+Since our model has dynamic shapes, we'll need to set up a TensorRT Optimization Profile.
+For details on how we can do this via the command-line,
+see [`convert` example 03](../../convert/03_dynamic_shapes_in_tensorrt).
+For simplicitly, we'll create a profile where `min` == `opt` == `max`.
+
+*NOTE: It is important that our optimization profile works with the shapes provided by our*
+    *custom data loader. In our very simple case, the data loader always generates inputs of*
+    *shape (1, 2, 28, 28), so we just need to ensure this falls within [`min`, `max`].*
+
+## Running The Example
+
+1. Run the script to save input data to the disk.
+    *NOTE: This is only necessary for option 2.*
+    ```bash
+    python3 data_loader.py
+    ```
+
+2. Run the model with TensorRT and ONNX-Runtime using custom input data:
+    - Option 1: Using the data loader script:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --data-loader-script data_loader.py
+    ```
+
+    - Option 2: Using the JSON file containing the saved inputs:
+
+    ```bash
+    polygraphy run dynamic_identity.onnx --trt --onnxrt \
+        --trt-min-shapes X:[1,2,28,28] --trt-opt-shapes X:[1,2,28,28] --trt-max-shapes X:[1,2,28,28] \
+        --load-inputs custom_inputs.json
+    ```

tools/Polygraphy/examples/cli/run/05_comparing_with_custom_input_data/data_loader.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Demonstrates two methods of loading custom input data in Polygraphy:
+
+Option 1: Defines a `load_data` function that returns a generator yielding
+    feed_dicts so that this script can be used as the argument for
+    the --data-loader-script command-line parameter.
+
+Option 2: Writes input data to a JSON file that can be used as the argument for
+    the --load-inputs command-line parameter.
+"""
+import numpy as np
+from polygraphy.json import save_json
+
+INPUT_SHAPE = (1, 2, 28, 28)
+
+# Option 1: Define a function that will yield feed_dicts (i.e. Dict[str, np.ndarray])
+def load_data():
+    for _ in range(5):
+        yield {"x": np.ones(shape=INPUT_SHAPE, dtype=np.float32)}  # Still totally real data
+
+
+# Option 2: Create a JSON file containing the input data using the `save_json()` helper.
+#   The input to `save_json()` should have type: List[Dict[str, np.ndarray]].
+#   For convenience, we'll reuse our `load_data()` implementation to generate the list.
+input_data = list(load_data())
+save_json(input_data, "custom_inputs.json", description="custom input data")