Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Refactor string input check #101

Merged
merged 6 commits into from
Jul 31, 2024

Conversation

yinggeh
Copy link
Contributor

@yinggeh yinggeh commented Jul 18, 2024

What does the PR do?

Move each individual check of string input in model backend repos to the bakend repo.

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • refactor

Related PRs:

backend: #101
tensorflow_backend: triton-inference-server/tensorflow_backend#104
pytorch_backend: triton-inference-server/pytorch_backend#136
onnxruntime_backend: triton-inference-server/onnxruntime_backend#263
python_backend: triton-inference-server/python_backend#370

Test plan:

  • CI Pipeline ID:
    17021222 and 17057045

Background

Initially just to add missing string input tensor checks to python_backend. Refactored because duplicate code from other model backends were found.

@yinggeh yinggeh added the enhancement New feature or request label Jul 18, 2024
@yinggeh yinggeh self-assigned this Jul 18, 2024
include/triton/backend/backend_common.h Outdated Show resolved Hide resolved
*const_cast<char*>(buffer) = 0;
}

if (*element_idx != expected_element_cnt) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here slightly different condition checks from original tensorflow.cc and libtorch.cc. Removing (*response != nullptr).

I don't think we want to skip sending this error or skip FillStringTensor if RESPOND_AND_SET_NULL_IF_ERROR is called before SetStringInputTensor in tensorflow.cc and libtorch.cc. I believe it was a mistake because other string format checks before this line do not require if (*response != nullptr) to send error or FillStringTensor. Open to discussions.

// Code from tensorflow.cc and libtorch.cc
if ((*response != nullptr) && (element_idx != request_element_cnt)) {
  RESPOND_AND_SET_NULL_IF_ERROR(
      response, TRITONSERVER_ErrorNew(
                    TRITONSERVER_ERROR_INTERNAL,
                    std::string(
                        "expected " + std::to_string(request_element_cnt) +
                        " strings for inference input '" + name + "', got " +
                        std::to_string(element_idx))
                        .c_str()));
  FillStringTensor(
      tensor, tensor_offset + element_idx, request_element_cnt - element_idx);
}

tanmayv25
tanmayv25 previously approved these changes Jul 26, 2024
Tabrizian
Tabrizian previously approved these changes Jul 26, 2024
@yinggeh yinggeh dismissed stale reviews from Tabrizian and tanmayv25 via 5d6905d July 28, 2024 21:16
@yinggeh yinggeh changed the title feature: Refactor string input check refactor: Refactor string input check Jul 30, 2024
@yinggeh yinggeh merged commit 30fa78a into main Jul 31, 2024
1 check passed
@yinggeh yinggeh deleted the yinggeh-DLIS-6832-validate-gpu-string-tensors branch September 18, 2024 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging this pull request may close these issues.

3 participants