Add native_op_trace_directory_create_config session property and setDirectoryCB in native #24156
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
yuandagits
requested review from
steveburnett,
elharo and
a team
as code owners
xiaoxmeng
previously approved these changes
Nov 27, 2024
| execTask.queryCtx()->queryId(), | ||
| execTask.taskId(), | ||
| includeNodeInSpillPath); | ||
| const auto [taskSpillDirPath, taskDateStrPath] = |
There was a problem hiding this comment.
s/taskDateStrPath/dateSpillDirPath/?
| @@ -402,8 +424,12 @@ std::unique_ptr<TaskInfo> TaskManager::createOrUpdateErrorTask( | |||
| if (includeNodeInSpillPath) { | |||
| folly::toAppend(fmt::format("{}_{}/", nodeIp, nodeId), &path); | |||
| } | |||
|
|
|||
| std::string dateStringPath = path; | |||
There was a problem hiding this comment.
s/dateStringPath/dateSpillDirPath/
| folly::toAppend(fmt::format("{}/{}/{}/", dateString, queryId, taskId), &path); | ||
| return path; | ||
| return std::make_tuple(std::move(path), std::move(dateStringPath)); |
There was a problem hiding this comment.
s/path/taskSpillDirPath/
| @@ -154,8 +154,9 @@ class TaskManager { | |||
| std::vector<velox::exec::Task::OpCallInfo>& stuckOpCalls) const; | |||
|
|
|||
| /// Build directory path for spilling for the given task. | |||
| /// Always returns non-empty string. | |||
| static std::string buildTaskSpillDirectoryPath( | |||
| /// Always returns tuple of non-empty string containing the spill directory | |||
There was a problem hiding this comment.
the task spill directory and date spill directory which is parent directory of task spill directory.
Not sure if we want to return the two as the last two output parameters of buildTaskSpillDirectoryPath?
std::string& taskSpillDirPath,
std::string& dateSpillDirPath);
12d8cca
to
d5fe7d7
Compare
yuandagits
force-pushed
the
directoryTtlSupport
branch
2 times, most recently
from
dcc1631
to
bd10f61
Compare
|
Pushed a wrong commit by mistake after rebase. After fixing, re-pushing https://github.com/yuandagits/presto/tree/directoryTtlSupport is now stuck on processing update Might abandon and recreate PR if the processing does not complete Edit: it finally finished processing... |
In native engine, add ability to set a create directory callback and add the corresponding config as a property. We also expose native_op_trace_directory_create_config as a session property. [Description] In native engine, add ability to set a create directory callback and add the corresponding config as a property. We also expose native_op_trace_directory_create_config as a session property.
yuandagits
force-pushed
the
directoryTtlSupport
branch
from
a2fb2c8
to
7cf709d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In native engine, add ability to set a create directory callback and add the corresponding config as a property. We also expose native_op_trace_directory_create_config as a session property.
Motivation and Context
See facebookincubator/velox#11631 for operator directory option support and API to set create directory facebookincubator/velox#11572
Impact
Test Plan
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.
If release note is NOT required, use: