Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: incorrect stats when auto-compaction is hooked #14042

Merged

Conversation

dantengsky
Copy link
Member

@dantengsky dantengsky commented Dec 15, 2023

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Reset the ctx's write_progress value, after compaction hook executed, so that correct number of rows written will be shown.

With this PR, in the test scenario of #13947, the correct stats 2 rows written will be shown

root@localhost:8000/default> create stage test_stage;

create table tmp(id int);

 insert into tmp values(1);

insert into tmp values(2);

copy into @test_stage from (select * from tmp);

copy into tmp from (select * from @test_stage);


CREATE stage test_stage

0 row written in 0.051 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)


CREATE TABLE tmp(id int)

0 row written in 0.173 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)


INSERT INTO
  tmp
VALUES
(1)

1 row written in 0.135 sec. Processed 1 row, 5 B (7.39 row/s, 36 B/s)


INSERT INTO
  tmp
VALUES
(2)

1 row written in 0.301 sec. Processed 1 row, 5 B (3.32 row/s, 16 B/s)


copy INTO @test_stage
FROM
  (
    SELECT
      *
    FROM
      tmp
  )

0 row written in 0.088 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)


copy INTO tmp
FROM
  (
    SELECT
      *
    FROM
      @test_stage
  )

┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                               File                              │ Rows_loaded │ Errors_seen │    First_error   │ First_error_line │
│                              String                             │    Int32    │    Int32    │ Nullable(String) │  Nullable(Int32) │
├─────────────────────────────────────────────────────────────────┼─────────────┼─────────────┼──────────────────┼──────────────────┤
│ data_b35ca9f3-7589-49b6-b204-75f764eee778_0000_00000000.parquet │           2 │           0 │ NULL             │             NULL │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2 rows written in 0.909 sec. Processed 2 rows, 9 B (2.2 rows/s, 9 B/s)

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

This change is Reviewable

@dantengsky dantengsky changed the title Fix: incorrect stats when compact hook fix: incorrect stats when compact hook Dec 15, 2023
@github-actions github-actions bot added the pr-bugfix this PR patches a bug in codebase label Dec 15, 2023
@dantengsky dantengsky force-pushed the fix-incorrect-stats-of-compact-hook branch from c3ca85f to 9541020 Compare December 15, 2023 15:08
@dantengsky dantengsky changed the title fix: incorrect stats when compact hook fix: incorrect stats when auto-compaction is hooked Dec 15, 2023
@dantengsky dantengsky marked this pull request as ready for review December 15, 2023 15:36
@dantengsky dantengsky requested a review from BohuTANG December 15, 2023 15:36
@dantengsky dantengsky requested a review from sundy-li December 15, 2023 15:37
@BohuTANG BohuTANG merged commit ef2ad54 into databendlabs:main Dec 16, 2023
70 checks passed
@BohuTANG
Copy link
Member

Does the refresh_hook have the same issue? @b41sh, need your help to check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-bugfix this PR patches a bug in codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: incorrect number of effected rows
2 participants