Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix Sort Merge Join to pass TPCH tests #10100

Closed
Tracked by #9846
comphead opened this issue Apr 16, 2024 · 5 comments · Fixed by #10747
Closed
Tracked by #9846

fix Sort Merge Join to pass TPCH tests #10100

comphead opened this issue Apr 16, 2024 · 5 comments · Fixed by #10747
Assignees

Comments

@comphead
Copy link
Contributor

comphead commented Apr 16, 2024

I was trying to mark the Sort Merge Join as stable and run TPCH tests with SMJ enforced. Got the issue below, we need to fix before returning to discuss SMJ stable status

To run benches
RESULTS_NAME=smj ./benchmarks/bench.sh run tpch_smj

Depends on #10092

thread 'tokio-runtime-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join.rs:1357:22:
index out of bounds: the len is 0 but the index is 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join.rs:1357:22:
index out of bounds: the len is 0 but the index is 1
thread 'tokio-runtime-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join.rs:1357:22:
index out of bounds: the len is 0 but the index is 1
thread 'tokio-runtime-worker' panicked at datafusion/physical-plan/src/joins/sort_merge_join.rs:1357:22:
index out of bounds: the len is 0 but the index is 1
Error: Context("Join Error", External(JoinError::Panic(Id(88693), ...)))

Originally posted by @comphead in #9846 (comment)

@comphead comphead self-assigned this Apr 16, 2024
@comphead
Copy link
Contributor Author

The TPCH benchmarks for SMJ #10092

@comphead
Copy link
Contributor Author

Only Q21 failing

@comphead
Copy link
Contributor Author

comphead commented Apr 18, 2024

Narrowed down the problem to query

    with 
    t1 as (select 1 a, 2 b) 
        select * from t1 where exists (select 1 from t1 t2 where t2.a = t1.a and t2.b != t1.b);

UPD: simpler reproduce query

@comphead
Copy link
Contributor Author

The problem is in LeftSemi/LeftAnti join types with extra join filter.
So the join side gets built correctly but join filter for notEq case gets crashed

@comphead
Copy link
Contributor Author

Full test to reproduce

    #[tokio::test]
    async fn test_() -> Result<()> {
        let ctx: SessionContext = SessionContext::new();

        let sql = "set datafusion.optimizer.prefer_hash_join = false;";
        let _ = ctx.sql(sql).await?.collect().await?;
    
        let sql = "
        with 
        t1 as (select 1 a, 2 b) 
            select * from t1 where exists (select 1 from t1 t2 where t2.a = t1.a and t2.b != t1.a);
        ";
        let _ = ctx.sql(sql).await?.collect().await?;
    
        Ok(())
    }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant