Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add table engine delta. #13949

Merged
merged 10 commits into from
Dec 11, 2023
Merged

feat: add table engine delta. #13949

merged 10 commits into from
Dec 11, 2023

Conversation

youngsofun
Copy link
Member

@youngsofun youngsofun commented Dec 7, 2023

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

create connection my_s3_conn storage_type = 's3' access_key_id ='minioadmin' secret_access_key ='minioadmin' ENDPOINT_URL='http://127.0.0.1:9900';

create table test_delta engine = delta location = 's3://testbucket/admin/data/delta/delta-table/' connection_name = 'my_s3_conn'
  1. basic read only.
  2. pruning will be added in next pr.

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Dec 7, 2023
@youngsofun
Copy link
Member Author

youngsofun commented Dec 7, 2023

build_address_sanitizer fail cc @ariesdevil

= note: mold: error: undefined symbol: _Py_Dealloc

https://github.com/datafuselabs/databend/actions/runs/7126808480/job/19405459308?pr=13949

@youngsofun youngsofun force-pushed the delta branch 2 times, most recently from cf3658f to cca5db2 Compare December 7, 2023 12:58
@youngsofun youngsofun requested review from Xuanwo, RinChanNOWWW, ariesdevil and sundy-li and removed request for ariesdevil December 7, 2023 13:57
@ariesdevil
Copy link
Contributor

build_address_sanitizer fail cc @ariesdevil

= note: mold: error: undefined symbol: _Py_Dealloc

https://github.com/datafuselabs/databend/actions/runs/7126808480/job/19405459308?pr=13949

seems no python-dev in the dev-container, but I wonder why we need dependent on pyo3 for build databend-query. cc @everpcpc

@everpcpc
Copy link
Member

everpcpc commented Dec 8, 2023

cc @sundy-li

@sundy-li
Copy link
Member

sundy-li commented Dec 8, 2023

The delta crate introduces pyo3

@youngsofun
Copy link
Member Author

@sundy-li
Copy link
Member

sundy-li commented Dec 8, 2023

maybe cargo.lock keeps all crates cache (which also covers bendpy)

@Xuanwo
Copy link
Member

Xuanwo commented Dec 8, 2023

Databend has enabled pyarrow by default: https://github.com/datafuselabs/databend/blob/main/Cargo.toml#L161. This PR introduces delta, which is solely affected by this.

@youngsofun
Copy link
Member Author

youngsofun commented Dec 8, 2023

Databend has enabled pyarrow by default: https://github.com/datafuselabs/databend/blob/main/Cargo.toml#L161. This PR introduces delta, which is solely affected by this.

same failure after c923a8d

(venv) ➜  databend git:(delta) ✗ cargo tree -i pyo3   -p  databend-query
error: package ID specification `pyo3` did not match any packages

@Xuanwo

@youngsofun
Copy link
Member Author

youngsofun commented Dec 11, 2023

@Xuanwo @sundy-li @ariesdevil
fix by specify --manifest-path /src/binaries/Cargo.toml

428a765#diff-1cb10c29237b056e8e7af7c9b6fe5ef5f4b62eab251fee366e00748d2d679b55R56

@BohuTANG BohuTANG merged commit 27950ec into databendlabs:main Dec 11, 2023
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants