feat: Geospatial Data Type and GIS Function Support for milvus #37417

tasty-gumi · 2024-11-04T10:48:23Z

Main Goals

Create and describe collections with geospatial fields, enabling both client and server to recognize and process geo fields.
Insert geospatial data as payload values in the insert binlog, and print the values for verification.
Load segments containing geospatial data into memory.
Ensure query outputs can display geospatial data.
Support filtering on GIS functions for geospatial columns.

Solution

Add Type: Modify the Milvus core by adding a Geospatial type in both the C++ and Go code layers, defining the Geospatial data structure and the corresponding interfaces.
Dependency Libraries: Introduce necessary geospatial data processing libraries. In the C++ source code, use Conan package management to include the GDAL library. In the Go source code, add the go-geom library to the go.mod file.
Protocol Interface: Revise the Milvus protocol to provide mechanisms for Geospatial message serialization and deserialization.
Data Pipeline: Facilitate interaction between the client and proxy using the WKT format for geospatial data. The proxy will convert all data into WKB format for downstream processing, providing column data interfaces, segment encapsulation, segment loading, payload writing, and cache block management.
Query Operators: Implement simple display and support for filter queries. Initially, focus on filtering based on spatial relationships for a single column of geospatial literal values, providing parsing and execution for query expressions.
Index Construction: Consider building an H3 index, utilizing the C interface provided by the H3 system.
Client Modification: Enable the client to handle user input for geospatial data and facilitate end-to-end testing.Check the modification in pymilvus.

delete incomplete H3 Index development and useless generated files.
fix conanfiles in milvus conan repo so that local can fetch the packages to build libraries

sre-ci-robot · 2024-11-04T10:48:34Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tasty-gumi
To complete the pull request process, please assign czs007 after the PR has been reviewed.
You can assign the PR to them by writing /assign @czs007 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-11-04T10:56:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T10:58:06Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T11:35:49Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T11:36:31Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

czs007 · 2024-11-04T12:12:13Z

rerun go-sdk

czs007 · 2024-11-04T12:12:24Z

rerun cpp-unit-test

mergify · 2024-11-04T12:52:00Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T13:10:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-11-04T13:15:43Z

/run-cpu-e2e

mergify · 2024-11-04T13:19:53Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T13:21:14Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:09:21Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-04T14:43:21Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

internal/core/unittest/test_utils/DataGen.h

mergify · 2024-11-05T08:36:01Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T08:57:51Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T09:11:29Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T12:02:43Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-08T12:29:49Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-08T13:51:04Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-08T14:26:42Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2024-11-09T01:59:01Z

@tasty-gumi
pytest : test] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=create auto index on type:JSON is not supported)>

[pytest : test] (api_request.py:45)

[pytest : test] [2024-11-08 14:24:30 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=create auto index on type:JSON is not supported)> (api_request.py:46)

[pytest : test] ---------- generated html file: file:///tmp/ci_logs/test/report.html -----------

[pytest : test] =========================== short test summary info ============================

[pytest : test] FAILED testcases/test_index.py::TestIndexInvalid::test_create_index_json

[pytest : test] !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!

mergify · 2024-11-11T11:32:09Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-12T10:13:53Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-12T12:38:32Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

czs007 · 2024-11-13T07:21:21Z

@tasty-gumi Intergration Test failed.

2024-11-12T13:06:50.0930073Z [2024/11/12 13:02:43.107 +00:00] [INFO] [querynodev2/services.go:483] ["start to load segments in parallel"] [collectionID=453880325253365774] [segmentType=Sealed] [requestSegments="[453880325253771188]"] [preparedSegments="[453880325253771188]"] [segmentNum=1] [concurrencyLevel=1]
2024-11-12T13:06:50.0931913Z [2024/11/12 13:02:43.107 +00:00] [WARN] [querynodev2/services.go:461] ["worker failed to load segments"] [collectionID=453880325253365774] [channel=by-dev-rootcoord-dml_0_453880325253365774v0] [replicaID=453880325350359041] [workID=1] [segments="[453880325253769583]"] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0933010Z [2024/11/12 13:02:43.108 +00:00] [INFO] [funcutil/parallel.go:86] ["load segment..."] [collectionID=453880325253365774] [segmentType=Sealed] [requestSegments="[453880325253771188]"] [preparedSegments="[453880325253771188]"] [partitionID=453880325253365775] [segmentID=453880325253771188] [segmentType=L1]
2024-11-12T13:06:50.0935035Z [2024/11/12 13:02:43.108 +00:00] [WARN] [querynode/service.go:301] ["delegator failed to load segments"] [collectionID=453880325253365774] [partitionID=453880325253365775] [shard=by-dev-rootcoord-dml_0_453880325253365774v0] [segmentID=453880325253769583] [level=L1] [currentNodeID=1] [dstNodeID=1] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0936237Z [2024/11/12 13:02:43.108 +00:00] [INFO] [segments/segment_loader.go:334] ["start loading segment files"] [collectionID=453880325253365774] [partitionID=453880325253365775] [shard=by-dev-rootcoord-dml_0_453880325253365774v0] [segmentID=453880325253771188] [rowNum=3000] [segmentType=Sealed]
2024-11-12T13:06:50.0938012Z [2024/11/12 13:02:43.108 +00:00] [WARN] [task/executor.go:160] ["failed to load segment"] [taskID=1731416037011] [collectionID=453880325253365774] [replicaID=453880325350359041] [segmentID=453880325253769583] [node=1] [source=segment_checker] [shardLeader=1] [error="At LoadSegment: => unsupported data type at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/ChunkedSegmentSealedImpl.cpp:442\n"]
2024-11-12T13:06:50.0939242Z [2024/11/12 13:02:43.109 +00:00] [INFO] [task/executor.go:142] ["execute ac

mergify · 2024-11-13T08:41:45Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

tasty-gumi · 2024-11-13T08:48:15Z

/run-cpu-e2e

mergify · 2024-11-13T09:00:33Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-13T09:25:13Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-13T09:44:41Z

@tasty-gumi cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-13T11:33:47Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-13T11:38:36Z

@tasty-gumi E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

add geospatial interface in src common change type define and add segcore support add storage & chunkdata support feature: go package storage & proxy & typeutil support geospatial type in internal and typeutil in pkg Signed-off-by: tasty-gumi <[email protected]> add geospatial interface in src common change type define and add segcore support change: use wkb only in core Signed-off-by: tasty-gumi <[email protected]> fix:the geospatial only use std::string as FieldDataImpl template paramters && add geospatial data generation && pass chunk ,growing , sealed test fix : merge confilcts after rebase ,test nullable not pass due to upstream feat:basic GIS Function expr and visitor impl and GIS proto support && add:storage test of geo data Signed-off-by: tasty-gumi <[email protected]> feat:add proxy validate (pass httpserver test) && plan parser of geospatialfunction fix:sealedseg && go tidy fix:go mod feat:can produce wkt result for pymilvus client feat: add parser and query operator for geos filed && print geos binlog as wkt fix:fielddataimpl interface Signed-off-by: tasty-gumi <[email protected]> fix: some format of code && segmentfault debug for rebase Signed-off-by: tasty-gumi <[email protected]> add: import util test for parquet and mix compaction test Signed-off-by: tasty-gumi <[email protected]> fix: delete useless file and fix error for rebase Signed-off-by: tasty-gumi <[email protected]> fix: git rebase for custom function feat Signed-off-by: tasty-gumi <[email protected]> fix:rename geospatial field && update proto && rewrite Geometry class with smart pointer Signed-off-by: tasty-gumi <[email protected]> add:last commit miss add files Signed-off-by: tasty-gumi <[email protected]> fix: geospatial name replace in test files && fix geomertry and parser fix:remove some file change for dev Signed-off-by: tasty-gumi <[email protected]> fix:remove size in if && add destory in ~Geometry() Signed-off-by: tasty-gumi <[email protected]> add:conan file gdal rep Signed-off-by: tasty-gumi <[email protected]> remove:gdal fPIC Signed-off-by: tasty-gumi <[email protected]> fix: for rebase Signed-off-by: tasty-gumi <[email protected]> remove:log_warn Signed-off-by: tasty-gumi <[email protected]> remove:gdal shared Signed-off-by: tasty-gumi <[email protected]> remove:tbbproxy Signed-off-by: tasty-gumi <[email protected]> fix:add gdal option && update go mod Signed-off-by: tasty-gumi <[email protected]> dev:change some scripts Signed-off-by: tasty-gumi <[email protected]> remove: dev scripts Signed-off-by: tasty-gumi <[email protected]> add:conan files dependency of gdal Signed-off-by: tasty-gumi <[email protected]> fix:fmt cpp code Signed-off-by: tasty-gumi <[email protected]> add:delete geos-config in cmake_bulid/bin which may cause permission deny Signed-off-by: tasty-gumi <[email protected]> fix: add go client geometry interface && fix group by test Signed-off-by: tasty-gumi <[email protected]> fix: mod tidy for tests go client Signed-off-by: tasty-gumi <[email protected]> fix:memory leak in test and go fmt Signed-off-by: tasty-gumi <[email protected]> fix: datagen function remove pkoffset Signed-off-by: tasty-gumi <[email protected]> fix: go-client test add entity.geometry Signed-off-by: tasty-gumi <[email protected]> fix: fix test args and add some annotations Signed-off-by: tasty-gumi <[email protected]> fix:name and remove wkt marshl MaxDecimalDigits limit Signed-off-by: tasty-gumi <[email protected]> fix:misspell Signed-off-by: tasty-gumi <[email protected]> fix:go client test Signed-off-by: tasty-gumi <[email protected]> fix:listA size Signed-off-by: tasty-gumi <[email protected]> add:field data in schema_test Signed-off-by: tasty-gumi <[email protected]> test:add mergefield data Signed-off-by: tasty-gumi <[email protected]> fix:test err code modify Signed-off-by: tasty-gumi <[email protected]> fmt code Signed-off-by: tasty-gumi <[email protected]> fix:add geo type in client Signed-off-by: tasty-gumi <[email protected]> fix:add type in chunksegment sealdimpl Signed-off-by: tasty-gumi <[email protected]> fix:add chunk writer for geometry Signed-off-by: tasty-gumi <[email protected]>

Signed-off-by: tasty-gumi <[email protected]>

mergify · 2024-12-07T04:42:44Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

Signed-off-by: tasty-gumi <[email protected]>

mergify · 2024-12-07T08:04:16Z

@tasty-gumi go-sdk check failed, comment rerun go-sdk can trigger the job again.

sre-ci-robot requested review from bigsheeper, congqixia and czs007 November 4, 2024 10:48

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/dependency Pull requests that update a dependency file area/test sig/testing test/integration integration test labels Nov 4, 2024

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 4, 2024

MrPresent-Han reviewed Nov 5, 2024

View reviewed changes

internal/core/unittest/test_utils/DataGen.h Show resolved Hide resolved

tasty-gumi force-pushed the geo_pr0905 branch from 3b943e5 to ebf2daa Compare November 5, 2024 08:04

tasty-gumi force-pushed the geo_pr0905 branch from 34ea50d to 4753bbe Compare November 9, 2024 08:19

mergify bot added the ci-passed label Nov 9, 2024

tasty-gumi force-pushed the geo_pr0905 branch from 4753bbe to 0173a25 Compare November 11, 2024 10:38

mergify bot removed the ci-passed label Nov 11, 2024

tasty-gumi force-pushed the geo_pr0905 branch from 0173a25 to 51ceb9a Compare November 12, 2024 09:23

tasty-gumi force-pushed the geo_pr0905 branch from bf84749 to 01c7024 Compare November 12, 2024 11:55

tasty-gumi force-pushed the geo_pr0905 branch from 01c7024 to 95b17a4 Compare November 13, 2024 08:28

tasty-gumi force-pushed the geo_pr0905 branch from 95b17a4 to 66bdeb8 Compare November 13, 2024 10:16

tasty-gumi added 3 commits December 6, 2024 10:40

fix:go mod change

58f369e

Signed-off-by: tasty-gumi <[email protected]>

fix 1207

7bde81d

Signed-off-by: tasty-gumi <[email protected]>

tasty-gumi force-pushed the geo_pr0905 branch from 66bdeb8 to 7bde81d Compare December 7, 2024 03:30

remove it test mixcompaction of geometry

e01c5fc

Signed-off-by: tasty-gumi <[email protected]>

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Are you sure you want to change the base?

feat: Geospatial Data Type and GIS Function Support for milvus #37417

Conversation

tasty-gumi commented Nov 4, 2024

Main Goals

Solution

sre-ci-robot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

czs007 commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024

mergify bot commented Nov 8, 2024

czs007 commented Nov 9, 2024

mergify bot commented Nov 11, 2024

mergify bot commented Nov 12, 2024

mergify bot commented Nov 12, 2024

czs007 commented Nov 13, 2024 • edited Loading

mergify bot commented Nov 13, 2024

tasty-gumi commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

mergify bot commented Dec 7, 2024

mergify bot commented Dec 7, 2024

czs007 commented Nov 13, 2024 •

edited

Loading