Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Invalid trace stack for exception of type: milvus::SegcoreError #37944

Open
1 task done
lxl0928 opened this issue Nov 22, 2024 · 7 comments
Open
1 task done

[Bug]: Invalid trace stack for exception of type: milvus::SegcoreError #37944

lxl0928 opened this issue Nov 22, 2024 · 7 comments
Assignees
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@lxl0928
Copy link

lxl0928 commented Nov 22, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.4.15-for-gpu
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): restful api
- OS(Ubuntu or CentOS): 220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
- CPU/Memory: Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz 
- GPU: Nvidia A10 * 4
- Others:

Current Behavior

When I set the TTL (Time-To-Live) for a Milvus collection to ttl=2360024, and concurrently perform vector insertion and vector search tests, while loading 390,000 rows of vectors, it consistently triggers the error:
W20241122 07:19:39.079182 2555 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

Expected Behavior

No Error。

Steps To Reproduce

0. running on GPUs
1. Set Collection's TTL
2. insert vector and search vector at the same time, insert qps: 112req/s, search qps: 108req/s
3. running 39w+ vectors inserted, the error `Invalid trace stack for exception of type: milvus::SegcoreError` excepted.

Milvus Log

milvus-standalone-for-gpu-11221521.log

[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/validate.go:50] ["read target partitions"] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [partitionIDs="[454101003657545081,454101003657545072,454101003657545075,454101003657545073,454101003657545079,454101003657545076,454101003657545074,454101003657545082,454101003657545084,454101003657545085,454101003657545087,454101003657545078,454101003657545080,454101003657545077,454101003657545083,454101003657545086]"]
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/validate.go:50] ["read target partitions"] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [partitionIDs="[454101003657545079,454101003657545076,454101003657545074,454101003657545078,454101003657545080,454101003657545082,454101003657545084,454101003657545085,454101003657545087,454101003657545077,454101003657545083,454101003657545086,454101003657545081,454101003657545072,454101003657545075,454101003657545073]"]
W20241122 07:19:39.055075  2525 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/segment.go:575] ["search segment..."] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [segmentID=454101003659154520] [segmentType=Growing] [withIndex=false]
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/segment.go:575] ["search segment..."] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [segmentID=454101003659154519] [segmentType=Growing] [withIndex=false]
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/segment.go:575] ["search segment..."] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [segmentID=454101003659154109] [segmentType=Growing] [withIndex=false]
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/segment.go:575] ["search segment..."] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [segmentID=454101003659154591] [segmentType=Growing] [withIndex=false]
[2024/11/22 07:19:39.055 +00:00] [DEBUG] [segments/segment.go:575] ["search segment..."] [traceID=60aa8711d5123a4f8fa0f9a896a7547e] [collectionID=454101003657545071] [segmentID=454101003659154586] [segmentType=Growing] [withIndex=false]
W20241122 07:19:39.064556  3193 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241122 07:19:39.079182  2555 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

Anything else?

Milvus.yaml:.log -> .yaml

milvus-yaml.log

milvus-standalone docker-compose:

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.15-gpu
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              capabilities: ["gpu"]
              device_ids: ["0"]
    depends_on:
      - "etcd"
      - "minio"

collection init.py:


# coding: utf-8
import time
import configparser
from tqdm import tqdm
from pymilvus import connections, utility, FieldSchema, CollectionSchema, DataType, Collection, MilvusClient

milvus_uri = "http://xx.xx.xx.xx:19530"
milvus_token = "root:Milvus"


# 创建 Collection
collection_name = "FaceV4"

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True, description="primary id"),
    FieldSchema(name="sn", dtype=DataType.VARCHAR, max_length=32, description="设备sn"),
    FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=768, description="特征向量", mmap_enabled=True),
    FieldSchema(name="app_id", dtype=DataType.VARCHAR, max_length=36, is_partition_key=True, description="租户"),
    FieldSchema(name="timestamp", dtype=DataType.INT64, description="特征产生的时间戳")
]

schema = CollectionSchema(fields, description="Demo collection for PyMilvus", ttl=24 * 3600 * 2)

client = MilvusClient(uri=milvus_uri, token=milvus_token, db_name="default")

index_params = client.prepare_index_params()

index_params.add_index(
    field_name="id",
    index_type="STL_SORT",  # 标准排序算法
    idnex_name="inx_id"
)
index_params.add_index(
    field_name="sn",
    index_type="INVERTED",  # 倒排索引
    idnex_name="inx_sn"
)
index_params.add_index(
    field_name="app_id",
    index_type="INVERTED",  # 倒排索引
    idnex_name="inx_app_id"
)
index_params.add_index(
    field_name="timestamp",
    index_type="STL_SORT",  # 标准排序算法
    idnex_name="inx_timestamp",
)

index_params.add_index(
    field_name="vector",
    index_type="GPU_IVF_FLAT",  # IVF_FLAT 不进行任何压缩,因此它生成的索引文件大小与原始的非索引向量数据大致相同
    metric_type="L2",
    params={"nlist": 256}
)

client.create_collection(
    collection_name=collection_name,
    schema=schema,
    index_params=index_params,
)
print(f"Collection '{collection_name}' created.")

time.sleep(5)

res = client.get_load_state(
    collection_name=collection_name
)

print(res)




@lxl0928 lxl0928 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 22, 2024
@yanliang567
Copy link
Contributor

/assign @Presburger
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 22, 2024
@yanliang567
Copy link
Contributor

yanliang567 commented Nov 22, 2024

/assign @smellthemoon
please also help to fix the logs with data info
/unassign

@lxl0928
Copy link
Author

lxl0928 commented Nov 29, 2024

if docker-compose.yaml enabled: restart:always, with the insert vectors growing, and search vectors at the same time, the milvus-for-gpu-standalone-v2.14.5 will restart many times for the Error: W20241129 04:01:20.444448 1106 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

the detail logs:

[2024/11/29 04:01:20.387 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454256381328398176]
[2024/11/29 04:01:20.387 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=08df80d8ceac89f9026e76ffb5cab22d] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454256381328398176,\"NumOfRows\":1,\"ExpireTime\":454256985996001283}]"]
[2024/11/29 04:01:20.387 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=757161bf4c6695227722fb843653eaf6] [code=200] [latency=6.409092ms] [client=172.18.0.9] [method=POST] [error=]
[2024/11/29 04:01:20.391 +00:00] [INFO] [datacoord/services.go:199] ["handle assign segment request"] [traceID=420db6cd8f3e0fb062ca5b02267769c1] [collectionID=454104051343163873] [partitionID=454104051343163885] [channelName=by-dev-rootcoord-dml_0_454104051343163873v0] [count=1] ["segment level"=Legacy]
[2024/11/29 04:01:20.392 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454190707293993753]
[2024/11/29 04:01:20.392 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=420db6cd8f3e0fb062ca5b02267769c1] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454190707293993753,\"NumOfRows\":1,\"ExpireTime\":454256985996001285}]"]
[2024/11/29 04:01:20.392 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=52bc5a13558138aee36971c6331e0db4] [code=200] [latency=6.442173ms] [client=172.18.0.9] [method=POST] [error=]
W20241129 04:01:20.404749  1706 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 04:01:20.419394  1710 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 04:01:20.444448  1106 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

@lxl0928
Copy link
Author

lxl0928 commented Nov 29, 2024

企业微信截图_4e468c04-e0d0-4fbe-8076-2bee1d77003b

first time:

[2024/11/29 04:01:20.387 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454256381328398176]
[2024/11/29 04:01:20.387 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=08df80d8ceac89f9026e76ffb5cab22d] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454256381328398176,\"NumOfRows\":1,\"ExpireTime\":454256985996001283}]"]
[2024/11/29 04:01:20.387 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=757161bf4c6695227722fb843653eaf6] [code=200] [latency=6.409092ms] [client=172.18.0.9] [method=POST] [error=]
[2024/11/29 04:01:20.391 +00:00] [INFO] [datacoord/services.go:199] ["handle assign segment request"] [traceID=420db6cd8f3e0fb062ca5b02267769c1] [collectionID=454104051343163873] [partitionID=454104051343163885] [channelName=by-dev-rootcoord-dml_0_454104051343163873v0] [count=1] ["segment level"=Legacy]
[2024/11/29 04:01:20.392 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454190707293993753]
[2024/11/29 04:01:20.392 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=420db6cd8f3e0fb062ca5b02267769c1] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454190707293993753,\"NumOfRows\":1,\"ExpireTime\":454256985996001285}]"]
[2024/11/29 04:01:20.392 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=52bc5a13558138aee36971c6331e0db4] [code=200] [latency=6.442173ms] [client=172.18.0.9] [method=POST] [error=]
W20241129 04:01:20.404749  1706 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 04:01:20.419394  1710 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 04:01:20.444448  1106 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

second time:

[2024/11/29 04:05:08.418 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=48fffdc91385841fbf9f61168a4ad2bb] [code=200] [latency=6.150262ms] [client=172.18.0.7] [method=POST] [error=]
[2024/11/29 04:05:08.418 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=0531636a93d47a457b4f7097e1db483c] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454190707293793203]"] [currentNodeID=175] [scope=Historical] [needTransfer=true]
[2024/11/29 04:05:08.418 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=0de27b02176f939e44f7326dc9ef88d7] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454190707293180902]"] [currentNodeID=175] [scope=Historical] [needTransfer=false]
[2024/11/29 04:05:08.418 +00:00] [INFO] [querynodev2/services.go:576] ["start to release segments"] [traceID=0de27b02176f939e44f7326dc9ef88d7] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454190707293180902]"] [currentNodeID=175]
[2024/11/29 04:05:08.418 +00:00] [INFO] [delegator/delegator_data.go:793] ["delegator start to release segments"] [traceID=0531636a93d47a457b4f7097e1db483c] [collectionID=454104051343163873] [channel=by-dev-rootcoord-dml_0_454104051343163873v0] [replicaID=454104737449246721] [segmentIDs="[454190707293793203]"] [nodeID=175] [scope=Historical] [force=false]
[2024/11/29 04:05:08.418 +00:00] [INFO] [segments/manager.go:810] ["remove segment from cache"] [traceID=0de27b02176f939e44f7326dc9ef88d7] [segmentID=454190707293180902]
[2024/11/29 04:05:08.418 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=0531636a93d47a457b4f7097e1db483c] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454190707293793203]"] [currentNodeID=175] [scope=Historical] [needTransfer=false]
[2024/11/29 04:05:08.418 +00:00] [INFO] [querynodev2/services.go:576] ["start to release segments"] [traceID=0531636a93d47a457b4f7097e1db483c] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454190707293793203]"] [currentNodeID=175]
[2024/11/29 04:05:08.418 +00:00] [INFO] [segments/manager.go:810] ["remove segment from cache"] [traceID=0531636a93d47a457b4f7097e1db483c] [segmentID=454190707293793203]
W20241129 04:05:08.436905   538 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 04:05:08.469408   353 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
[2024/11/29 04:05:18.949 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
[2024/11/29 04:05:18.970 +00:00] [INFO] [roles/roles.go:333] ["starting running Milvus components"]
[2024/11/29 04:05:18.970 +00:00] [INFO] [roles/roles.go:182] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2024/11/29 04:05:18.983 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/11/29 04:05:18.985 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"svector-etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/11/29 04:05:18.985 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[svector-etcd:2379]"] [minVersion=1.3]
[2024/11/29 04:05:18.989 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[svector-etcd:2379]"]
[2024/11/29 04:05:18.989 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/11/29 04:05:18.994 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/11/29 04:05:18.995 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]

third time:

[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=7ed76182e060cc0baf374334e99f7b90] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454256381326787255]"] [currentNodeID=180] [scope=Historical] [needTransfer=false]
[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=afeb0067aa40fcfe3eee42a776738597] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454257571508670526]"] [currentNodeID=180] [scope=Historical] [needTransfer=false]
[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:546] ["received release segment request"] [traceID=714b778c67fbcaf33efb5cd4276da372] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454256381328398176]"] [currentNodeID=180] [scope=Historical] [needTransfer=false]
[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:576] ["start to release segments"] [traceID=7ed76182e060cc0baf374334e99f7b90] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454256381326787255]"] [currentNodeID=180]
[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:576] ["start to release segments"] [traceID=afeb0067aa40fcfe3eee42a776738597] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454257571508670526]"] [currentNodeID=180]
[2024/11/29 05:21:59.369 +00:00] [INFO] [querynodev2/services.go:576] ["start to release segments"] [traceID=714b778c67fbcaf33efb5cd4276da372] [collectionID=454104051343163873] [shard=by-dev-rootcoord-dml_0_454104051343163873v0] [segmentIDs="[454256381328398176]"] [currentNodeID=180]
[2024/11/29 05:21:59.369 +00:00] [INFO] [segments/manager.go:810] ["remove segment from cache"] [traceID=7ed76182e060cc0baf374334e99f7b90] [segmentID=454256381326787255]
[2024/11/29 05:21:59.369 +00:00] [INFO] [segments/manager.go:810] ["remove segment from cache"] [traceID=714b778c67fbcaf33efb5cd4276da372] [segmentID=454256381328398176]
[2024/11/29 05:21:59.369 +00:00] [INFO] [segments/manager.go:810] ["remove segment from cache"] [traceID=afeb0067aa40fcfe3eee42a776738597] [segmentID=454257571508670526]
[2024/11/29 05:21:59.374 +00:00] [INFO] [datacoord/services.go:199] ["handle assign segment request"] [traceID=c91a66416d2b868fd3a0a6815c10bbb3] [collectionID=454104051343163873] [partitionID=454104051343163881] [channelName=by-dev-rootcoord-dml_0_454104051343163873v0] [count=1] ["segment level"=Legacy]
[2024/11/29 05:21:59.375 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454256988539149693]
[2024/11/29 05:21:59.375 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=c91a66416d2b868fd3a0a6815c10bbb3] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454256988539149693,\"NumOfRows\":1,\"ExpireTime\":454258254503215108}]"]
[2024/11/29 05:21:59.376 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=af36e323c8e3ee5dd8936e64b85c8322] [code=200] [latency=8.541964ms] [client=172.18.0.7] [method=POST] [error=]
[2024/11/29 05:21:59.378 +00:00] [INFO] [datacoord/services.go:199] ["handle assign segment request"] [traceID=41f1394011cfe4d2a9e65aa91d5e82cc] [collectionID=454104051343163873] [partitionID=454104051343163874] [channelName=by-dev-rootcoord-dml_0_454104051343163873v0] [count=1] ["segment level"=Legacy]
[2024/11/29 05:21:59.379 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454257678002721787]
[2024/11/29 05:21:59.380 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=41f1394011cfe4d2a9e65aa91d5e82cc] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454257678002721787,\"NumOfRows\":1,\"ExpireTime\":454258254503215110}]"]
[2024/11/29 05:21:59.381 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=1d1df3a974eb31eb18f29d9cb0222477] [code=200] [latency=8.012974ms] [client=172.18.0.8] [method=POST] [error=]
[2024/11/29 05:21:59.416 Z] [GIN] [/v2/vectordb/entities/search] [traceID=953bebae04b0ee5f4cce954e3489ee51] [code=200] [latency=49.050999ms] [client=172.18.0.7] [method=POST] [error=]
[2024/11/29 05:21:59.421 +00:00] [INFO] [datacoord/services.go:199] ["handle assign segment request"] [traceID=8bd25fe942fd450b6b1848901f973fa9] [collectionID=454104051343163873] [partitionID=454104051343163879] [channelName=by-dev-rootcoord-dml_0_454104051343163873v0] [count=1] ["segment level"=Legacy]
[2024/11/29 05:21:59.422 +00:00] [INFO] [datacoord/meta.go:1290] ["meta update: add allocation - complete"] [segmentID=454257086947953834]
[2024/11/29 05:21:59.422 +00:00] [INFO] [datacoord/services.go:228] ["success to assign segments"] [traceID=8bd25fe942fd450b6b1848901f973fa9] [collectionID=454104051343163873] [assignments="[{\"SegmentID\":454257086947953834,\"NumOfRows\":1,\"ExpireTime\":454258254516322308}]"]
[2024/11/29 05:21:59.423 Z] [GIN] [/v2/vectordb/entities/insert] [traceID=f66ae836f9705e8ba18b8bc2e85f22cf] [code=200] [latency=9.170595ms] [client=172.18.0.8] [method=POST] [error=]
[2024/11/29 05:21:59.426 Z] [GIN] [/v2/vectordb/entities/search] [traceID=a41862ba334e6d234c15525771f8c08f] [code=200] [latency=57.632462ms] [client=172.18.0.8] [method=POST] [error=]
W20241129 05:21:59.431946   364 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 05:21:59.489991  1926 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError
W20241129 05:21:59.499696  2190 ExceptionTracer.cpp:187] Invalid trace stack for exception of type: milvus::SegcoreError

@zhagnlu
Copy link
Contributor

zhagnlu commented Dec 10, 2024

image
image
it seems remove directory failed.

@zhagnlu
Copy link
Contributor

zhagnlu commented Dec 10, 2024

this error happens when release segment, but interface not show which dir path remove failed.

@zhagnlu
Copy link
Contributor

zhagnlu commented Dec 17, 2024

find the root cause:

  1. inverted index need to remove index related dir when release segment.
  2. but for standalone mode, root dir not correct because local chunk manager init twice for query/index roles. so will mix the rootpath

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

5 participants