[Bug]: In milvus-standalone docker container. Issue related to goroutine, automaticly exits with code 134 after sometime. If no collection is loaded, no problem is caused.
#29744
Assignees
Labels
kind/bug
Issues or changes related a bug
stale
indicates no udpates for 30 days
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Milestone
Comments
devasheeshG
added
kind/bug
|
could you offer the full log. All the goroutines you offers seems to be reasonable. There might be some system signal that trigger the standalone process to stop. |
|
|
it seems that milvus keeps failing in reading&writing to the volumes. @congqixia could you please help to double check that? |
yanliang567
added
triage/accepted
|
failed to flush insert data"] [error="Resource requested is unreadable, please reduce your request rate] how many collections you have? is it S3, minio or other storage? |
|
I have only one collection and standard installation of milvus locally on unraid. If you want I can share a zip of volume folder. |
|
I also have the same questions , do you have a solution ？ |
|
Could you please refer this doc to export the whole Milvus logs for investigation? |
|
This is not fixed yet. My docker compose file looks like this version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
privileged: true
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
# - TZ=Asia/Kolkata
# - TZ=UTC+5:30
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
privileged: true
environment:
# TZ: Asia/Kolkata
# TZ: UTC+5:30
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.3.5
command: ["milvus", "run", "standalone"]
privileged: true
security_opt:
- seccomp:unconfined
environment:
# TZ: Asia/Kolkata
# TZ: UTC+5:30
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
networks:
default:
name: milvus |
|
你好，请问有没有解决办法呢？ |
|
any one can share the full logs of milvus? we cannot tell the solutions without knowing what was hanppening. please refer this doc to export the whole Milvus logs for investigation? |
|
我跟题主的报错完全一样，请问有解决方法了吗 |
|
I thought [2024/01/07 17:17:37.866 +00:00] [WARN] [storage/remote_chunk_manager.go:128] ["failed to put object"] [bucket=a-bucket] [path=files/insert_log/446748065920713215/446748065920713216/446748065920916737/101/446863183117025457] [error="Resource requested is unreadable, please reduce your request rate"] is the main problem. The minio is unreadable for some reason. Did you check your disk usage from both iops and disk space? |
|
@congqixia |
|
@yanliang567 |
Did you check your disk iops? what kind of disk you are using as data directory? |
|
I also have the same questions , do you have a solution ？Or does this have something to do with the long-term loading of collections when I deploy the interface? |
not able to donwload the log file |
|
sorry, i have solve this error.
compare the config ,find rootCoord without address ，so it can
n't connect the rootCoord Service。 Since chage the config ，it works correct.
address: localhost
修复了这个问题，就是配置文件里面 少了一些配置导致的。
***@***.***
From: yanliang567
Date: 2024-03-20 15:57
To: milvus-io/milvus
CC: 胡洋; Comment
Subject: Re: [milvus-io/milvus] [Bug]: In milvus-standalone docker container. Issue related to `goroutine`, automaticly exits with code `134` after sometime. If no collection is loaded, no problem is caused. (Issue #29744)
一样的报错 Uploading milvcus_error.log…
not able to donwload the log file
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
/reopen |
|
@likhitha283: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
@likhitha283 do you have the same situation above above? |
|
I have the same problem as the author. Can you guys help me to solve it? |
|
/reopen |
|
@NamNX1409: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
files/insert_log/452337435696709883/452337435696709884/452337435696909953/102/452354935055712590: IO failed[key=files/insert_log/452337435696709883/452337435696709884/452337435696909953/102/452354935055712590]: Resource requested is unreadable, please reduce your request rate [recovered] i think you got this error becasue your minio/S3 is rated limited. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Environment
Current Behavior
container
milvus-standaloneautomatically exits after loading a collection.Milvus Log
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc0044d5fe0 sp=0xc0044d5f88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0044d5fe8 sp=0xc0044d5fe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 9740 [chan receive, 1 minutes, locked to thread]:
runtime.gopark(0x730d868?, 0x4faff38?, 0x0?, 0x0?, 0x8?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0043cced0 sp=0xc0043cceb0 pc=0x1a2a116
runtime.chanrecv(0xc001f4c780, 0xc0043ccfa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0043ccf60 sp=0xc0043cced0 pc=0x19f25fd
runtime.chanrecv2(0xc001ce6460?, 0xc002d59770?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0043ccf88 sp=0xc0043ccf60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc0043ccfe0 sp=0xc0043ccf88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0043ccfe8 sp=0xc0043ccfe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 9741 [chan receive, 1 minutes, locked to thread]:
runtime.gopark(0x730d868?, 0x4faff38?, 0x0?, 0x0?, 0x8?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003bc1ed0 sp=0xc003bc1eb0 pc=0x1a2a116
runtime.chanrecv(0xc001f4c7e0, 0xc003bc1fa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc003bc1f60 sp=0xc003bc1ed0 pc=0x19f25fd
runtime.chanrecv2(0xc001ce6460?, 0xc002d59830?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc003bc1f88 sp=0xc003bc1f60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc003bc1fe0 sp=0xc003bc1f88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003bc1fe8 sp=0xc003bc1fe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 9742 [chan receive, 1 minutes, locked to thread]:
runtime.gopark(0x730d868?, 0x4faff38?, 0x0?, 0x0?, 0x8?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0033fded0 sp=0xc0033fdeb0 pc=0x1a2a116
runtime.chanrecv(0xc001f4c840, 0xc0033fdfa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0033fdf60 sp=0xc0033fded0 pc=0x19f25fd
runtime.chanrecv2(0xc001ce6460?, 0xc002d598f0?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc0033fdf88 sp=0xc0033fdf60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc0033fdfe0 sp=0xc0033fdf88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0033fdfe8 sp=0xc0033fdfe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 9743 [chan receive, 1 minutes, locked to thread]:
runtime.gopark(0x730d868?, 0x4faff38?, 0x0?, 0x0?, 0x8?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc002395ed0 sp=0xc002395eb0 pc=0x1a2a116
runtime.chanrecv(0xc001f4c8a0, 0xc002395fa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc002395f60 sp=0xc002395ed0 pc=0x19f25fd
runtime.chanrecv2(0xc001ce6460?, 0xc002d599b0?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc002395f88 sp=0xc002395f60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc002395fe0 sp=0xc002395f88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002395fe8 sp=0xc002395fe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 12460 [chan receive]:
runtime.gopark(0xc00401a120?, 0xc0024bd7a8?, 0x0?, 0x0?, 0xc003dc8712?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000594ed0 sp=0xc000594eb0 pc=0x1a2a116
runtime.chanrecv(0xc001fb7560, 0xc000594fa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000594f60 sp=0xc000594ed0 pc=0x19f25fd
runtime.chanrecv2(0xc001065040?, 0xc0027b98f0?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000594f88 sp=0xc000594f60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc000594fe0 sp=0xc000594f88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000594fe8 sp=0xc000594fe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
goroutine 12663 [chan receive]:
runtime.gopark(0x6332affbd4c01ff?, 0xc00276cf18?, 0x0?, 0x0?, 0xc000e91230?)
/usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00276ced0 sp=0xc00276ceb0 pc=0x1a2a116
runtime.chanrecv(0xc001bbede0, 0xc00276cfa8, 0x1)
/usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc00276cf60 sp=0xc00276ced0 pc=0x19f25fd
runtime.chanrecv2(0xc0003b9220?, 0xc0056a4390?)
/usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc00276cf88 sp=0xc00276cf60 pc=0x19f2138
github.com/panjf2000/ants/v2.(*goWorker).run.func1()
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:63 +0x7b fp=0xc00276cfe0 sp=0xc00276cf88 pc=0x3396e1b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00276cfe8 sp=0xc00276cfe0 pc=0x1a609e1
created by github.com/panjf2000/ants/v2.(*goWorker).run
/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
The text was updated successfully, but these errors were encountered: