[Bug]: Docker Compose Standalone running Start fail

[ClarkThan]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.2.14
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

startup fail

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

: MADV_DONTNEED does not work (memset will be used instead)
: (This is the expected behaviour if you are running under QEMU)
[2023/08/25 06:36:15.980 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"]
[2023/08/25 06:36:15.989 +00:00] [INFO] [paramtable/quota_param.go:780] ["init disk quota"] [diskQuota(MB)=+inf]
[2023/08/25 06:36:15.990 +00:00] [INFO] [paramtable/quota_param.go:802] ["init disk quota per DB"] [diskQuotaPerCollection(MB)=+inf]
[2023/08/25 06:36:15.994 +00:00] [INFO] [paramtable/component_param.go:1578] ["init segment max idle time"] [value=10m0s]
[2023/08/25 06:36:15.994 +00:00] [INFO] [paramtable/component_param.go:1583] ["init segment min size from idle to sealed"] [value=16]
[2023/08/25 06:36:15.994 +00:00] [INFO] [paramtable/component_param.go:1593] ["init segment max binlog file to sealed"] [value=32]
[2023/08/25 06:36:15.995 +00:00] [INFO] [paramtable/component_param.go:1588] ["init segment expansion rate"] [value=1.25]
[2023/08/25 06:36:15.995 +00:00] [WARN] [paramtable/component_param.go:1928] ["DeployModeEnv is not set, use default"] [default=0.5]
[2023/08/25 06:36:15.996 +00:00] [INFO] [paramtable/base_table.go:150] ["cannot find etcd.rootPath"]
[2023/08/25 06:36:15.997 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}]
2023/08/25 06:36:16 maxprocs: Leaving GOMAXPROCS=5: CPU quota undefined

__  _________ _   ____  ______

/ |/ / / /| | / / / / / __/
/ /|/ // // /| |/ / // /\
// /////_/__/

Welcome to use Milvus!
Version: v2.2.14
Built: Wed Aug 23 12:01:10 UTC 2023
GitCommit: b34d409
GoVersion: go version go1.18.3 linux/amd64

open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2023/08/25 06:36:16.100 +00:00] [INFO] [roles/roles.go:228] ["starting running Milvus components"]
[2023/08/25 06:36:16.100 +00:00] [INFO] [roles/roles.go:154] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so]
[2023/08/25 06:36:16.138 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"]
[2023/08/25 06:36:16.139 +00:00] [INFO] [paramtable/quota_param.go:780] ["init disk quota"] [diskQuota(MB)=+inf]
[2023/08/25 06:36:16.139 +00:00] [INFO] [paramtable/quota_param.go:802] ["init disk quota per DB"] [diskQuotaPerCollection(MB)=+inf]
[2023/08/25 06:36:16.140 +00:00] [INFO] [paramtable/component_param.go:1578] ["init segment max idle time"] [value=10m0s]
[2023/08/25 06:36:16.140 +00:00] [INFO] [paramtable/component_param.go:1583] ["init segment min size from idle to sealed"] [value=16]
[2023/08/25 06:36:16.140 +00:00] [INFO] [paramtable/component_param.go:1593] ["init segment max binlog file to sealed"] [value=32]
[2023/08/25 06:36:16.140 +00:00] [INFO] [paramtable/component_param.go:1588] ["init segment expansion rate"] [value=1.25]
[2023/08/25 06:36:16.141 +00:00] [INFO] [paramtable/base_table.go:150] ["cannot find etcd.rootPath"]
[2023/08/25 06:36:16.141 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}]
[2023/08/25 06:36:16.178 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"]
[2023/08/25 06:36:16.178 +00:00] [INFO] [server/global_rmq.go:55] ["initializing global rmq"] [path=/var/lib/milvus/rdb_data]
[2023/08/25 06:36:16.181 +00:00] [INFO] [server/rocksmq_impl.go:154] ["Start rocksmq "] ["max proc"=5] [parallism=1] ["lru cache"=536870912]
panic: Invalid argument: Column families not opened: properties

goroutine 1 [running]:
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc000e1fe48, 0x1, {0x0, 0x0})
/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:245 +0xcf5
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xc0006701e0, {0xc00004e0c0?, 0x3, 0x3}, 0xc000c68c60)
/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:117 +0x6ae
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc00004e0c0?, 0x3, 0x3})
/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e
main.main()
/go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e

Anything else?

No response

[yanliang567]

/assign @LoveEachDay
could you please help on this issue?

/unassign

[ikuru-ma]

Also ran into the same with a standalone installation of 2.2.8. The issue was triggered when I tried to restart the milvus containers with a YAML file for milvus 2.3.x instead of 2.2.8 without removing the volumes. Unfortunately, I had to remove the volumes from the 2.2.8 version to get the milvus containers started again. I did keep the volume folders in case there is a way to restore the db data back into milvus?

[yanliang567]

Also ran into the same with a standalone installation of 2.2.8. The issue was triggered when I tried to restart the milvus containers with a YAML file for milvus 2.3.x instead of 2.2.8 without removing the volumes. Unfortunately, I had to remove the volumes from the 2.2.8 version to get the milvus containers started again. I did keep the volume folders in case there is a way to restore the db data back into milvus?

what is the error in your case? could you please attach the milvus logs for investigation?

[ikuru-ma]

Unfortunately, in a rush to get my services up and running, I did not store my logs. However, from my search history I did have both of these lines in my logs when I viewed them:

"panic: Invalid argument: Column families not opened: properties"
"/go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e"

I will try to reproduce the error once I get the chance.

[xiaofan-luan]

Unfortunately, in a rush to get my services up and running, I did not store my logs. However, from my search history I did have both of these lines in my logs when I viewed them:

"panic: Invalid argument: Column families not opened: properties" "/go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e"

I will try to reproduce the error once I get the chance.

Did you upgrade to 2.3 and trying to rollback?

[xiaofan-luan]

the reason is 2.3 introduced another column family in rocksdb and use old code will report error

[ikuru-ma]

Unfortunately, in a rush to get my services up and running, I did not store my logs. However, from my search history I did have both of these lines in my logs when I viewed them:
"panic: Invalid argument: Column families not opened: properties" "/go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e"
I will try to reproduce the error once I get the chance.

Did you upgrade to 2.3 and trying to rollback?

Yes, that's exactly what I did. I had downloaded the compose file for 2.3.x without realizing it and restarted my containers with that.

the reason is 2.3 introduced another column family in rocksdb and use old code will report error

That probably explains the error. Is there any way to restore the old data back? I kept a backup of all the volumes.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[darthjaja6]

@ikuru-ma Did you find a solution? I had the same issue after performing the same upgrade. Any idea how to migrate to 2.3+?

[master006]

@ikuru-ma Did you find a solution? I had the same issue after performing the same upgrade. Any idea how to migrate to 2.3+?