otel-collector not starting with fresh install

[Ruppsn]

Bug description

Deploying a fresh installation like described in the docs: https://signoz.io/docs/install/docker-swarm/

After deploying the otel-collector stops immediatly with the following error:

{"level":"error","timestamp":"2024-11-18T11:10:55.339Z","caller":"opamp/server_client.go:268","msg":"Collector failed for restart during rollback","component":"opamp-server-client","error":"failed to build pipelines: failed to create \"clickhousetraces\" exporter for data type \"traces\": error connecting to primary db: code: 81, message: Database signoz_traces does not exist","stacktrace":"github.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).reload\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:268\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).applyRemoteConfig\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:173\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*agentConfigManager).Apply\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/config_manager.go:159\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onRemoteConfigHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:209\ngithub.com/SigNoz/signoz-otel-collector/opamp.(*serverClient).onMessageFuncHandler\n\t/home/runner/work/signoz-otel-collector/signoz-otel-collector/opamp/server_client.go:199\ngithub.com/open-telemetry/opamp-go/client/types.CallbacksStruct.OnMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/types/callbacks.go:162\ngithub.com/open-telemetry/opamp-go/client/internal.(*receivedProcessor).ProcessReceivedMessage\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/internal/receivedprocessor.go:131\ngithub.com/open-telemetry/opamp-go/client/internal.(*wsReceiver).ReceiverLoop\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/internal/wsreceiver.go:57\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runOneCycle\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/wsclient.go:243\ngithub.com/open-telemetry/opamp-go/client.(*wsClient).runUntilStopped\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/wsclient.go:265\ngithub.com/open-telemetry/opamp-go/client/internal.(*ClientCommon).StartConnectAndRun.func1\n\t/home/runner/go/pkg/mod/github.com/open-telemetry/[email protected]/client/internal/clientcommon.go:197"}

Expected behavior

Everything works

How to reproduce

Setup is exactly like in the docs. Fresh installation, fresh swarm etc.

https://signoz.io/docs/install/docker-swarm/

Version information

• Signoz version: 0.58.1
• Your OS and version: RedHat
• Your CPU Architecture(ARM/Intel): Intel

[welcome]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

[Ruppsn]

Changed the Host and tried again. Same Error and inside clickhosue DB

{"date_time":"1732002011.832477","thread_name":"TCPServerConnection ([#3])","thread_id":"50","level":"Error","query_id":"","logger_name":"ServerErrorHandler","message":"Code: 81. DB::Exception: Database signoz_metrics does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x0000000010d1b4cf in \/usr\/bin\/clickhouse\n3. DB::Context::setCurrentDatabase(String const&) @ 0x0000000010c6b352 in \/usr\/bin\/clickhouse\n4. DB::TCPHandler::runImpl() @ 0x000000001292292f in \/usr\/bin\/clickhouse\n5. DB::TCPHandler::run() @ 0x0000000012933eb9 in \/usr\/bin\/clickhouse\n6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n8. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n10. ? @ 0x00007faf90fac609\n11. ? @ 0x00007faf90ed1353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"} {"date_time":"1732002011.832752","thread_name":"TCPServerConnection ([#4])","thread_id":"722","level":"Error","query_id":"","logger_name":"ServerErrorHandler","message":"Code: 81. DB::Exception: Database signoz_logs does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x0000000010d1b4cf in \/usr\/bin\/clickhouse\n3. DB::Context::setCurrentDatabase(String const&) @ 0x0000000010c6b352 in \/usr\/bin\/clickhouse\n4. DB::TCPHandler::runImpl() @ 0x000000001292292f in \/usr\/bin\/clickhouse\n5. DB::TCPHandler::run() @ 0x0000000012933eb9 in \/usr\/bin\/clickhouse\n6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n8. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n10. ? @ 0x00007faf90fac609\n11. ? @ 0x00007faf90ed1353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"} {"date_time":"1732002011.833500","thread_name":"TCPServerConnection ([#5])","thread_id":"723","level":"Error","query_id":"","logger_name":"ServerErrorHandler","message":"Code: 81. DB::Exception: Database signoz_metrics does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x0000000010d1b4cf in \/usr\/bin\/clickhouse\n3. DB::Context::setCurrentDatabase(String const&) @ 0x0000000010c6b352 in \/usr\/bin\/clickhouse\n4. DB::TCPHandler::runImpl() @ 0x000000001292292f in \/usr\/bin\/clickhouse\n5. DB::TCPHandler::run() @ 0x0000000012933eb9 in \/usr\/bin\/clickhouse\n6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n8. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n10. ? @ 0x00007faf90fac609\n11. ? @ 0x00007faf90ed1353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"} {"date_time":"1732002011.846768","thread_name":"TCPServerConnection ([#3])","thread_id":"50","level":"Error","query_id":"","logger_name":"ServerErrorHandler","message":"Code: 81. DB::Exception: Database signoz_traces does not exist. (UNKNOWN_DATABASE), Stack trace (when copying this message, always include the lines below):\n\n0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c800f1b in \/usr\/bin\/clickhouse\n1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007219a43 in \/usr\/bin\/clickhouse\n2. DB::DatabaseCatalog::assertDatabaseExists(String const&) const @ 0x0000000010d1b4cf in \/usr\/bin\/clickhouse\n3. DB::Context::setCurrentDatabase(String const&) @ 0x0000000010c6b352 in \/usr\/bin\/clickhouse\n4. DB::TCPHandler::runImpl() @ 0x000000001292292f in \/usr\/bin\/clickhouse\n5. DB::TCPHandler::run() @ 0x0000000012933eb9 in \/usr\/bin\/clickhouse\n6. Poco::Net::TCPServerConnection::start() @ 0x00000000153a5a72 in \/usr\/bin\/clickhouse\n7. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153a6871 in \/usr\/bin\/clickhouse\n8. Poco::PooledThread::run() @ 0x000000001549f047 in \/usr\/bin\/clickhouse\n9. Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001549d67d in \/usr\/bin\/clickhouse\n10. ? @ 0x00007faf90fac609\n11. ? @ 0x00007faf90ed1353\n (version 24.1.2.5 (official build))","source_file":"src\/Common\/Exception.cpp; void DB::tryLogCurrentExceptionImpl(Poco::Logger *, const std::string &)","source_line":"222"}

who or what initialises thes database?

[Ruppsn]

When i try a 0.55.0 Version with git checkout v0.55.0 everything works out of the box.

[srikanthccv]

It says db doesn't exist which means the migration did run at all or didn't run successfully. Please share more info about the migratory container.

[Ruppsn]

Thanks for your reply.

I am using it in a docker-swarm.

I did a checkout on 0.58.2 and removed the old stack like described in the docs. Then i did a docker stack deploy.

My Services are all up but not the otelcol and migrator:

docker service ls
jxo1kashs9uy   signoz_alertmanager              replicated   1/1        signoz/alertmanager:0.23.7                   
r22sevxttc66   signoz_clickhouse                replicated   1/1        clickhouse/clickhouse-server:24.1.2-alpine   
294zwmp4eryp   signoz_frontend                  replicated   1/1        signoz/frontend:0.58.2                       
5gax6eqk6duf   signoz_otel-collector            global       0/1        signoz/signoz-otel-collector:0.111.8         *:4317-4318->4317-4318/tcp, *:24224->24224/tcp
yd9o091c5tbw   signoz_otel-collector-migrator   replicated   0/1        signoz/signoz-schema-migrator:0.111.8        
sg1r4vxa0vg8   signoz_query-service             replicated   1/1        signoz/query-service:0.58.2                  
su2uu3ipenpe   signoz_zookeeper-1               replicated   1/1        bitnami/zookeeper:3.7.1                      *:2181->2181/tcp, *:2888->2888/tcp, *:3888->3888/tcp

docker service ps signoz_otel-collector-migrator 
ID             NAME                               IMAGE                                   NODE      DESIRED STATE   CURRENT STATE            ERROR     PORTS
2r13rs72ngol   signoz_otel-collector-migrator.1   signoz/signoz-schema-migrator:0.111.8   covm77    Shutdown        Complete 4 minutes ago   

docker service logs signoz_otel-collector-migrator 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Signoz Schema Migrator
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Usage:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   signoz-schema-migrator [command]
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Available Commands:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   async       Run migrations in async mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   completion  Generate the autocompletion script for the specified shell
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   help        Help about any command
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   sync        Run migrations in sync mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Flags:
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --cluster-name string   Cluster name to use while running migrations (default "cluster")
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --dev                   Development mode
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --dsn string            Clickhouse DSN
signoz_otel-collector-migrator.1.2r13rs72ngol@    |   -h, --help                  help for signoz-schema-migrator
signoz_otel-collector-migrator.1.2r13rs72ngol@    |       --replication           Enable replication
signoz_otel-collector-migrator.1.2r13rs72ngol@    | 
signoz_otel-collector-migrator.1.2r13rs72ngol@    | Use "signoz-schema-migrator [command] --help" for more information about a command.

It seems it seems like the migrator doenst do anything.

Thank you very much

[Ruppsn]

#6489