[BUG]java.lang.ClassNotFoundException: org.opensearch.common.lifecycle.LifecycleListener

[xiaodizi]

Please consult, my Opensearch is the master branch pulled, job-scheduler is also the master branch pulled. The version number is 3.0.0, and after build, the installation is fine.

但是我kill掉opensearch的进程，重新运行就出现了错误。
[2023-10-27T16:26:39,074][WARN ][o.o.p.PluginsService ] [lei.local] Non-plugin file located in the plugins folder with the following name: [.DS_Store] [2023-10-27T16:26:39,350][WARN ][o.o.p.PluginsService ] [lei.local] Non-plugin file located in the plugins folder with the following name: [.DS_Store] [2023-10-27T16:26:39,532][INFO ][o.o.n.Node ] [lei.local] version[3.0.0], pid[71829], build[tar/unknown/unknown], OS[Mac OS X/13.5/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/11.0.18/11.0.18+9-LTS-195] [2023-10-27T16:26:39,533][INFO ][o.o.n.Node ] [lei.local] JVM home [/Users/lei.fu/java/jdk-11.0.18.jdk/Contents/Home], using bundled JDK [false] [2023-10-27T16:26:39,533][INFO ][o.o.n.Node ] [lei.local] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djdk.attach.allowAttachSelf=true, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management.rmi/com.sun.jmx.remote.internal.rmi=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED, --add-exports=java.rmi/sun.rmi.server=ALL-UNNAMED, --add-exports=java.sql/java.sql=ALL-UNNAMED, --add-opens=java.base/java.lang.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED, --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED, --add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED, --add-opens=java.base/jdk.internal.math=ALL-UNNAMED, --add-opens=java.base/jdk.internal.module=ALL-UNNAMED, --add-opens=java.base/jdk.internal.util.jar=ALL-UNNAMED, --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED, -Djava.io.tmpdir=/var/folders/10/j8h9mmqd6mdb_04_tyql5p7r0000gn/T/opensearch-9604843300381789533, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/Users/lei.fu/work/RatuDB-3.0.0, -Dopensearch.path.conf=/Users/lei.fu/work/RatuDB-3.0.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true] [2023-10-27T16:26:39,534][WARN ][o.o.p.PluginsService ] [lei.local] Non-plugin file located in the plugins folder with the following name: [.DS_Store] [2023-10-27T16:26:40,900][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [lei.local] fatal error in thread [main], exiting java.lang.NoClassDefFoundError: org/opensearch/common/lifecycle/LifecycleListener at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?] at java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) ~[?:?] at java.lang.Class.getConstructors(Class.java:1943) ~[?:?] at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:765) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.node.Node.<init>(Node.java:455) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.node.Node.<init>(Node.java:372) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:215) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:206) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0.jar:3.0.0] at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:172) ~[RatuDB-3.0.0.jar:3.0.0] at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:117) ~[RatuDB-3.0.0.jar:3.0.0] Caused by: java.lang.ClassNotFoundException: org.opensearch.common.lifecycle.LifecycleListener at java.net.URLClassLoader.findClass(URLClassLoader.java:476) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:588) ~[?:?] at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?] ... 19 more fatal error in thread [main], exiting java.lang.NoClassDefFoundError: org/opensearch/common/lifecycle/LifecycleListener at java.base/java.lang.Class.getDeclaredConstructors0(Native Method) at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137) at java.base/java.lang.Class.getConstructors(Class.java:1943) at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:765) at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) at org.opensearch.node.Node.<init>(Node.java:455) at org.opensearch.node.Node.<init>(Node.java:372) at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:215) at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:206) at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) at org.opensearch.cli.Command.main(Command.java:101) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:172) at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:117) Caused by: java.lang.ClassNotFoundException: org.opensearch.common.lifecycle.LifecycleListener at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588) at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ... 19 more

I want to ask, org/opensearch/common/lifecycle/LifecycleListener, in opensearch jars, opensearch jar package in the job - the scheduler is how to build? Why is this class missing? I can't find a LifecycleListener class in the Opensearch source code.

[prudhvigodithi]

[Triage]
Hey @xiaodizi have you tried with the latest OpenSearch working 3.0.0 snapshot, for scheduler ./gradlew build should create a complied zip for installing in the cluster. Can you also try downloading https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/dist/opensearch/opensearch-3.0.0-linux-x64.tar.gz the tar which already have the compiled zip for job scheduler, coming from build https://build.ci.opensearch.org/job/distribution-build-opensearch/9664/.
Thanks
@bbarani