Gatling crashing because of: Failed to start the native thread for java.lang.Thread

[amra]

I have created a simulation for testing jdbc load on database to test Postgres cluster.

The test starts crashing because of problems with creating threads:

[110.738s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[110.738s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-3-thread-21677"

I am running the test on a virtual machine with plenty resources (8 CPU cores, 32GB RAM), Red Hat 9 and OpendJDK 17.0.12.
The test crashes when the process is using around 21k of threads and 1GB RAM.
The system can use up to 450k threads and limit for process is over 150k of threads.

Simulation

The simulation is pretty simple:

setUp(
        PgbenchSimulation.SCN.injectOpen(
            nothingFor(ofSeconds(5)),
            rampUsersPerSec(0).to(100).during(ofSeconds(20)),
            constantUsersPerSec(100).during(ofSeconds(10)),
            rampUsersPerSec(100).to(250).during(ofSeconds(10)),
            constantUsersPerSec(250).during(ofSeconds(10)),
            rampUsersPerSec(250).to(500).during(ofSeconds(20)),
            constantUsersPerSec(500).during(ofSeconds(10)),
            rampUsersPerSec(500).to(1000).during(ofSeconds(30)),
            constantUsersPerSec(1000).during(ofSeconds(10)),
            rampUsersPerSec(1000).to(1500).during(ofSeconds(30))
        )
    ).protocols(dataBase);

Log

...

================================================================================
2024-10-08 13:21:37 GMT                                     104s elapsed
---- Requests ------------------------------------------------------------------
> Global                                                   (OK=13063  KO=0     )
> pgbenchTransaction                                       (OK=13063  KO=0     )

---- Pgbench scenario ----------------------------------------------------------
[##########----------------                                                ] 14%
          waiting: 56672  / active: 19016  / done: 13062
================================================================================

[111.237s][warning][os,thread] Attempt to protect stack guard pages failed (0x00007ff2ccda3000-0x00007ff2ccda7000).
#
# A fatal error has been detected by the Java Runtime Environment:
# Native memory allocation (mprotect) failed to protect 16384 bytes. Error detail: memory to guard stack pages
# An error report file with more information is saved as:
# /root/postgres-ha/hs_err_pid3917.log
[111.238s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[111.238s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-3-thread-21667"
[111.239s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[111.240s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-3-thread-21668"
[111.240s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[111.241s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-3-thread-21669"
[111.241s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[111.242s][warning][os,thread] Failed to start the native thread for java.lang.Thread "pool-3-thread-21670"
[111.242s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
13:21:41.085 [gatling-1-14] ERROR o.g.g.jdbc.actions.DBRawQueryAction - Gatling crashed: j.l.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.base/java.lang.Thread.start0(Native Method)
        at java.base/java.lang.Thread.start(Thread.java:809)
        at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:945)
        at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
        at scala.concurrent.impl.ExecutionContextImpl.execute(ExecutionContextImpl.scala:21)
        at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:429)
        at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:338)
        at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:312)
        at scala.concurrent.impl.Promise$DefaultPromise.map(Promise.scala:182)
        at scala.concurrent.Future$.apply(Future.scala:687)
        at org.galaxio.gatling.jdbc.db.JDBCClient.connectionResource(JDBCClient.scala:46)
        at org.galaxio.gatling.jdbc.db.JDBCClient.statementResource(JDBCClient.scala:64)
        at org.galaxio.gatling.jdbc.db.JDBCClient.executeRaw(JDBCClient.scala:105)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.$anonfun$execute$4(DBRawQueryAction.scala:26)
        at scala.runtime.java8.JFunction1$mcVJ$sp.apply(JFunction1$mcVJ$sp.scala:18)
        at io.gatling.commons.validation.Success.map(Validation.scala:41)
        at io.gatling.commons.validation.Validation.map$mcJ$sp(Validation.scala:28)
        at io.gatling.commons.validation.Validation.map$mcJ$sp$(Validation.scala:28)
        at io.gatling.commons.validation.Success.map$mcJ$sp(Validation.scala:40)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.$anonfun$execute$3(DBRawQueryAction.scala:21)
        at io.gatling.commons.validation.Success.flatMap(Validation.scala:42)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.$anonfun$execute$2(DBRawQueryAction.scala:20)
        at io.gatling.commons.validation.Success.flatMap(Validation.scala:42)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.$anonfun$execute$1(DBRawQueryAction.scala:19)
        at io.gatling.commons.validation.Success.flatMap(Validation.scala:42)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.execute(DBRawQueryAction.scala:18)
        at io.gatling.core.action.Action.$bang(Action.scala:42)
        at io.gatling.core.action.Action.$bang$(Action.scala:39)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.io$gatling$core$action$ChainableAction$$super$$bang(DBRawQueryAction.scala:12)
        at io.gatling.core.action.ChainableAction.$bang(Action.scala:75)
        at io.gatling.core.action.ChainableAction.$bang$(Action.scala:73)
        at org.galaxio.gatling.jdbc.actions.DBRawQueryAction.$bang(DBRawQueryAction.scala:12)
        at io.gatling.core.controller.inject.Workload.startUser(Workload.scala:53)
        at io.gatling.core.controller.inject.Workload.$anonfun$injectUser$2(Workload.scala:66)
        at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:156)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:840)

log.txt
pom.xml.txt