FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[rorymc]

Without any change on our backend servers which run this code in a container, this out-of-memory (OOM) exception on our alerting started happening on August 24, 2020 -- surpassing the 220 MB per container threshold, going up to almost 1 GB before being evicted:

• FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

We've upgraded the kin-node SDK, but it didn't have any effect.

We believe this was triggered via some backend update on the Kin Agora side, which didn't play nicely with what we have installed. We have not been able to trace what is causing the memory leak.

On the monitoring graphs, the containers exhibit classic sawtooth memory usage, which indicates the container is increasing in memory past the 220 MB threshold we had set up to 1 GB per container, until the container gets evicted by the cluster manager, and then a new instance is created. Before Aug 24, 2020 we did not see this behavior, as we've had this monitoring/alerting 220 MB memory usage threshold policy in place since 2019.

here is the yarn.lock which we have for the container:

"@kinecosystem/agora-api@^0.25.0":
  version "0.25.0"
  resolved "https://registry.yarnpkg.com/@kinecosystem/agora-api/-/agora-api-0.25.0.tgz#ce7a47fe293d725528be54051e8918c5ca1df0af"
  integrity sha512-qHReSX8QG2CT7I3znZQzxDv3KFIK+Zb4aMftfU5shNB0GNiJLclEbthckSKTzSd/5UX+eP5zfgc6WhU1N/CcUw==
  dependencies:
    "@types/google-protobuf" "^3.7.2"
    google-protobuf "^4.0.0-rc.2"
    grpc "^1.24.3"

"@kinecosystem/kin-sdk-v2@^0.3.5":
  version "0.3.11"
  resolved "https://registry.yarnpkg.com/@kinecosystem/kin-sdk-v2/-/kin-sdk-v2-0.3.11.tgz#cd363ee62116706a7e5b7382a87bfe17eae9d321"
  integrity sha512-5+U3IQkbfOzJ/nkUGM9cBugi+UIcpPHgm7YjrugMUabKQBceadhQGBTwoXnNEa8ppyb1zO0HyGQfhWCsq23EVw==
  dependencies:
    "@grpc/grpc-js" "^1.1.2"
    "@kinecosystem/agora-api" "^0.25.0"
    "@solana/web3.js" "^0.86.3"
    "@types/bs58" "^4.0.1"
    "@types/express" "^4.17.7"
    "@types/google-protobuf" "^3.7.2"
    "@types/lru-cache" "^5.1.0"
    "@types/uuid" "^8.3.0"
    bignumber.js "^9.0.0"
    bs58 "^4.0.1"
    bytebuffer "^5.0.1"
    express "^4.17.1"
    google-protobuf "^4.0.0-rc.2"
    grpc "^1.24.3"
    hash.js "^1.1.7"
    lru-cache "^6.0.0"
    protobufjs "^6.10.1"
    stellar-base "3.0.3"
    tweetnacl "^1.0.3"
    uuid "^8.3.2"