cfn-lint using all system memory.

[whoDoneItAgain]

cfn-lint version: (cfn-lint --version) 0.58.0 (windows python 3.9.9 x64)

Description of issue.

This is not an issue with the linting process itself, that works fine, but with the linter in general.

I have a fairly complex conditions block. It contains 24 iterations of the following, 6 equals with a FindInMap for the left compare and an AND containing a EQUALS with REF for the left compare and an OR referencing the 6 equals conditions already described before. There is also a different OR condition referencing 3 other OR conditions. each of the 3 other ORs reference, between them all, the 24 iterations first mentioned here.

In addition to the conditions section, the resource section has a resource for each of the 24 iteration conditions defined in the conditions section. Each of these are conditionally created. Of the resources property, one of them is a list that each list member has its own IF based on one of the 6 equals with FindInMaps

The problem I run into is cfn-lint will allocate so much system memory that my system will hand for 1-5 minutes and sometimes i have to hard reset it due to other processes not being able to get access to memory. We're talking 20GB to 52GB utilization; my system has 64GB. I find the higher utilizations occur when cfn-lint is called from from ATOM via the atom-cfn-lint plugin.

Is there a way to limit memory usage? If not, a feature request would be to add functionality to limit memory usage?

[PatMyron]

Do you mind pasting the template in plaintext? Tougher to copy-paste / reproduce with screenshots
(can redact any sensitive snippets / wrap it in a details HTML tag here if it's super long)

[whoDoneItAgain]

Couldn't get it to retain the formatting of my yaml, so its attached.

vpc-endpoints - Copy.yaml.txt

[andyvans]

Hi there. Is there any update on this? Is there a known workaround?

[andyvans]

I think I know what triggers this memory leak is when you have the VS Code git diff view open. Steps to reproduce:

• Make changes to a CF yaml file and close the editor tab
• Switch to the git section in VS Code
• double click on the file in the 'Changes' view to see the before and after diff.

[PatMyron]

linking aws-cloudformation/cfn-lint-visual-studio-code#222

[kddejong]

@whoDoneItAgain want to test with version v0.61.0 I think I got this reduced.

[whoDoneItAgain]

Looks like the issue. it runs MUCH quicker now. running directly through cli (windows) and atom both result in normal processing times and no excessive memory usage

[rheathcote-deloitte]

I'm experiencing this issue on an AWS workspace via VSCode - when we save a file, multiple cfn-lint processes kick off using around 400mb ram each. Leads to thrashing on the system. observed both on v0.61.0 and v0.64.1 installed via pip. What info can I give to help debug?

[SchnitzelKopf]

I'm experiencing this issue on an AWS workspace via VSCode - when we save a file, multiple cfn-lint processes kick off using around 400mb ram each. Leads to thrashing on the system. observed both on v0.61.0 and v0.64.1 installed via pip. What info can I give to help debug?

This is VERY annoying, as it is also happening on my local computer! Feels like every code change triggers another instance of cfn-lint. Or pressing CTRL + S. I am running Version 0.65.1

Edit + Workaround:
Seems like the VS Code Extension "CloudFormation Linter" is causing the described behavior. When switching to "Serverless IDE", which also offers linting via cfn-lint, linting won't be triggered constantly by typing.

[kddejong]

Do you have a cfn-lint config file file?

Validation is configured to only run on open and save and not on change. https://github.com/aws-cloudformation/cfn-lint-visual-studio-code/blob/master/server/src/handlers/validationHandler.ts#L38-L43

[kddejong]

I may have to add some debugging output to get more details but can you provide me the abbreviated Output from the plugin? We should see the execution for every file that was opened or saved.

[tomotway]

Do you have a cfn-lint config file file?

Validation is configured to only run on open and save and not on change. https://github.com/aws-cloudformation/cfn-lint-visual-studio-code/blob/master/server/src/handlers/validationHandler.ts#L38-L43

Hi, does this mean that using VSCode with "Auto Save" turned on can cause this issue?

[kddejong]

Possible. Let me try that out and see if I can get this replicated.

[kddejong]

My observation is that auto save does not make this situation worse. Auto save appears smart enough to know what files have been changed and save them. cfn-lint will only scan them at that time.

Also related to a bunch of files being open on launch of VSCode I have observed that files are only scanned when they are focused

[whoDoneItAgain]

this may have regressed in a recent release. currently using 0.77.3 (windows 11 x64 python3.10.11). running against a single file. after 8min, its consumed 60gb ram. cpu is not bad. only 5-6% the entire time.
vpc-endpoints - Copy.yaml.txt

[kddejong]

On it. Fixing another bug with SAM and conditions should have a release out this morning

[kddejong]

Released in v0.74.4. @whoDoneItAgain I'll let you close this to make sure we have it covered.

[whoDoneItAgain]

that did it @kddejong. works as expected again. thanks!