Allowing passing in multiple directories

[SharkyND]

Introduction

Currently, the --recursive flag only works with --filter if an exact file is mentioned. For example:

tflint --recursive --filter main.tf

But if we want to filter based on a certain subdir in a project folder, it doesn't work.

Working with --recursive and --chdir gives the following error:

➜ tflint --chdir='./test_tf_1/' --recursive
Failed to find workspaces; cannot use --recursive and --chdir at the same time

Proposal

For the following configuration: if we have two folders under a project folder:

|── test_tf_1
| └── main.tf
| └── .tflint.hcl
|── test_tf_2
│ └── main.tf
│ └── .tflint.hcl

We want to target .tf files in test_tf_1 only, how will be able to achieve that? In this case scenario, something like this could really help tflint --recursive --filter ./test_tf_1/.. or tflint --chdir='./test_tf_1' --recursive when running from the root of the project.

References

This request was initially raised as a bug issue first:

• "--recursive" and "--filter" options together not working as expected  #1904

[wata727]

It's probably possible to use --chdir and --recursive at the same time, but I couldn't think of a reason why you'd want to take advantage of it. In this case, cd test_tf_1; tflint --recursive should be appropriate.

Are there any issues that cd test_tf_1 doesn't solve?

[mkemmerz]

I am also interested in this feature but maybe I rephrase it a bit. The possibility to exclude a folder if using the --recursive flag would be great.

[bendrucker]

This is a related but distinct feature. It equally needs justification and details.

Do you want to exclude one directory or many? By explicit list or by pattern matching (glob, regex, etc.)?

TFLint cannot/should not offer infinite flexibility. Users always have the option to handle recursion themselves (using find etc.) and execute TFLint in each directory. Prior to --recursive, this was the only option.

To make the case for these features, we need more than a rough sketch of the command line interface. We need to hear why this is important and how the feature should work. Presenting a workaround for handling the recursion yourself with find would be a good way to illustrate these elements.

[SharkyND]

@bendrucker @wata727

Thanks for your replies, what I would like to do is: for example, I have three folders except for two (as state above):

.
|── test_tf_1
| └── main.tf
| └── .tflint.hcl
| └── .... <other sub files and dir with terraform code and .hcl file>
|── test_tf_2
│ └── main.tf
│ └── .tflint.hcl
| └── .... <other sub files and dir with terraform code and .hcl file>
|── test_tf_3
│ └── main.tf
│ └── .tflint.hcl

And now I want to run tflint for test_tf_1 and test_tf_2, and not for test_tf_3 with one command. So instead of cd into each directory, is there a way in which, from the root of the project, I can run something like:

tflint --chdir ./test_tf_1 --chdir ./test_tf_2 --recursive. The recursive flag will take care of not just the main.tf file but other terraform files on the same level or in nested directories

[SharkyND]

Either works. I think with --recursive which aims to target all, we can introduce a flag to exclude certain directories. Either a list or using this flag again and again like:

`.
|── test_tf_1
| └── main.tf
| └── .tflint.hcl
| └── .... <other sub files and dir with terraform code and .hcl file>
|── test_tf_2
│ └── main.tf
│ └── .tflint.hcl
| └── .... <other sub files and dir with terraform code and .hcl file>
|── test_tf_3
│ └── main.tf
│ └── .tflint.hcl
|── test_tf_4
│ └── main.tf
│ └── .tflint.hcl

`

tflint --recursive --remove ./test_tf_3 --remove ./test_tf_4

to exclude test_tf_3 and test_tf_4

[bendrucker]

Either works

Of course, we need you to describe your intent more than propose interface ideas. I'll repeat this note from above:

To make the case for these features, we need more than a rough sketch of the command line interface. We need to hear why this is important and how the feature should work. Presenting a workaround for handling the recursion yourself with find would be a good way to illustrate these elements.

It sounds like your goal is to exclude a few modules from the recursive inspection. Why do you want to exclude them? How do you decide which modules to exclude?

Without this background, we have no way to gauge how common or reasonable this need is and whether it's something we want to work on and/or would accept as a PR.

[SharkyND]

So for our terraform pipelines, we give our users the option to choose different directories they want to run tflint. For instance, rather than running tflint for all the terraform files, run the ones that were changed in a commit, to speed up the development process.
To speed it up further, rather than cd into individual dir and running tflint in a loop, it would be much easier and efficient to provide tflint a list of directories it should include in its run or a list of directories to exclude.

[bendrucker]

Ok great, this is helpful. Running only on changed directories would mean passing in an exact list of those directories. Using recursion wouldn't be appropriate since that would include unchanged child directories.

It would diverge from Terraform but it seems like the smallest change that would enable this would be to allow repeated --chdir flags and serially process them in the same way recursive mode does to the directories it finds via its walk.

You still have the problem of directories changing that aren't Terraform modules, eg .github. So you'd have to first filter to modules (at least one .tf or .tf.json file) as it doesn't seem appropriate for TFLint to ignore them.

[SharkyND]

Yeah makes sense, a simple check for if the dir contains the .tf or .tf.json and .tflint.hcl file (or using one .tflint.hcl) for all. Let me know if you need further use cases from our end, if you want to convert this into a Github issue, I would be happy to contribute.

[HristoDimitrovv]

Hello,

Our use case of tflint and why it will be helpful to have an option to ignore whole folders when running tflint with --recursive (if possible)

At my company we are managing around 100 repositories for different clients.
We have recreantly implemented a centralized uCI checks that is been executed on each repository on every PR.
Right now we are in the process of cleaning up all the repositories. Some of them have hundreds of tflint unused variables and data sources.

The reason why we need to ignore some folders is that the code there is from modules that are managed from different teams and having to clean all the checks from there (sometimes 50-60-100) would be an error prone process as they are related to CICD pipelines, Landing zones etc. and we are not aware if those unused declarations are left on purpose and will not be used in the future.

To manually enter 100 lines of comments in the code for each of the resources also sounds not right as it adds too many lines of unnecessary code to the repositories.
Also to use the find command or other workaround with playing with the command will not help us as the same commands for the uCI checks are used for each repository and we will need to exclude different folders in each of them.

It would be great if we can specify somewhere in the .tflint.hcl file for example in each repository which folder path we want to ignore. Or similar functionality with a file. That would help us a lot and I believe others will agree.

[wata727]

Thank you for sharing your use case.

I'm thinking about a solution to this problem, but currently I'd like to solve it with Configuration cascading or Recursive inspection config mentioned in #1929. What do you think about this idea?

[HristoDimitrovv]

Ok, thank you for the answer!
Will look on that and see if it will work for us.
But will be awaiting a more applicable solution that can work from a centralized place for the future.

[SharkyND]

Hi @bendrucker, circling back to this, Is it something I can contribute to? Would love to have this feature in tflint:

an option to ignore list of folder (or matching patterns) when running tflint with --recursive

[bendrucker]

A feature proposal is welcome and we can discuss that before moving to a PR. Repeated --chdir flags would be one approach.

[wata727]

Using --chdir is an idea, but it may seem a little strange that the meaning changes when combined with other flags.

Personally, it might be better to use the --filter option. The current --filter option, when combined with --chdir and --recursive, filters based on whether there is a match in the changed directory, which I feel is not intuitive. This behavior is due to the --filter option, which was originally introduced as an alternative to arguments #1638

So, returning to the original issue, I'm positive about changing the behavior of the --filter option as discussed here #1904
However, introducing breaking changes should be gradual and carefully.

[SharkyND]

Yeah @bendrucker and @wata727 I do agree with you but do we want to have a breaking change? I'm in favour of introducing another --ignore, which will ignore all the folders that we don't want to lint as described by @HristoDimitrovv in the post above. What do you guys think?

If that's okay with you, that's something I can work and open a PR?

[wata727]

I'm in favor of introducing a new flag like the --ignore, but my main concern is the inconsistency in that the --filter matches globs while --ignore matches directories. I believe these should be paired and the values should be interpreted the same way.

So, I guess what is needed is to first change the behavior of the --filter option, which is the opposite of the --ignore option, before introducing one that meets the requirements.

Also, the name is worth thinking about the --ignore is too general (for example, we already have --ignore-module flag) and we need a different name.