decoder: Enable completion when turning invalid config into valid

[radeksimko]

Background

Currently when the user attempts completion (either by simply typing or explicitly via Ctrl+Space or equivalent hotkey) and we cannot offer any candidates, we often times treat the completion list as complete.

Specifically, in LSP terms we return isIncomplete: false along with empty list of candidates.

In reality though this is only appropriate if we have confidence in that empty list. There can be situations we cannot know until the user continues to type and makes the configuration valid, which in turn makes it possible for us to provide candidates.

These candidates are currently ignored because the isIncomplete: false effectively instructs the clients to cache that empty list and avoid asking for candidates again in that context. The LSP doesn't go into detail about what is considered context worth caching/invalidating but anecdotally it impacts completion within the same expression.

Example:

default = {
  foo = "foo-${}"
}

^ this is considered invalid configuration, to the point that we're unable to calculate any candidates at all as the whole LOC beginning with foo = ... is ignored.

Even after the user types a single character, making it valid (at least in HCL's terms), e.g.

default = {
  foo = "foo-${v}"
}

the client does not ask for completion again.

While the question of partial AST returned for the first example is to be addressed in hashicorp/hcl#641 we could dampen the impact of that on our end by still enabling completion in the second example. i.e. we may not be able to offer completion on the first keystroke but at least on the second one.

Proposal

Consider avoiding IsComplete: true as a default state here and maybe consider updating the expression interface to also include IsComplete, i.e. return lang.Candidates instead of []lang.Candidate here.