API chaining POC

[mildwonkey]

POC: "templatizing" URL, headers, and parameters in the API domain with results from previous API calls.

[soltysh]

Nit: but I've noticed this being heavily used in lula codebased, but something that makes me 😿
how about rewriting this like so:

for i, request := range a.Spec.Requests {
    if i < 1 {
        // the first request cannot use outputs
        continue
    }
    if a.Spec.outputs != nil {
	    var err error
...

The benefit is that we're having less indented code which improves readability.

[mildwonkey]

I'm onboard with the idea of reducing indentation, but that won't work in this example - here I'm not continuing out of the entire loop, just skipping the outputs processing (those first 10 lines).

[soltysh]

Consider if you want to process all the request and gather []error and return that, or fail on the first error like you have here.

[mildwonkey]

That's the current design, and there's more (not yet implemented) nuance to the statement I'm about to make, but once we introduce templating we need to fail as soon as one API call fails, since the output of that call is the input of another. We cannot complete a chain of requests if a request in that chain fails.

I will say (here's the nuance) that I'd like refactor Lula so it has an idea of a request chain vs a single request, or some other way to identify while operations much happen first and succeed before the next can continue. This POC is meant to be the API chaining option that doesn't require a huge refactor, but I'm team refactor, myself.

[mildwonkey]

This is also on my list of arguments for a DSL : the go-cty type system has Unknown and Dynamic type placeholders, and I want that or something like it in Lula, so it's easier to look at a single API request and know that we're still waiting for placeholder values to be expanded. (we'd also need a dependency graph)(i know i'm just describing terraform, i'm sorry)

[soltysh]

Several ideas, take whatever you feel like makes sense:

1. I'd not introduce this intermediary in the external types at all. Even though it's a private field, it still feels like exposing part of internal code. I'd suggest creating an internal structure, if you need to pass the data between one data structure and the other.
2. I looked into the templating bits and I have similar concerns as you pointed out on slack, that in the long run this might cause pains. I might be missing some bits of information about the goal, but have you considered using jsonpath, which is already used in several places and can potentially provide a better escaping logic, and be easier to understand by users?

[mildwonkey]

That is basically the syntax I have for users to define what fields they want from the previous API calls (i'm using the kyaml library, for consistency with other parts of lula).

That doesn't help me with the user input template part of things, though - since we're "templatizing" strings (parameters, headers, the URL) I do need some kind of string-based templating syntax?