Dependency definitions to include inputs and outputs keys

[brokenmass]

Goals

The objective of this document is to propose a solution for dependency definition that empowers the users with the tools required to express their own task pipelines.

Previous discussions

Shallow dependsOn / after

Background

There are 4 different type of tasks that turborepo manage (or will manage soon)

Default dependency:

{
  "tasks": {
    "build": {
      "dependsOn": ["^build"] 
    }
  }
}

From documentation: https://turbo.build/repo/docs/crafting-your-repository/configuring-tasks#running-tasks-in-the-right-order

This is achieved using the standard 'dependsOn' field and is used to indicate that the task being evaluated must be run IF it's dependencies inputs have changed AFTER those dependencies have been evaluated and eventually executed.

Parallel dependency (virtual):

{
  "tasks": {
    "transit": {
      "dependsOn": ["^transit"]
    },
    "check-types": {
      "dependsOn": ["transit"]
    }
  }
}

From documentation: https://turbo.build/repo/docs/crafting-your-repository/configuring-tasks#dependent-tasks-that-can-be-ran-in-parallel

This is achieved using intermediate transit tasks is used to indicate that the task being evaluated must be run IF it's dependencies inputs have changed without waiting for the dependency execution to have completed.

Sibling dependency (internal only ):

{
  "tasks": {
    "svcA#dev": {
      "siblings": ["svcB#dev"] 
    }
  }
}

public configuration guessed from PR #9504

This is currently not publicly achievable but is used to indicate that the task being evaluated must also evaluate and eventually execute it's siblings without caring about their inputs or waiting for their completion.

After dependency (proposed):

{
  "tasks": {
    "buildApi": {
      "outputs": ["dist/"]
    },
    "publishApi": {
      "after": ["buildApi"],
      "inputs": ["dist/openapi.json"]
    }
  }
}

proposal being discussed on discord

This is currently not achievable but is used to indicate that the task being evaluated must be executed after another task ignoring it's actual hash. Typical use case is for when the input of the evaluated task depends on the output of the after task but not of it's inputs , usually when different inputs might lead to identical outputs.

Proposal

The proposal it to stop the proliferation of different configuration fields and normalise the above requirements into a standard, but flexible structure:

type Dependency = {
  // name of the task like `^build` or `lint`
  task: string,
  // either a boolean to indicates it depends on ALL/NONE of the inputs or an array of string (globs) to filter the task inputs when calculating the hash
  inputs: boolean | string[],
  // either a boolean to indicates it depends on ALL/NONE of the inputs or an array of string (globs) to filter the task outputs when calculating the hash
  outputs: boolean | string[],
}
type Task = {
  // other fields are not modified
  dependsOn?: (string | Dependency)[]
}

The previously defined depency types can be represented as:

Default dependency

{
  "tasks": {
    "build": {
      "dependsOn": ["^build"] 
    }
  }
}

or the equivalent

{
  "tasks": {
    "build": {
      "dependsOn": [{
        "task": "^build",
        "inputs": true,
        "outputs": true,
      }]
    }
  }
}

Parallel dependency
No longer needs a transit task:

{
  "tasks": {
    "check-types": {
      "dependsOn": [{
        "task": "^check-types",
        "inputs": true,
        "outputs": false
      }]
    }
  }
}

Sibling dependency

{
  "tasks": {
    "svcA#dev": {
      "dependsOn": [{
        "task": "svcB#dev",
        "inputs": false,
        "outputs": false
      }]
    }
  }
}

After dependency

{
  "tasks": {
    "buildApi": {
      "outputs": ["dist/"]
    },
    "publishApi": {
      "dependsOn": [{
        "task": "buildApi",
        "inputs": false,
        "outputs": ["/dist/openapi.json"]
      }]
    }
  }
}

This clarifies that:

• if the evaluated task depends on the dependency output then the task must be executed in sequence and the task cache is invalidated if those files have changed
• if the evaluated task depends on the dependency inputs the task cache is invalidated if those files have changes
• persistent task (that have no output as they do not terminate) can be depended upon IF and only outputs: false (instead of current design where you cannot depend on them at all).
• hashing the outputs of a task allow the code to be able to avoid having to copy the cache back in place if the current files are already up to date

In short this mean that a task execution no longer has a unique hash like now, but that it just holds hashes over its inputs and outputs and the final hash calculation is driven by the dependant requirements.

Other example

Other examples:

Efficient ts compilation

{
  "tasks": {
    "compile": {
      "inputs": ["src/", "package.json"],
      "outputs": ["dist/"],
      "dependsOn": [{
        "task": "^compile",
        "inputs": ["package.json"],
        "outputs": ["dist/**/*.d.ts"]
      }]
    }
  }
}

[anthonyshew]

This might be a hot take but I find things like after and siblings vastly easier to understand at a glance. 😄 It's notable, though, that your proposal has some extra control of hashing inputs/outputs to it.

I'll ask a question: If Turborepo's caching was module based (per-file) rather than package-based (per-package), would you still feel as strongly about this proposal? Particularly when comparing against after, siblings, etc.

[brokenmass]

The requirements can be summarised as :

Definition of effective task inputs should be configurable and separate (or possible to separate) from task topological order.

Current turborepo does not fulfill this requirement as if taskA has taskB in the dependsOn list then that implies BOTH topological order and cache inputs (taskA is executed if any input of taskB has changed).

NX instead achive the above using a different approach: dependsOn list only refer to task topological order and actual dependency must be explicitated in the inputs:

inputs: [production, ^production, { "dependentTasksOutputFiles": "**/*.d.ts" }]

(meaning the input of this task are all production file of this project and all production file of all the dependencies and the "**/*.d.ts" generated by any dependent task )

The above proposal is a way to explicitate such separation expanding on current turborepo desing, instead of subverting it.
If you can fulfill the same requirement with a different design (i guess similar to nx), then I would be interested in seeing it (and being involved in the conversation if possible)