From 84c57cfad763c4409350435006d34822804eb682 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Sat, 29 Apr 2023 10:35:09 -0700 Subject: [PATCH 1/4] draft of how to enable shell integration with commands --- .../RFCXXXX-Command-Shell-Integration.md | 287 ++++++++++++++++++ 1 file changed, 287 insertions(+) create mode 100644 Draft-Accepted/RFCXXXX-Command-Shell-Integration.md diff --git a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md new file mode 100644 index 00000000..61de125f --- /dev/null +++ b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md @@ -0,0 +1,287 @@ +--- +RFC: RFCNNNN +Author: Steve Lee +Status: Draft +SupercededBy: NA +Version: 1.0 +Area: Interactive Shell +Comments Due: June 1, 2023 +Plan to implement: Yes +--- + +# Command Shell Integration + +Executables can provide a more integrated experience with a particular shell by indicating features it supports which the shell can leverage. + +Shell integration feature areas: + +- Structured output and formatting +- Tab completion +- Help +- Predictors (PowerShell) +- Feedback Providers (PowerShell) + +Traditionally, POSIX shells have pre-defined filesystem locations where tools would put their tab completion scripts or help content. +However, this typically works for the system and not for a particular user who doesn't have administrative rights to the system. +This can also make it more complicated to uninstall tools that place files across the filesystem. + +Although the examples here are focused on PowerShell, the design is intended to be generic and can be used by other shells. + +## Motivation + +As a tool developer, +I can easily integrate with a shell, +so that users are more productive having a consistent experience within that shell. + +## Current Experience + +Some tools that are executables and not modules/cmdlets have already reached out on how they can have better integration with PowerShell. +The current experience requires the user to know about the integration scripts and add them to their `$profile` so that it gets loaded. +This greatly decreases usage and thus motivation for tooling to produce integration scripts due to lack of discovery and also +manual effort by the user. + +## Enhanced User Experience + +In these examples, we'll use a hypothetical version of `az` (Azure CLI) that implements all of the shell integration features. +The user would simply install the tool and it would automatically be integrated with the shell. + +### Structured Output with Formatting + +```powershell +az group list +``` + +Current output: + +```output +[ + { + "id": "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/RG1", + "location": "westcentralus", + "managedBy": null, + "name": "RG1", + "properties": { + "provisioningState": "Succeeded" + }, + "tags": { + "RequestID": "1111" + }, + "type": "Microsoft.Resources/resourceGroups" + }, + { + "id": "/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/RG2", + "location": "westcentralus", + "managedBy": null, + "name": "RG2", + "properties": { + "provisioningState": "Succeeded" + }, + "tags": { + "RequestID": "1111" + }, + "type": "Microsoft.Resources/resourceGroups" + } +] +``` + +Although in PowerShell, this output can be piped to `ConvertFrom-Json`, it's an additional step for every `az` command execution. +It would be preferrable to have the tool output structured data in a format that PowerShell can automatically convert to an object +and used similarly to a cmdlet. + +Proposed output with custom formatting showing the most relevant information as well as a nested property: + +```output +id location name provisioningState +-- -------- ---- ---------- +/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/ERNetwork-PvtApp westcentralus RG1 Succeeded +/subscriptions/11111111-1111-1111-1111-111111111111/resourceGroups/ERNetwork-SVC westcentralus RG2 Succeeded +``` + +### Tab Completion + +```powershell +az g +``` + +This should result in: + +```powershell +az group +``` + +As the tab completer would automtically be discovered by PowerShell and used. In this example, there is only one subcommand starting with `g`. + +### Help + +```powershell +get-help az +``` + +The help content would preferrably be in markdown with advanced rendering available in the shell. +Normal help searches would also look within the help content defined by the tool. + +### Predictors and Feedback Providers + +PowerShell has an extensibility model for Predictors and Feedback Providers. +Typically, these are implemented as modules with a cmdlet to register them with the PowerShell engine. +In this case, the extension would still need to be managed code, but would be automatically discovered and loaded by the shell. +Since the user installed the tool, the expectation is they want to use it. + +Based on user feedback, there can be a configuration file to disable specific extensions. + +## Specification + +A JSON manifest file would be placed alongside the executable (or technically anywhere within `$env:PATH`) and discovered by the shell. +All paths are relative to the location of the manifest file and must use the forward slash path separator. + +### Command shell integration manifest + +The JSON manifest file name would be `.command.json`. +The `` portion is only used to differentiate between multiple manifests and does not need to match the executable name, +but it is recommended to do so. + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.json", + "executable": "az", + "version": "1.0.0" +} +``` + +The `$schema` includes the version of the manifest with the URL pointing to a published JSON schema. +The `executable` is the name of the executable that the manifest is for and expected to be found in `$env:PATH` +The `version` is the version of the manifest for the tool and only used for informational purposes. + +### Structured Output and Custom Formatting Registration + +Tools can indicate they can output [JSONLines](https://jsonlines.org/) by default and also provide custom formatting. + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", + "executable": "az", + "version": "1.0.0", + "structuredOutput": { + "default": "jsonlines" + } +} +``` + +If a tool indicates it can output structured data in a format supported by the shell, +the shell will set the env var `COMMAND_SHELL_STRUCTURED_OUTPUT` with the value `jsonlines` in this case +in the environment block of the process that is launched. + +Tools should check for this environment variable and output structured data in the format indicated if supported. + +If the tool calls other executables, it is responsible for removing or propagating the environment variable which +can affect the behavior of child processes. + +JSON output should include a `$typeNames` member which is an array of strings that indicate the type hierarchy of the object. +PowerShell would use this as the same as the `TypeNames` member of a PSObject for formatting. +Future enhancement would support a JSON defined formatting schema that could be used by other shells. + +### Tab Completion Registration + +Within the same manifest, a tool can indicate it supports tab completion. + +This can either be an executable with arguments. + +Arguments can be built-in variables to provide context for the tab completion. + +Currently supported variables: + +- `{commandLine}` - The full command line that the user has typed so far. +- `{cursorPosition}` - The current cursor position within the command line, starting at 0 index. + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", + "executable": "az", + "version": "1.0.0", + "tabCompletion": { + "command": { + "executable": "az", + "arguments": [ + "completion", + "powershell", + "--commandLine", + "{commandLine}", + "--cursorPosition", + "{cursorPosition}" + ] + } + } +} +``` + +or a shell specific script: + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", + "executable": "az", + "version": "1.0.0", + "tabCompletion": { + "script": { + "powershell": "az-completion.ps1", + "bash": "az-completion.sh", + "zsh": "az-completion.zsh" + } + } +} +``` + +Multiple shells can be specified and up to the shell to determine which one to use. +Both a `command` and `script` can be specified, but only one of each and also up to the shell to determine which one to use. + +### Help Registration + +The same manifest may define a path to the help content: + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", + "executable": "az", + "version": "1.0.0", + "help": { + "tags": [ + "azure", + "az" + ], + "path": "./help/az-help.md" + } +} +``` + +Optional `tags` can be specified to help with searching for help content. + +### Predictors and Feedback Providers Registration + +Specifically for PowerShell, a command may also want to optionally register a Predictor or Feedback Provider. + +```json +{ + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", + "executable": "az", + "version": "1.0.0", + "predictor": { + "powershell": "az-predictor.dll" + }, + "feedbackProvider": { + "powershell": "az-feedbackprovider.dll" + } +} +``` + +Similar to `tabCompletion`, alternate shells can choose to adopt their own mechanism for registering these extensions +under their shell specific key. + +## Alternate Proposals and Considerations + +Users may decide they don't want to use a tools shell integration feature by selectively disabling it. +This is shell specific and not part of this proposal. + +Given that JSON is typically deeply nested, it may be useful to have PowerShell render _list view_ as YAML instead of +the current flat format. +Additionally, have PowerShell detect and render YAML for a single object and a table for multiple objects, by default. +This would apply to all objects and not just those from a shell integrated tool. From 4a57790088456c0442ef959783afb26014fcdce5 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Sat, 29 Apr 2023 20:06:55 -0700 Subject: [PATCH 2/4] Update Draft-Accepted/RFCXXXX-Command-Shell-Integration.md Co-authored-by: Dongbo Wang --- Draft-Accepted/RFCXXXX-Command-Shell-Integration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md index 61de125f..2d2d69a9 100644 --- a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md +++ b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md @@ -176,7 +176,7 @@ Tools should check for this environment variable and output structured data in t If the tool calls other executables, it is responsible for removing or propagating the environment variable which can affect the behavior of child processes. -JSON output should include a `$typeNames` member which is an array of strings that indicate the type hierarchy of the object. +JSON output can optionally include a `$typeNames` member which is an array of strings that indicate the type hierarchy of the object. PowerShell would use this as the same as the `TypeNames` member of a PSObject for formatting. Future enhancement would support a JSON defined formatting schema that could be used by other shells. From 1a30452ada3d53db336419baac7d6d9a6c061953 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Sun, 30 Apr 2023 11:51:41 -0700 Subject: [PATCH 3/4] address Dongbo's feedback and support MSIX scenario --- .../RFCXXXX-Command-Shell-Integration.md | 53 ++++++++++++++++--- 1 file changed, 46 insertions(+), 7 deletions(-) diff --git a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md index 2d2d69a9..9c5c08f3 100644 --- a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md +++ b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md @@ -26,6 +26,7 @@ However, this typically works for the system and not for a particular user who d This can also make it more complicated to uninstall tools that place files across the filesystem. Although the examples here are focused on PowerShell, the design is intended to be generic and can be used by other shells. +However, PowerShell specific implementation details are included. ## Motivation @@ -131,26 +132,48 @@ Based on user feedback, there can be a configuration file to disable specific ex ## Specification +### Discovery + A JSON manifest file would be placed alongside the executable (or technically anywhere within `$env:PATH`) and discovered by the shell. All paths are relative to the location of the manifest file and must use the forward slash path separator. +On Unix systems tools are sometimes synlinked to `/usr/bin`, for example, and on Windows, MSIX installed CLI tools are reparse points +to a different location. +Shells should look for the manifest file in the target location of a symlink or reparse point in addition to within `$env:PATH`. + +In the case that multiple JSON manifests are for the same executable, the last one wins. +So if a user wants to override a manifest shipped with a tool, they can place their own in a location that is later in `$env:PATH`. + ### Command shell integration manifest -The JSON manifest file name would be `.command.json`. -The `` portion is only used to differentiate between multiple manifests and does not need to match the executable name, -but it is recommended to do so. +The JSON manifest file name would be `..command.json`. + +- `` should match the executable name without the extension. +- `` is the name of the author or organization that created the tool to avoid collisions. ```json { "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.json", "executable": "az", - "version": "1.0.0" + "version": "1.0.0", + "author": "Microsoft", + "description": "Azure CLI", + "copyright": "Copyright (c) Microsoft", + "license": "https://github.com/Azure/azure-cli/blob/dev/LICENSE", } ``` -The `$schema` includes the version of the manifest with the URL pointing to a published JSON schema. -The `executable` is the name of the executable that the manifest is for and expected to be found in `$env:PATH` -The `version` is the version of the manifest for the tool and only used for informational purposes. +The mandatory fields are: + +- `$schema` includes the version of the manifest with the URL pointing to a published JSON schema. +- `executable` is the name of the executable that the manifest is for and expected to be found in `$env:PATH` +- `version` is the version of the manifest for the tool and only used for informational purposes. + +The optional fields are: + +- `author` is the name of the author or organization that created the tool. +- `description` is a short description of the tool. +- `license` is the URL to the license for the tool. ### Structured Output and Custom Formatting Registration @@ -180,6 +203,11 @@ JSON output can optionally include a `$typeNames` member which is an array of st PowerShell would use this as the same as the `TypeNames` member of a PSObject for formatting. Future enhancement would support a JSON defined formatting schema that could be used by other shells. +Users may want to selectively disable structured output for a tool. +How this is done is shell specific. +For PowerShell, a user could set the environment variable `COMMAND_SHELL_STRUCTURED_OUTPUT` to `none` to disable structured output for all tools +and remove that environment variable to re-enable it. + ### Tab Completion Registration Within the same manifest, a tool can indicate it supports tab completion. @@ -234,6 +262,9 @@ or a shell specific script: Multiple shells can be specified and up to the shell to determine which one to use. Both a `command` and `script` can be specified, but only one of each and also up to the shell to determine which one to use. +In this PowerShell example, the `az-completion.ps1` would be called one time to register the argument completer. +A different shell may decide to call the script for each tab completion request. + ### Help Registration The same manifest may define a path to the help content: @@ -276,6 +307,14 @@ Specifically for PowerShell, a command may also want to optionally register a Pr Similar to `tabCompletion`, alternate shells can choose to adopt their own mechanism for registering these extensions under their shell specific key. +In the case of PowerShell, predictors and feedback providers are assemblies so the value is the path to the assembly +instead of a `.ps1` script. +To enable support for MSIX installed tools and not load the dll directly from their location, PowerShell will +copy the dll to the user's folder if it is not already there (SHA256 hash comparison) and load it from there: + +- On Unix, the dll will be copied to `$HOME/.local/share/powershell/Predictors` and `$HOME/.local/share/powershell/FeedbackProviders` +- On Windows, the dll will be copied to `$HOME\AppData\Local\Microsoft\PowerShell\Predictors` and `$HOME\AppData\Local\Microsoft\PowerShell\FeedbackProviders` + ## Alternate Proposals and Considerations Users may decide they don't want to use a tools shell integration feature by selectively disabling it. From 01f2153265f3d430648d228d38f33afa57630c0d Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Wed, 10 May 2023 17:31:26 -0700 Subject: [PATCH 4/4] resolve feedback --- .../RFCXXXX-Command-Shell-Integration.md | 58 ++++++++++++++++--- 1 file changed, 49 insertions(+), 9 deletions(-) diff --git a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md index 9c5c08f3..830a81ab 100644 --- a/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md +++ b/Draft-Accepted/RFCXXXX-Command-Shell-Integration.md @@ -44,7 +44,9 @@ manual effort by the user. ## Enhanced User Experience In these examples, we'll use a hypothetical version of `az` (Azure CLI) that implements all of the shell integration features. -The user would simply install the tool and it would automatically be integrated with the shell. +The user would simply install the tool and it would automatically be integrated with the shell if the session is interactive. + +PowerShell 7 would emit on startup any automatically loaded extensions unless `-NoLogo` is specified. ### Structured Output with Formatting @@ -85,9 +87,9 @@ Current output: ] ``` -Although in PowerShell, this output can be piped to `ConvertFrom-Json`, it's an additional step for every `az` command execution. +In PowerShell, this output can be piped to `ConvertFrom-Json` and it's an additional step for every `az` command execution. It would be preferrable to have the tool output structured data in a format that PowerShell can automatically convert to an object -and used similarly to a cmdlet. +and used similarly to a cmdlet but only if the `$typeNames` property is set (see below). Proposed output with custom formatting showing the most relevant information as well as a nested property: @@ -137,23 +139,25 @@ Based on user feedback, there can be a configuration file to disable specific ex A JSON manifest file would be placed alongside the executable (or technically anywhere within `$env:PATH`) and discovered by the shell. All paths are relative to the location of the manifest file and must use the forward slash path separator. +Additionally, to support Windows Apps, symlinks and reparse points should be followed from the exe to locate the manifest file. + On Unix systems tools are sometimes synlinked to `/usr/bin`, for example, and on Windows, MSIX installed CLI tools are reparse points to a different location. Shells should look for the manifest file in the target location of a symlink or reparse point in addition to within `$env:PATH`. -In the case that multiple JSON manifests are for the same executable, the last one wins. +In the case that multiple JSON manifests are for the same executable, the first one wins. So if a user wants to override a manifest shipped with a tool, they can place their own in a location that is later in `$env:PATH`. ### Command shell integration manifest -The JSON manifest file name would be `..command.json`. +The JSON manifest file name would be `..shellintegration.json`. - `` should match the executable name without the extension. - `` is the name of the author or organization that created the tool to avoid collisions. ```json { - "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.json", + "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", "executable": "az", "version": "1.0.0", "author": "Microsoft", @@ -200,14 +204,16 @@ If the tool calls other executables, it is responsible for removing or propagati can affect the behavior of child processes. JSON output can optionally include a `$typeNames` member which is an array of strings that indicate the type hierarchy of the object. -PowerShell would use this as the same as the `TypeNames` member of a PSObject for formatting. +PowerShell would convert this to the PSObject `TypeNames` member for formatting. Future enhancement would support a JSON defined formatting schema that could be used by other shells. -Users may want to selectively disable structured output for a tool. -How this is done is shell specific. +Users may want to selectively disable structured output for a tool for a session or for a specific command use. For PowerShell, a user could set the environment variable `COMMAND_SHELL_STRUCTURED_OUTPUT` to `none` to disable structured output for all tools and remove that environment variable to re-enable it. +Alternatively, if the users uses `Start-Process -Environment @{COMMAND_SHELL_STRUCTURED_OUTPUT='none'}` that would override PowerShell +from setting the environment variable for the process. + ### Tab Completion Registration Within the same manifest, a tool can indicate it supports tab completion. @@ -221,6 +227,8 @@ Currently supported variables: - `{commandLine}` - The full command line that the user has typed so far. - `{cursorPosition}` - The current cursor position within the command line, starting at 0 index. +A literal `{` would need to be escaped: `\{`. + ```json { "$schema": "https://schemas.microsoft.com/commandshell/2023/05/01/command.shellintegration.json", @@ -265,6 +273,8 @@ Both a `command` and `script` can be specified, but only one of each and also up In this PowerShell example, the `az-completion.ps1` would be called one time to register the argument completer. A different shell may decide to call the script for each tab completion request. +File paths are relative to the manifest file location. + ### Help Registration The same manifest may define a path to the help content: @@ -315,6 +325,36 @@ copy the dll to the user's folder if it is not already there (SHA256 hash compar - On Unix, the dll will be copied to `$HOME/.local/share/powershell/Predictors` and `$HOME/.local/share/powershell/FeedbackProviders` - On Windows, the dll will be copied to `$HOME\AppData\Local\Microsoft\PowerShell\Predictors` and `$HOME\AppData\Local\Microsoft\PowerShell\FeedbackProviders` +To avoid conflicts with other tools that have dlls with the same name, the dlls wil be in a subfolder named after the SHA256 hash of the original tool. +At this point, there is no automatic cleanup of these folders. + +If there are dependencies needed for the predictor and feedback provider dlls, they should be included in the same folder +using the [PowerShell 7 native dependency RID folders](https://learn.microsoft.com/en-us/powershell/scripting/dev-cross-plat/writing-portable-modules?view=powershell-7.3#dependency-on-native-libraries). + +### PowerShell opt-out of automatic registration of extensions + +Users may want to selectively disable the automatic registration of extensions for a tool. +PowerShell will lazy load extensions when they are first used. +A new automatic variable `$PSCommandShellExtensionsNoAutoload` will be an array of names of commands or modules that should not be automatically loaded. +There is no provision for selecting specific extensions for a command or module. + +### PowerShell Modules leveraging automatic extension loading + +Within the module manifest file, +there would be new members within `PrivateData` that work similarly to the command manifest file: + +```powershell +@{ + # Other members of the module manifest + PrivateData = @{ + PSExtensions = @{ + Predictor = 'az-predictor.dll' + FeedbackProvider = 'az-feedbackprovider.dll' + } + } +} +``` + ## Alternate Proposals and Considerations Users may decide they don't want to use a tools shell integration feature by selectively disabling it.