Output Refactor

[rgoldberg]

Output Refactoring Proposal

This output refactoring will resolve the following, or simplify resolving the following:

• JSON output option #190: JSON output.
• --debug option to show JSON responses, other debug info #317: JSON debug output.
• list output column misalignment #412: Tabular formatting via column command.
• Consistent display justification of app IDs #598: Tabular formatting via column command.
• Output improvements & options #104: Filter, order & reformat JSON output.
• info: retrieve additional metadata about apps #395: Filter, order & reformat JSON output.
• outdated/info: display release notes  #60: Filter, order & reformat JSON output.
• list: display purchaser Apple ID #63: Filter, order & reformat JSON output.
• zsh tab completion #222: JSON output can be used to generate completions for zsh.
• bash completion script #453: JSON output can be used to generate completions for bash.
• Parallelize operations #586: Easier to parallelize when all output streamed thread-safe.
• Support multiple app ID arguments for all relevant commands #225: Multiple errors / warnings / successes handled.

Apple Raw Data Types

1. JSON (ordering & spacing): e.g., iTunes Search API results
2. Non-JSON text (ordering & ignorable spacing): e.g., Spotlight format (not currently used)
3. Ordered non-text (ordering): e.g., Swift array
4. Unordered non-text: e.g., Swift object

Output Categories

1. Normal: expected output
2. Ephemeral: e.g., downloading & installing progress bars
3. Warning: nonfatal problems
4. Error: fatal problems

Exit Codes

Return 0 (success) unless 1 or more errors occur.

3 Output Formats

Tabular

mas will output tabular output by default. It will be an improved version of the existing output.

Raw JSON

Raw JSON is a JSON representation of Apple raw data that's as similar to the raw data as
possible.

Raw JSON output can be requested via the --raw-json flag.

All output categories (except possibly ephemeral) will be output as JSON.

Standard JSON

Standard JSON is a JSON representation of Apple raw data mapped to sensible mas-wide standards.

e.g.: appID would be used as the standard property key for the following equivalent properties
from different data sources:

• trackId from the iTunes Search API
• kMDItemAppStoreAdamID from Spotlight
• CKSoftwareProduct.itemIdentifier from CommerceKit
• …

Standard JSON output can be requested via the --json flag.

All output categories (except possibly ephemeral) will be output as JSON.

Messages

A message is a fragment of text representing the output of a self-contained event (like info for
an app, a list of installed apps, an invalid argument error list, an unknown app ID argument,
etc.)

Streaming

Messages will be streamed to the console; in both the JSON formats, each message will thus be
its own JSON object in a stream of JSON objects.

Invalid Data

If any Apple raw data is corrupt, and thus cannot be represented as valid JSON (like text that
should be JSON not being valid JSON), that data will be output as a properly escaped single JSON
string in an error message.

External-to-Swift Implementation

Shell Wrapper Script Around Swift Binary Executable

It is impossible to properly align tabular console data in Swift.

The column executable, however, can properly align tabular console data.

mas will be restructured to have a zsh wrapper script named mas that will perform tabular &
other formatting. mas will be in the executable search path. Users will normally interact with
the new mas zsh wrapper just like they currently interact with the existing Swift mas, except
the new mas will support new features, like outputting raw or standard JSON.

mas-json Swift Binary Executable

The existing mas Swift binary executable will be renamed to something like mas-json.
It will remain in the executable search path. It will be called by the new mas wrapper script.

All the output from mas-json will be in JSON (except possibly ephemeral output),
regardless of the format that the user has requested frommas.

Normal users will never directly interact with mas-json. They will only call mas,
which will forward all arguments to mas-json, then format the JSON messages that it receives.

One Output Stream per Output Category

Each output category will have its own dedicated stream to which its messages will be written:

1. Normal: stdout
2. Ephemeral: file descriptor 4 (suppressed when not a TTY)
3. Warning: file descriptor 3
4. Error: stderr

Separate streams disambiguate output categories & allow piping/formatting per category
(such formatting would be done by mas or a custom user script that calls mas-json). e.g.:

1. Normal: no formatting
2. Ephemeral: blue text
3. Warning: pink text
4. Error: red text

Formatting would be suppressed for any stream that isn't a TTY
(might provide a per-stream setting to retain formatting for non-TTY).

Formatting could be user-configurable via one environment variable per stream.

If file descriptor 3 or 4 have not been redirected, then mas will redirect them to stderr.
If either has been redirected, mas will not interfere with the redirection.

JSON Format Generation

mas-json will output raw JSON if it is called with the --raw-json flag, which mas will
pass through to mas-json.

mas-json will otherwise output standard JSON.

Duplicate Keys Within an Object

If a JSON object contains duplicate keys, most JSON parsers either fail or only output the
property for the last of the duplicate keys. mas will output all properties, including all
duplicate-keyed properties.

Spacing

Spacing options for mas-json & mas raw & standard JSON:

1. As per raw data: only when data is JSON: as raw as possible
2. Minified: most efficient
3. Pretty printed with one property / element / value per line: easiest to read. Indents:
   • tabs?
   • 2 spaces?
4. User-configurable via some standard pretty printing format syntax: flexible, but should
   spacing be handled by piping mas JSON output to other programs?

Tabular Format Generation

If using the default tabular output format, mas will generate tabular output from standard
(not raw) JSON returned by mas-json.

mas will use jq to generate tab-separated data from JSON, then column will be used to
align the data like:

mas-json list | jq -r '(.[] | [.id, .name, .version]) | @tsv' | column -ts $'\t' -R 1

Configuring Tabular Format Generation

Given that the JSON output will preserve all data, and that JSON can be parsed & tabulated
using jq & column, we could require that users who want anything other than the default
tabular output use one of the 2 JSON outputs and format it themselves.

If that is too cumbersome for users, we could support options / flags like:

--verbose
--all-fields
--fields <comma-separated list of field names>

This isn't necessary for the initial release of the Output Refactoring, but will be easy to
implement afterward.

Internal-to-Swift Implementation

When Messages Are Written

All messages will be written to the appropriate stream when they occur.

Exit Code

mas-json will have an exit code singleton variable which will default to 0. Utility
functions that write to stderr can optionally specify a non-zero value that will be bitwise
ORed with the existing exit code to produce a new combined exit code before writing to stderr.
stderr will be observed such that, if anything is written to it outside the utility functions,
the exit code will be bitwise ORed with 1 (or with some other power of 2).

Parallelism

If we allow parallel operations, then output (especially ephemeral output) could get jumbled. We
can discuss this in more detail later, after all existing & planned parallelism has been
identified.

Output Ordering

Raw JSON Output Ordering

Do not sort ordered raw data.

For unordered raw data, order as per Standard JSON.

Standard JSON Output Ordering

Keyed Data

By default, either sort standard JSON object properties alphabetically by key (retaining the
order of properties with the same key), or use some logical ordering (primary ID first, followed
by secondary IDs, etc., with the remaining properties sorted alphabetically by key). Properties
with duplicate keys would retain their relative order from the raw data.

Sequential Data

Never sort sequential data where the raw ordering matters (e.g., mas search).

Sort some specific top-level sequential data (e.g., mas list) in a logical manner.

Possibly alphabetically/numerically sort specified JSON arrays that only contain scalar elements
(i.e. numbers, strings, booleans, and nulls, but no arrays or objects).

Normally do not sort JSON arrays that are intended to contain arrays or objects.

Custom Ordering

If users want custom ordering, they can pipe output to other commands like sort. The initial
Output Refactoring needn't concern itself with custom ordering, even if we might support it in
the future, since it would be easy to graft custom ordering onto any Output Refactoring
implementation.

Location of ordering

Assuming reordering JSON in zsh is simple, mas-json should output all standard JSON ordered as
per the raw JSON data, while mas should reorder it for standard JSON or tabular output. If it
will be much more difficult to reorder in zsh than in Swift, then reordering of standard JSON
can be performed in Swift.

Configuration

If mas is to be heavily configurable, probably simplest to mainly use environment variables.

• All potential mas executables & scripts can easily read environment variables.
• Avoids creating & handling command-line flags, options, and/or arguments.
• Avoids reading config files from set locations or from locations from arguments, etc.
• Any files sourced before running mas executables or scripts can serve as config files.
• A command-line argument would override an equivalent environment variable.

Dependencies

The mas brew formula would depend on the jq & util-linux formulae, because jq doesn't come with macOS & column from macOS doesn't support right justification (at least the versions I've seen).