Code Editor Syntax Highlighting Sucks

[Zooce]

Syntax highlighting in editors is a pain in the ass. The way to prove this to yourself is to make your own syntax highlighting color theme for multiple editors.

The Problem

The fundamental way syntax highlighting is handled in editors today is to pick a specification format (like TextMate) and then hope that color theme authors follow "roughly" the same token/scope assignments. Editors also try to rely a little bit on LSPs to provide "better" tokens/scopes for highlighting.

The solutions so far try to treat all file formats generically, but this just doesn't work well - there are concepts in programming languages that just don't exist in simpler text files -- this is even the case between some programming languages, and again, we're trying to get them all to conform to the same set of generic "tokens."

A Possible Solution

Create a standard similar to LSP that focuses specifically on syntax highlighting, an HSP (highlight server protocol). Why not just use LSP? Because LSPs are supposed to provide language specific functionality like autocomplete and finding references, not syntax highlighting. Same goes for the HSP, its purpose is to provide information specifically for highlighting, not for language features.

From the HSP Author Perspective

This HSP would produce a set of tokens and token names that the editor would use specifically for highlighting.

From the LSP Author Perspective

The LSP author wouldn't have to worry about syntax highlighting at all anymore, just focus on the actual features of the language.

From the Color Theme Author Perspective

The color theme author would have a well defined set of tokens and token names to use when adding support for a file format. No more guessing

From the Editor Author Perspective

Use the HSP's tokens and token names to highlight the syntax, obviously. But what if an HSP isn't available? Well I guess it's up to the editor - fallback to something else or maybe just require an HSP.