Update lexers

[rnpnr]

This should be just about ready to merge now. One thing I'm still not
sure about is the modifying of the default styles. I don't think it is
necessary to bloat up the lexers table with all the possible style tags
but there should still be some way of indicating to users how they can
add styles for missing tags. Let me know if someone has a better idea
than just copying from upstream and leaving them commented out.

As indicated by the commit authors this contains the work of a few
different people whom I pass on my thanks!

[mcepl]

Very hesitant +1 from me. Shouldn’t we first make some vivisection like what I suggested in https://paste.sr.ht/~mcepl/ef4aba41df3c516bc6a96ed1ee9643fed78e01d0 (and orbitalquark/scintillua#99 (comment)) so that at least we have something half-functional?

[rnpnr]

So right now any styles added with lex:add_style will not get
added. This applies to the following (excluding lexer.lua):

# grep 'add_style' lua/lexers/* | cut -d ':' -f 1 | uniq
antlr.lua
clojure.lua
crystal.lua
gleam.lua
icon.lua
jq.lua
julia.lua
lexer.lua
mediawiki.lua
meson.lua
moonscript.lua
rest.lua
txt2tags.lua

Which out of ~150 lexers isn't bad. You can still get styling for all
of these by adding extra entries to your theme. For example:

vis/lua/themes/dark-16.lua

Lines 132 to 144 in d37a20c

	-- reST
	lexers.STYLE_LITERAL_BLOCK = lexers.STYLE_EMBEDDED
	lexers.STYLE_FOOTNOTE_BLOCK = lexers.STYLE_LABEL
	lexers.STYLE_CITATION_BLOCK = lexers.STYLE_LABEL
	lexers.STYLE_LINK_BLOCK = lexers.STYLE_LABEL
	lexers.STYLE_CODE_BLOCK = lexers.STYLE_EMBEDDED
	lexers.STYLE_DIRECTIVE = lexers.STYLE_KEYWORD
	lexers.STYLE_SPHINX_DIRECTIVE = lexers.STYLE_KEYWORD
	lexers.STYLE_UNKNOWN_DIRECTIVE = lexers.STYLE_KEYWORD
	lexers.STYLE_SUBSTITUTION = lexers.STYLE_VARIABLE
	lexers.STYLE_INLINE_LITERAL = lexers.STYLE_EMBEDDED
	lexers.STYLE_ROLE = lexers.STYLE_CLASS
	lexers.STYLE_INTERPRETED = lexers.STYLE_STRING

So I don't really think this is half functional as you say.

[mcepl]

Yes, “half functional” is too harsh, but what I meant is that current state of that branch is a regression against the previous released version.

[TwoF1nger]

@rnpnr I think there's a redrawtime leftover in the man page.

[fischerling]

The markdown theming example is missing:

• STYLE_BOLD
• STYLE_ITALIC
• STYLE_LIST
• STYLE_LINK
• STYLE_REFERENCE
• STYLE_HEADING_H1 - STYLE_HEADING_H6

Here is what I use in my theme:

lexers.STYLE_BOLD = 'fore:default,back:default,bold'
lexers.STYLE_ITALIC = 'fore:default,back:default,italics'
lexers.STYLE_LIST = lexers.STYLE_KEYWORD
lexers.STYLE_LINK = lexers.STYLE_KEYWORD
lexers.STYLE_REFERENCE = lexers.STYLE_KEYWORD
lexers.STYLE_HR = 'fore:red'
for i=1, 6 do lexers['STYLE_HEADING_H'..i] = 'fore:red' end

[rnpnr]

Thanks! The STYLE_HEADING_H? are already in the txt2tags example but I can add your line to both. As for the others I missed them since they were defined elsewhere in the scintillua themes. They are also used in a couple other lexers so it might be useful to just define them by default.

By the way stuff like:

lexers.STYLE_BOLD = 'fore:default,back:default,bold'

is equivalent to:

lexers.STYLE_BOLD = 'bold'

[rnpnr]

Oops the txt2tags is STYLE_H? (which from what I see in the lexer is never actually used). Either way both can be replaced by the for loop.

[fischerling]

Oops the txt2tags is STYLE_H? (which from what I see in the lexer is never actually used). Either way both can be replaced by the for loop.

It is actually used at:

vis/lua/lexers/txt2tags.lua

Line 24 in b753520

return token('h' .. level, equal + plus) * header_label^-1

I have not confirmed it but looking at the code the tokens h1 up to h6 are defined.

[rnpnr]

@rnpnr I think there's a redrawtime leftover in the man page.

Good catch, I will fix it.

[rnpnr]

Do people think that the commented out examples should just be enabled by default? It would result in some small amount of extra runtime memory usage but should make it so this patchset doesn't remove the working existing styles.

[fischerling]

Do people think that the commented out examples should just be enabled by default? It would result in some small amount of extra runtime memory usage but should make it so this patchset doesn't remove the working existing styles.

I don't really care, because I use my own theme definitions.
But I think that people would definitely expect proper syntax highlighting (highlighting all tokens) for all supported file types out of the box.

[mcepl]

But I think that people would definitely expect proper syntax highlighting (highlighting all tokens) for all supported file types out of the box.

That +1 was for this, not for the first sentence.

[rnpnr]

Sounds good to me. In that case here is an idea: combine light-16.lua and dark-16.lua into a single file and replace all fore:white,back:black etc. with empty declarations to just use the default terminal foreground and background. This is what I do in my term.lua theme so that I can have a light or dark background depending on the time of day. This should also address some of the complaints about the default style not matching the terminal.

As for the zenburn.lua and solarized.lua I don't know much about them. So I can either try to reuse styles, as in lexers.STYLE_H1 = lexers.STYLE_HEADING, or just leave it for someone else to fix.

[mcepl]

Sounds a like a sound plan ;).

[mcepl]

You made a huge amount of work with reviving those themes. Respect and thank you!

Applied to my devel branch and after five minutes testing it seems to work almost perfectly.

We should also collect what changes (if any) we want to push upstream and somehow resolve orbitalquark/scintillua#99.

[rnpnr]

We should also collect what changes (if any) we want to push upstream

Actually everything here is upstream besides your change to search in the correct path.

and somehow resolve orbitalquark/scintillua#99.

Unfortunately I don't have any knowledge of reST so I don't know which add_styles() can be dropped and which need to stay. Some can probably be changed to the styles added for the other markup languages.

Sounds a like a sound plan ;).

I'm still working on this btw. There are some bugs in the logic for applying themes which is probably why no one else has attempted to merge the styles. Work in progress is in my selstyle branch. If I can figure it out I will apply it to master and then update this patch set accordingly.

[rnpnr]

I think I'm going to cherry-pick the following commits into master:
lua: drop redrawtime option
lua: filetype: use alt_name field to alias to other lexers
These are strictly bugfixes and shouldn't be contentious.

[mcepl]

On Tue Dec 5, 2023 at 8:25 PM CET, Randy Palamar wrote: I think I'm going to cherry-pick the following commits into master: [lua: drop redrawtime option](26b59c9) [lua: filetype: use alt_name field to alias to other lexers](33f6826) These are strictly bugfixes and shouldn't be contentious.

Certainly, go for it!

[mcepl]

On Wed Mar 27, 2024 at 1:06 PM CET, Randy Palamar wrote: Merged #1133 into master.

Thank you! Matěj P.S.: So, now only to merge tests, right? ;) -- http://matej.ceplovi.cz/blog/, @***@***.*** GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 If only there were evil people somewhere insidiously committing evil deeds, and it were necessary only to separate them from the rest of us and destroy them. But the line dividing good and evil cuts through the heart of every human being. And who is willing to destroy a piece of his own heart? -- Aleksandr Solzhenitsyn: The Gulag Archipelago