Random thoughts

[haydenflinner]

Hi, just some random thoughts, feel free to respond or not respond at your leisure 😄 I'll be using lnav some more in the next months and probably contributing back some easier 'getting started' stuff, if I can ever figure out the easiest powerful ways to use this tool 😄

• CHANGE: Timestamps that do not have a day or month are rewritten to a full timestamp like YYYY-MM-DD HH:MM:SS.

  • I noticed this locally actually, second time starting lnav. I am guessing it guesses the date based on filename, which in my case I don't trust the accuracy of. Iif my log starts at 10PM and the file is named 2022-04-05, it is actually 2022-04-04 10PM on the first line. Maybe the "date" portion of the timestamp should be colored somehow to indicate that it is synthetic, and/or a cmd to hide it when I know that the whole log spans less than a day and I know what day that is.
  • On that note, between the date not being a real part of the log line, and the white highlight at the beginning of the line, I can't easily copy-paste from my terminal into Slack without introducing funny characters in Slack. Not sure what to do about this, for now I'll just edit out the funny stuff and try adding a fix later.

• CHANGE: Removed the summary overlay at the bottom of the log view that displayed things like "Error rate" and the time span. It doesn't seem like anyone used it.

  • I used it 😄 Hopefully you are right and it wasn't a case of the silent majority haha. It was also great to watch it update to know how far into my log file we had loaded (my log files are in the GB).

And some thoughts on usability

• visidata is basically the golden standard here; it actually managed amazing beginner friendliness and intuitive-ness even before it had the "File Edit View ..." toolbar up in the top, maybe because all of its binds were stolen from vim. And has lots of documentation. Contrast to my first day in lnav, I somehow managed to crash it after adding a filter and trying to get my cursor back into the text view. I think I still have the crash log of this lol. The prompts leading me into the filter page were super helpful though, without those I may have given up on lnav.
• The custom parser format file examples look intimidating, and as a beginner lnav user I'm not exactly sure what benefits will come from writing a custom parser since my format is treated as plaintext. I suspect being able to filter out things based on their tag? Color highlighting capture groups for easier visual segmenting of log lines? Filling out those sqlite tables so I can build queries on top of those? To get started, I may actually build a "my-format-to-logfmt" converter, and tinker that way. And this may actually turn into my primary workflow rather than a custom format, just because i don't want to paste my work log lines into a public website.
• I spent around half an hour analyzing a log the other day, and at the end had some neat filters I wanted to preserve. It turned out the "write session to file" thing was .11+ and I was on .10, so I copied the file that persists between restarts. But then I wasn't sure how to use this file on the next logfile from this application. Visidata I think has a good standard here, you can tell it what format you want by name when starting the app, and you can configure regexes on filenames to select a certain parser. Maybe this is coming in .11 or beyond, but hoping for some more neat usability here.

[tstack]

I am guessing it guesses the date based on filename.

The modified time of the file is used as the "base" time for log messages that aren't fully specified. If the file is gzipped, the timestamp embedded in the header is used as the base time instead.

I can't easily copy-paste from my terminal into Slack without introducing funny characters in Slack.

Have you looked at the FAQ?

https://docs.lnav.org/en/latest/faq.html#q-how-can-i-copy-paste-without-decorations

Does that not work for you? In v0.11.0, there is new support for the OSC 52 escape sequence that integrates with the terminal so copying to the clipboard should even work over ssh.

It was also great to watch it update to know how far into my log file we had loaded (my log files are in the GB).

Ah, sorry. The file panel should give you some idea of how much has loaded. Can that be improved somehow?

Contrast to my first day in lnav, I somehow managed to crash it after adding a filter and trying to get my cursor back into the text view. I think I still have the crash log of this lol.

Apologies for the crash. If you are able to replicate it, I'd be interested to know if it still happens on the latest version (v0.11.0).

visidata is basically the golden standard here; it actually managed amazing beginner friendliness and intuitive-ness even before it had the "File Edit View ..." toolbar up in the top,

I think visidata is an impressive piece of work and I'm trying to lift things that are applicable in lnav. However, I think this must be a matter of personal preference, because I remember having a hard time with it. (I just tried it again and pressing Space while viewing a text file activates the command prompt instead of scrolling a page. Muscle memory for pagers is pretty strong, so deviating from the convention there is a bit irritating.)

The custom parser format file examples look intimidating

The regex101 integration is intended to streamline creating a format file. I think the error messages lnav reports for issues in formats are pretty extensive and helpful in getting things working. I can also help out in crafting the file if you run into trouble.

a beginner lnav user I'm not exactly sure what benefits will come from writing a custom parser

The most basic thing lnav needs is a timestamp. If there is a timestamp at the front of your log message, it will usually be picked up automatically by the generic_log format.

As for the benefits, the main one is collating log messages by time so that you can open multiple files and look at them in a single view instead of having them separated.

since my format is treated as plaintext.

Are you saying your log files show up in the "TEXT" view? Can you supply a sample of your log messages? If you don't want to post them publicly, you can send them to [email protected] and we can work it out over email.

so I copied the file that persists between restarts

You copied the view-info-* file from ~/.lnav/?

It turned out the "write session to file" thing was .11+ and I was on .10

There's a statically-linked linux binary attached to the release page:

https://github.com/tstack/lnav/releases/tag/v0.11.0-beta2

Or, if you're on MacOS, you can try running brew install --HEAD lnav

But then I wasn't sure how to use this file on the next logfile from this application.

The view-info-* file captures the session so that when you reload the same set of files, the state can be restored.
If you want something repeatable, you can create an lnav script that contains the commands to execute. Then, you can run it within lnav using the | prompt or from the command-line with the -f option. The :export-session-to command is writing out an lnav script.

Visidata I think has a good standard here, you can tell it what format you want by name when starting the app, and you can configure regexes on filenames to select a certain parser. Maybe this is coming in .11 or beyond, but hoping for some more neat usability here.

I'm not sure what you mean here. Are you talking about what log format to use with a given file? lnav will automatically figure out which log format is appropriate, it doesn't require any user intervention.

Thanks for your interest and for taking the time to write this up!