feat: add Wait to wait for expected output #257

mastercactapus · 2023-03-12T15:00:55Z

This PR adds two new commands:

MatchLine that takes a regular expression and waits until the current line matches
MatchScreen that takes a regular expression and waits until the "screen" matches

The purpose is to allow waiting for things like network or serial devices that might have more variable delays while using VHS.

MatchLine example

Output "out.gif"

Type "sudo apt update" Enter
MatchLine "password"

Type "hunter2" Enter # fake password :D
MatchLine "^>"

Type "sudo apt dist-upgrade" Enter
MatchLine "^>"

Sleep 1

MatchScreen example

Output "out.gif"

Set Width 1280
Set Height 720

Type "telnet telehack.com" Enter
MatchScreen "command list"

Type "starwars" Enter
MatchScreen "A long time ago"

Ctrl+C
MatchLine "^\."

Type "exit" Enter
MatchLine "^>"

Sleep 1

Additional Info

A few string(s[0]) + strings.ToLower(s[1:]) places were changed to a toCamel helper with unit test (fixes issue with MUTLI_WORD commands formatting)
A .Buffer() method was added to *VHS to allow getting the current buffer as text (screen)
A .CurrentLine() method was added to *VHS to allow getting the current line as text
.SaveOutput() was updated to use the shared method (largely the same code as before)
Possibly closes Wait for command to finish before continuing #70

rbergmanaf · 2023-05-16T19:29:04Z

I've been experimenting with this and it's a useful feature, thanks for creating this!

I've noticed though that it would benefit from a timeout option in the event the expected pattern isn't encountered?

mastercactapus · 2023-05-24T13:44:59Z

@rbergmanaf Yeah, I could see using that; should it be optional or required? If optional, should it have a default? It seems odd to have something hang forever by default. Something short, like 3 seconds, and then you can manually raise it for long commands.

If the default is shorter, then it's more likely a user will put their value in for longer-running commands -- the last thing we want is a script that takes a long time failing halfway through because the user didn't realize there was a 30-second timeout or something.

MatchLine <regex> [timeout]

I considered using @ like Type (e.g., MatchLine@1s), but it doesn't read well.

Alternatively, since it is a waiting command, we could just make it mandatory.

rbergmanaf · 2023-05-24T14:08:35Z

I don't feel strongly, but do like the idea of a short default. @ would probably be the least surprising given the precedent in Type, but as long as it's documented I imagine either way being acceptable personally. Maybe the charm team would have a preference?

maaslalani · 2023-05-24T15:27:43Z

I agree on the time out being a great option, I agree on the default of 3 seconds as well having the user manually bump it if they know it will be longer.

Additionally, I would probably have this renamed to Wait (possibly WaitFor / WaitUntil) so it reads a bit like english commands:

Type 'echo "Hello"'
Enter
Wait /hello/ 5s

maaslalani

This is great! I'm really excited about this feature, just a bit concerned about the naming. I don't know if MatchLine and MatchScreen are the best names.

I also think waiting for the prompt i.e. MatchLine "^>" will be a very popular incantation of the command. I wonder if it's worth having a Wait command that simply Sleeps until the prompt shows up.

maaslalani · 2023-05-24T15:28:29Z

token.go

 	}
 	return string(t)
 }

+func toCamel(s string) string {


Thanks for this extraction ❤️

maaslalani · 2023-05-24T15:29:07Z

testing.go

+
+// CurrentLine returns the current line from the buffer.
+func (v *VHS) CurrentLine() (string, error) {
+	buf, err := v.Page.Eval("() => term.buffer.active.getLine(term.buffer.active.cursorY+term.buffer.active.viewportY).translateToString().trimEnd()")


Nice, this is great! 👌

maaslalani

This is great! I'm really excited about this feature, just a bit concerned about the naming. I don't know if MatchLine and MatchScreen are the best names.

I also think waiting for the prompt i.e. MatchLine "^>" will be a very popular incantation of the command. I wonder if it's worth having a Wait command that simply Sleeps until the prompt shows up.

rbergmanaf · 2023-05-24T19:58:04Z

A simplified command like Wait that just waits for the next prompt to appear without needing an explicit expression for that one case does seem really convenient. It's the main way we are using it at this point, though we have some other cases that benefit from also having the ability to customize the regex.

mastercactapus · 2023-05-24T22:05:18Z

I like how Wait reads as well as implies what it does -- waits for a thing. Follow the what rather than the how.

I think WaitForLine is pretty clear, not sure if WaitForScreen reads as well, though. What about WaitOnScreen?

As for the common case of waiting for prompts, we could potentially do a Wait variation that uses WaitForLine but with a Set-able PROMPT variable or something like that with a sane default. Line & Screen could require a pattern, and Wait would not accept one to prevent the overlap.

maaslalani · 2023-05-24T22:12:59Z

I would propose something like the following:

Wait[+Context][@timeout] [regex]

Wait for prompt

Wait # simply wait for the prompt to appear (default timeout of 5 seconds)

Wait with timeout.

Wait@3m # wait for a maximum of 3 minutes for the prompt to appear

Wait with regex, we can decided whether the default is a screen or line.

Wait /hello/ # wait for hello to appear

Wait with context

Wait+Screen /hello/ # wait for hello to appear on the screen.
Wait+Line /hello/ # wait for hello to appear on the line.

Wait with all options:

Wait+Screen@1m /hello/

What are your thoughts? I think this syntax is nicer since it limits the commands but I'm very open to having different API that is potentially nicer?

mastercactapus · 2023-05-24T22:21:32Z

Context is a nice idea, is that used elsewhere or would this be a new syntax?

As for defaulting, I think we should set the defaults for our "wait for prompt" use case.

Set WaitPattern />$/ # default wait pattern
Set WaitTimeout 5s   # default wait timeout

Wait # can be used in most cases to wait for the next prompt
Wait+Line # identical to the above
Wait@1m # for longer-running commands
Wait /custom-pattern/

Wait+Screen /dialog title/ # match anywhere on the screen

maaslalani · 2023-05-24T22:23:27Z

As for defaulting, I think we should set the defaults for our "wait for prompt" use case.

Yup, sounds good!

Context is a nice idea, is that used elsewhere or would this be a new syntax?

This + syntax is used with the Ctrl and Alt commands, i.e.:

Ctrl+L
Alt+Enter

In terms of implementation, the Command would be Wait and the context would be the Options.

mastercactapus · 2023-05-24T22:30:43Z

Okay, that all sounds reasonable, and thanks for the insight on context.

The last bit to clarify is regex parsing -- we can probably piggyback on readString and pass / as the endChar with a new token type REGEX to keep things clear.

Does all sound good?

maaslalani · 2023-05-25T01:06:45Z

The new REGEX token sounds great!

mastercactapus · 2023-05-25T01:09:05Z

Excellent, I'll switch this PR to a draft until I'm able to get those changes pushed up for review

maaslalani · 2024-06-05T18:23:41Z

Hey @fenio, sorry for the delayed response. To test out this branch you would do something like:

brew install go

export GO_HOME=$HOME/.config/go
export GO_BIN=$HOME/.config/go/bin
go install github.com/mastercactapus/vhs@match

and then run vhs from the GO_BIN or add it to the $PATH.

$HOME/.config/go/bin/vhs wait.tape

fenio · 2024-06-06T06:55:14Z

@maaslalani thanks for sharing info how to build it but it fails this way with:

❯ ~/demo go install github.com/mastercactapus/vhs@match
go: downloading github.com/mastercactapus/vhs v0.0.0-20240531212618-997662d08c11
go: github.com/mastercactapus/vhs@match: version constraints conflict:
	github.com/mastercactapus/[email protected]: parsing go.mod:
	module declares its path as: github.com/charmbracelet/vhs
	        but was required as: github.com/mastercactapus/vhs

fenio · 2024-06-06T07:35:39Z

@maaslalani
Ok nevermind. I've just cloned your repo and built vhs locally.
Works lovely!

I just had to set WaitTimeout to 30s cause with default it was not enough.
Other than that it works perfectly fine and my demo is now 41s long instead of 49s before and I no longer have to blindly guess sleep values.

I LOVE IT ;)

fenio · 2024-06-06T10:22:06Z

It's just 8s shorter but (not sure why) size dropped from around 1,05MB to 350kB which is significant amount.
For my other demo time dropped from 50s to 45s and size from 387kB to 360kB.
Lovely results.
Final demos can be seen here: https://github.com/fenio/pv-mounter

Please merge it as I already updated my tape files ;)

maaslalani · 2024-06-06T15:30:10Z

Yeah the 5s wait timeout also caught me off guard. I think it should probably be either 15 or 30 seconds by default.

fenio · 2024-06-06T15:48:13Z

Definitely. If someone tries to find something that will allow to work with commands that finish in unpredictable time then it's probably way more than 5 seconds ;)
I'd say 30s as default should fit most of the use cases.
And maybe a bit more clear statement that it timed out. I didn't write down how exactly it fails in case it exceeds timeout but it definitely wasn't clearly communicated and probably should have been.
Anyway even with default set to 0.1s it's still something I want to be included in main branch ;)
This is just killer feature.

mastercactapus · 2024-06-06T18:32:52Z

I updated the default to 15s for now, though I'm happy to adjust it if it should be higher. I expect use cases to vary a lot, so I'm not sure of the best value.

fenio · 2024-06-19T16:00:54Z

So how about merge? What are the blockers for it?

3v1n0 · 2024-06-19T22:50:32Z

Maybe documentation should be updated too?

gmile · 2024-07-21T17:35:41Z

FWIW, I also cloned astercactapus:match branch and ran the following tape (found earlier in this thread) - it worked perfectly:

# Where should we write the GIF?
Output demo.gif
Set Shell fish

# Set up a 1200x600 terminal with 46px font.
Set FontSize 25
Set Width 1500
Set Height 1000
Set BorderRadius 10

Type "secator x httpx testphp.vulnweb.com"
Sleep 500ms
Enter
Wait

kyletaylored · 2024-07-31T21:21:34Z

I just tried running this locally and ran into this error (slightly modified the test script to run traceroute instead)

➜  vhs git:(adaeead) ./vhs ../example.tape                                                  
File: ../example.tape
Output .gif demo.gif
Set FontSize 25
Set Width 1500
Set Height 1000
Set BorderRadius 10
Type traceroute testphp.vulnweb.com
Sleep 500ms
Enter 1
Wait Line
failed to execute command: timeout waiting for "Line" to match >$; last value was:  6  * ae1.3515.edge1.dallas2.net.lumen.tech (4.69.209.114)  5.908 ms
recording failed

I ran a different tape, got a similar error:

failed to execute command: timeout waiting for "Line" to match >$; last value was:

davixcky · 2024-08-20T15:47:48Z

Hey I was really curious about this feature, any update?

testing.go

meowgorithm · 2024-10-02T13:59:04Z

Alright, so let's get this one done. I think we're there in terms of API and it should just be a matter of vetting the execution. Thanks for your patience with this one, everyone.

…sts and actually run them in race mode (#560) Since we rely a lot on sleeping on the integration tests, due to the fact we can't yet go with charmbracelet/vhs#257, we need to be able to tune this value depending on the context we're running the tests. In general sanitizers slows down the runtime quite a bit (especially the thread one), so make possible to use dynamic sleep times values depending on variables that we adjust depending on the context we're running in. Tests can now slowed up/down using `AUTHD_TESTS_SLEEP_MULTIPLIER` variable too (this can be used for helping local testing too, e.g. personally I can reliably run the tests faster in my machine - with no failures - using `AUTHD_TESTS_SLEEP_MULTIPLIER=0.3`; but also it helps to quickly "fix" slower machines). Tune the tape files so that we don't miss some changing contents (as in the MFA/QrCode tests) and make the example broker to be a bit lazier. Then, I noticed that we were not actually running the integration tests in `-race` mode, so fix this. Indeed this implies slower tests, but at least now we're fully checking for races both the daemon and the client. This worked fine in various builds both in my fork and in a private repo fork I did where the builders are way slower than the public ones. UDENG-4793

Co-authored-by: Christian Rocha <[email protected]>

caarlos0 · 2024-10-07T16:48:34Z

code-wise this lgtm! thanks everyone for the effort in this! ❤️

that said, one thing is that waiting on a line will not always work, e.g.

Type "sleep 5; echo hello"
Enter
Wait+Line@10s /hello/

This might fail because at the time the line was check it was already just the prompt (>).

I guess there's no way around it though...

But I thought this would work:

Type "sleep 2; echo hello; sleep 1"
Enter
Wait+Line@10s /hello/

and i still got

failed to execute command: timeout waiting for "Line hello" to match hello; last value was: >

maybe we should get the last line that isn't the prompt?

mastercactapus · 2024-10-07T21:34:31Z

@caarlos0 Wait (Line variant) looks for a "ready" indicator prefix on the current line, as determined by the cursor:

echo prints "hello" followed by a newline, so the current line never contains "hello".
Wait is intended to detect command completion, not specific output, and just checks behind the cursor.

Solutions for your examples:

Use printf instead of echo to avoid the newline.
Use the multiline variant to search the entire buffer: Wait+Screen@10s /hello\n/

Both of these should also work over serial or SSH sessions.

While potentially confusing, the current behavior serves the intended purpose of detecting when execution has finished (as best as possible). Alternatives like line-by-line processing of the buffer could introduce instability with large outputs since we can only scan what fits in the buffer (e.g., logs, large files), so we'll need to make a tradeoff somewhere on complexity/behavior. Determining if we're checking the same line twice would also be challenging, especially because the buffer can scroll as new lines are fed in.

Conversely, tools like expect (which inspired this idea) can be deterministic and behave as you're expecting (no pun intended); however, that's only possible because they operate directly on a stream (i.e., a child process). Our options here are limited by operating through a "looking glass" to ttyd.

caarlos0 · 2024-10-08T12:20:57Z

right, yeah, that makes sense!

cool, so, IMO, we can merge this.

@meowgorithm any additional remarks?

meowgorithm · 2024-10-09T14:03:14Z

LGTM. Let’s get it done.

caarlos0 · 2024-10-09T17:08:58Z

thanks everyone! <3

We were relying on sleeping when running integration tests tapes, but this is quite unreliable because: - Depends on the speed of the machine running the tests - Using generous sleeping times implies slower tests (this saves around 10m on CI tests...) So, use `Wait` commands from VHS charmbracelet/vhs#257 to only send input once the UI has the expected contents. Added some `Wait+$Extensions` to make regular expressions easier to maintain, and reducing complexity of the tape files. Fix some potential races that I've found while testing this. Based on #641 UDENG-4964

mastercactapus added 6 commits November 26, 2022 15:15

add Match and MatchAny commands

bb16d08

allow changing Match value

476a839

switch to MatchLine and MatchScreen

4a41562

Merge branch 'main' into match

b741ff8

remove leftover code

78c0e37

Merge remote-tracking branch 'origin/main' into match

4d1b73d

Merge remote-tracking branch 'origin/main' into match

53ed73a

maaslalani reviewed May 24, 2023

View reviewed changes

mastercactapus marked this pull request as draft May 25, 2023 01:09

mastercactapus added 8 commits May 24, 2023 20:17

add wait and regex to lexer

7ae1fbb

add parseWait

4213ef3

Add wait option defaults

ffb4024

add execute for wait

d6216c2

allow setting WAitTimeout and WaitPattern

73a4c20

remove MATCH_LINE and MATCH_SCREEN

8d0302f

add description to Buffer method

5b60921

add wait to parser test and fix setting parsing

944d605

set default wait timeout to 15s

adaeead

3v1n0 mentioned this pull request Sep 30, 2024

fix(pam/integration-tests): Improve reliability of PAM integration tests and actually run them in race mode ubuntu/authd#560

Merged

meowgorithm reviewed Oct 2, 2024

View reviewed changes

testing.go Outdated Show resolved Hide resolved

Update testing.go

2d9e9eb

Co-authored-by: Christian Rocha <[email protected]>

caarlos0 approved these changes Oct 7, 2024

View reviewed changes

caarlos0 merged commit 9624cda into charmbracelet:main Oct 9, 2024
8 checks passed

mastercactapus deleted the match branch October 9, 2024 18:37

b-per mentioned this pull request Nov 22, 2024

New release to include #257 #550

Open

3v1n0 mentioned this pull request Nov 29, 2024

feat(pam/integration-tests): Use VHS Wait command on test tapes ubuntu/authd#661

Merged

pmonks mentioned this pull request Jan 5, 2025

Instead of Sleep'ing, can the tape 'auto-advance' upon completion of the previous command? #537

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Wait to wait for expected output #257

feat: add Wait to wait for expected output #257

mastercactapus commented Mar 12, 2023

rbergmanaf commented May 16, 2023

mastercactapus commented May 24, 2023 •

edited

Loading

rbergmanaf commented May 24, 2023

maaslalani commented May 24, 2023

maaslalani left a comment

maaslalani May 24, 2023

maaslalani May 24, 2023

maaslalani left a comment

rbergmanaf commented May 24, 2023

mastercactapus commented May 24, 2023

maaslalani commented May 24, 2023 •

edited

Loading

mastercactapus commented May 24, 2023 •

edited by maaslalani

Loading

maaslalani commented May 24, 2023 •

edited

Loading

mastercactapus commented May 24, 2023

maaslalani commented May 25, 2023

mastercactapus commented May 25, 2023

maaslalani commented Jun 5, 2024

fenio commented Jun 6, 2024

fenio commented Jun 6, 2024

fenio commented Jun 6, 2024

maaslalani commented Jun 6, 2024

fenio commented Jun 6, 2024

mastercactapus commented Jun 6, 2024

fenio commented Jun 19, 2024

3v1n0 commented Jun 19, 2024

gmile commented Jul 21, 2024 •

edited

Loading

kyletaylored commented Jul 31, 2024

davixcky commented Aug 20, 2024

meowgorithm commented Oct 2, 2024 •

edited

Loading

caarlos0 commented Oct 7, 2024

mastercactapus commented Oct 7, 2024 •

edited

Loading

caarlos0 commented Oct 8, 2024

meowgorithm commented Oct 9, 2024

caarlos0 commented Oct 9, 2024

feat: add Wait to wait for expected output #257

feat: add Wait to wait for expected output #257

Conversation

mastercactapus commented Mar 12, 2023

rbergmanaf commented May 16, 2023

mastercactapus commented May 24, 2023 • edited Loading

rbergmanaf commented May 24, 2023

maaslalani commented May 24, 2023

maaslalani left a comment

Choose a reason for hiding this comment

maaslalani May 24, 2023

Choose a reason for hiding this comment

maaslalani May 24, 2023

Choose a reason for hiding this comment

maaslalani left a comment

Choose a reason for hiding this comment

rbergmanaf commented May 24, 2023

mastercactapus commented May 24, 2023

maaslalani commented May 24, 2023 • edited Loading

mastercactapus commented May 24, 2023 • edited by maaslalani Loading

maaslalani commented May 24, 2023 • edited Loading

mastercactapus commented May 24, 2023

maaslalani commented May 25, 2023

mastercactapus commented May 25, 2023

maaslalani commented Jun 5, 2024

fenio commented Jun 6, 2024

fenio commented Jun 6, 2024

fenio commented Jun 6, 2024

maaslalani commented Jun 6, 2024

fenio commented Jun 6, 2024

mastercactapus commented Jun 6, 2024

fenio commented Jun 19, 2024

3v1n0 commented Jun 19, 2024

gmile commented Jul 21, 2024 • edited Loading

kyletaylored commented Jul 31, 2024

davixcky commented Aug 20, 2024

meowgorithm commented Oct 2, 2024 • edited Loading

caarlos0 commented Oct 7, 2024

mastercactapus commented Oct 7, 2024 • edited Loading

caarlos0 commented Oct 8, 2024

meowgorithm commented Oct 9, 2024

caarlos0 commented Oct 9, 2024

mastercactapus commented May 24, 2023 •

edited

Loading

maaslalani commented May 24, 2023 •

edited

Loading

mastercactapus commented May 24, 2023 •

edited by maaslalani

Loading

maaslalani commented May 24, 2023 •

edited

Loading

gmile commented Jul 21, 2024 •

edited

Loading

meowgorithm commented Oct 2, 2024 •

edited

Loading

mastercactapus commented Oct 7, 2024 •

edited

Loading