Is it possible to test sequentially changing states?

[danilo-bc]

This feature is very useful for TDD with the simplest applications. One example I'm trying is this cell

begin
	test = WasRun("some_name") #this is a simple struct which holds a bool wasRun
	@test test.wasRun == false
	run(test) # this toggles the variable from false to true to simulate 'running'
	@test test.wasRun == true
end

The problem is that the output of the whole cell (and its pass state) is defined by the last state. If I define test.wasRun as true by default, the cell will still give me a green ball, even though it should be red because of the second line.

There are numerous applications for this 'before-and-after' behavior.

One alternative I can think of right now is to create a function which returns a boolean of the resulting steps, but this would either mean running all steps to reach a conclusion or write a bit more code to get the same @test behavior as regular Julia Tests does (return false when the first assertion fails, pointing to the state of the failure)

[dralletje]

First thing you can do is collect the different results, and return them as an array or Dict:

begin
	test = WasRun("some_name") #this is a simple struct which holds a bool wasRun
	pre_result = @test test.wasRun == false
	run(test) # this toggles the variable from false to true to simulate 'running'
	post_result = @test test.wasRun == true
        [pre_result, post_result]
end

or

---

write a bit more code to get the same @test behavior as regular Julia Tests does

What is this @test behavior? You have an example?

[danilo-bc]

That's a great suggestion and should help with my problem!

I put the behavior in parentheses right after mentioning them: "return false when the first assertion fails, pointing to the state of the failure". I may not have translated the actual behavior, but the idea is that if 10 @test statements are in sequence, if one of them fails, that test set halts and points to the failed line, not executing any code after that.

What I meant by having to write a bit more code was maybe adding some sort of flow control to make test statements not run if the previous was false.

[dralletje]

I was surprised with how cool it was to experiment with this:

Screen.Recording.2021-11-05.at.00.24.59.mov

This currently requires the github version PlutoTest, and this macro:

macro test_that_throws(expr)
	# esc(..) used like this (around the whole expression)
	# is not something you'll usually see... but it's something @test needs now
	esc(quote
		result = @test $(expr)
		if result isa PlutoTest.Fail
			throw(result)
		end
		result
	end)
end

I'm curious what coolness we can do with @testset to do stuff like this automatically.
Currently you can't use assignments (x = 10) or control flow (if ... else .. end) in @test calls (because they make the partial evaluation harder), but I'm wondering how necessary those are even.

[fonsp]

I got inspired to make #17, though you were probably thinking bigger than that :)

[danilo-bc]

That's fantastic! @testset support should be very useful to test-drive Pluto code!