Skip to content

Commit

Permalink
Merge branch 'develop'
Browse files Browse the repository at this point in the history
* develop: (43 commits)
  specify next release
  add groups support
  disable time assertions and composite shrinking proofs in the CI
  add Tag::ci and ::local
  Revert "include the last values that result in failure"
  include the last values that result in failure
  add memory limit for the coverage as well
  fix test
  add memory limit to verify blackbox can disable it
  discard psalm error
  allow to display the generated scenario inside the phpunit emulation
  fix messages not being displayed
  do not print vendor stack trace on macOS CI
  add Set\MutuallyExclusive
  increase allowed delta as usleep is not very precise
  add Set\Slice to the changelog
  reduce the number of scenarri for the time assertions
  add Set\Slice
  add Assert::memory()
  add Assert::time()
  ...
  • Loading branch information
Baptouuuu committed Aug 27, 2023
2 parents 640334a + bda398d commit 2aa9f57
Show file tree
Hide file tree
Showing 48 changed files with 2,465 additions and 150 deletions.
9 changes: 6 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ jobs:
php-version: ${{ matrix.php-version }}
extensions: mbstring, intl
coverage: none
ini-values: memory_limit=256M
- name: Composer
uses: "ramsey/composer-install@v2"
with:
dependency-versions: ${{ matrix.dependencies }}
- name: BlackBox
run: php blackbox.php
run: php blackbox.php ci
coverage:
runs-on: ${{ matrix.os }}
strategy:
Expand All @@ -43,18 +44,20 @@ jobs:
php-version: ${{ matrix.php-version }}
extensions: mbstring, intl
coverage: xdebug
ini-values: memory_limit=256M
- name: Composer
uses: "ramsey/composer-install@v2"
with:
dependency-versions: ${{ matrix.dependencies }}
- name: BlackBox
run: php blackbox.php
run: php blackbox.php ci
env:
ENABLE_COVERAGE: 'true'
BLACKBOX_SET_SIZE: '1'
- uses: codecov/codecov-action@v1
with:
token: ${{ secrets.CODECOV_TOKEN }}
phpunit_latest:
phpunit:
runs-on: ${{ matrix.os }}
strategy:
matrix:
Expand Down
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,24 @@
# Changelog

# 5.3.0 - 2023-08-27

### Added

- `Innmind\BlackBox\PHPUnit\Framework\TestCase`
- `Innmind\BlackBox\PHPUnit\Load`
- `Innmind\BlackBox\Application::disableMemoryLimit()`
- `Innmind\BlackBox\Runner\Assert::matches()`
- `Innmind\BlackBox\Runner\Assert::time()`
- `Innmind\BlackBox\Runner\Assert::memory()`
- `Innmind\BlackBox\Set\Slice`
- `Innmind\BlackBox\Set\MutuallyExclusive`
- `Innmind\BlackBox\Tag::ci`
- `Innmind\BlackBox\Tag::local`

### Fixed

- The message `Failing scenarii:` was always printed in PHPUnit output even when there was no failures

## 5.2.0 - 2023-08-19

### Added
Expand Down
30 changes: 30 additions & 0 deletions documentation/assert_memory.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Assert memory usage

You can use BlackBox to make sure use less than a specified amount of memory to make sure your code doesn't have a memory leak. You can do so with the `Assert::memory()` method.

In order for this assertion to work properly you need to:
- add `declare(ticks = 1);` at the top of file where you call the assertion
- the callable cannot use the anonymous function short notation

For example:

```php
<?php
declare(strict_types = 1);
declare(ticks = 1);

return static function() {
yield test(
'Your test name',
static function($assert) {
$assert
->memory(static function() {
$yourSystem = new YourSystem();
$yourSystem->doStuff();
})
->inLessThan()
->megaBytes(1);
},
);
};
```
39 changes: 39 additions & 0 deletions documentation/own_assertions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Adding assertions

BlackBox comes with a fixed set of assertions provided by the `Assert` class. But sometime you may want to reuse an assertion (or set of assertions) thoughout your proofs. You can do so with the `Assert::matches()` method.

Let's say you want to validate a Uuid, you could create a static method on a class like this:

```php
use Innmind\BlackBox\Runner\Assert;

final class Uuid
{
/**
* @return callable(Assert): void
*/
public static function of(mixed $value): callable
{
return static fn(Assert $assert) => $assert
->string($value)
->matches('~^[a-f0-9]{8}(-[a-f0-9]{4}){3}-[a-f0-9]{12}$~');
}
}
```

And in your proof you would use it like this:

```php
use Innmind\BlackBox\{
Set,
Runner\Assert,
};

yield proof(
'Name of your proof',
given(Set\Type::any()),
static fn(Assert $assert, $value) => $assert->matches(Uuid::of($value)),
);
```

**Note**: this example proof will fail because generated values are not uuids.
40 changes: 40 additions & 0 deletions documentation/phpunit.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,43 @@ final class MyTestCase extends TestCase
- you won't benefit from the [shrinking mechanism](proof.md#the-power-of-shrinking)
- you won't benefit from the output of the generated data that make you test fail
- you may run out of memory (since PHPUnit keep in memory all scenarii data)

## Running your tests via BlackBox

If you wish to migrate to BlackBox but don't want to rewrite all your existing tests you can run them directly via BlackBox.

The first step is to prefix the `PHPUnit\Framework\TestCase` class with `Innmind\BlackBox\`.

The second step is to load the tests like this:

```php
use Innmind\BlackBox\{
Application,
PHPUnit\Load,
};

Application::new()
->tryToProve(function() {
yield from Load::testsAt('path/to/your/tests');
})
->exit();
```

If you want to take a look at a migration you can look at [BlackBox's own PHPUnit tests](../tests/) that are now run via BlackBox itself.

**Note**: Running BlackBox's PHPUnit tests via BlackBox increase execution speed by 35% (from ~7.1s down to ~4.6s) on a MackBook Pro M1 Max.

### Feature coverage

PHPUnit is a very large testing framework with lots of features. BlackBox doesn't support all its features when running your tests.

Supported features:
- test `setUp()`/`tearDown()`
- assertions that have a correspondance in BlackBox
- data providers declared with an attribute
- groups declared with an attribute (the name must have a correspondance in `Innmind\BlackBox\Tag`)

Some important features that are not supported:
- mocks
- classes `setUpBeforeClass()`/`tearDownAfterClass()`
- assertions that don't have a correspondance in BlackBox (such as files assertions)
2 changes: 2 additions & 0 deletions documentation/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,6 @@ composer require --dev innmind/black-box
- [Organize your proofs](organize.md)
- [Using tags](tags.md)
- [Configuring the test runner](config.md)
- [Asserting memory usage](assert_memory.md)
- [Adding assertions](own_assertions.md)
- [Compatibility with PHPUnit](phpunit.md)
4 changes: 2 additions & 2 deletions phpunit.xml.dist
Original file line number Diff line number Diff line change
Expand Up @@ -10,15 +10,15 @@
</extensions>
<testsuites>
<testsuite name="Test suite">
<directory>./tests</directory>
<directory>./phpunit</directory>
</testsuite>
</testsuites>
<source>
<include>
<directory>.</directory>
</include>
<exclude>
<directory>./tests</directory>
<directory>./phpunit</directory>
<directory>./vendor</directory>
</exclude>
</source>
Expand Down
71 changes: 71 additions & 0 deletions phpunit/PHPUnit/BlackBoxTest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<?php
declare(strict_types = 1);

namespace Tests\Innmind\BlackBox\PHPUnit;

use Innmind\BlackBox\{
PHPUnit\BlackBox,
Set,
};
use PHPUnit\{
Framework\TestCase,
Framework\Attributes\DataProvider,
};

class BlackBoxTest extends TestCase
{
use BlackBox;

public function testTrait()
{
$class = new class() {
use BlackBox;

public function assert(): int
{
$called = 0;
$this
->forAll(Set\Integers::any())
->then(static function() use (&$called) {
++$called;
});

return $called;
}
};

// 200 because it reads the `BLACKBOX_SET_SIZE` env var
$this->assertSame(200, $class->assert());
}

public function testDoesntFailWhenTheExceptionIsExpected()
{
$this
->forAll(Set\Strings::any(), Set\Integers::above(0))
->then(function($message, $code) {
$exception = new class($message, $code) extends \Exception {
};

$this->expectException(\get_class($exception));
$this->expectExceptionMessage($message);
$this->expectExceptionCode($code);

throw $exception;
});
}

#[DataProvider('ints')]
public function testDataProviderCompatibility($a, $b)
{
$this->assertIsInt($a);
$this->assertIsInt($b);
}

public static function ints(): iterable
{
return self::forAll(
Set\Integers::any(),
Set\Integers::any(),
)->asDataProvider();
}
}
11 changes: 7 additions & 4 deletions proofs/add.php
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
<?php
declare(strict_types = 1);

use Innmind\BlackBox\Set;
use Innmind\BlackBox\{
Set,
Tag,
};

function add($a, $b): string
{
Expand All @@ -16,7 +19,7 @@ function add($a, $b): string
Set\Integers::any(),
),
static fn($assert, $a, $b) => $assert->same(add($a, $b), add($b, $a)),
);
)->tag(Tag::ci, Tag::local);

yield proof(
'add is associative',
Expand All @@ -29,11 +32,11 @@ function add($a, $b): string
add(add($a, $b), $c),
add($a, add($b, $c)),
),
);
)->tag(Tag::ci, Tag::local);

yield proof(
'add is an identity function',
given(Set\Integers::any()),
static fn($assert, $a) => $assert->same((string) $a, add($a, 0)),
);
)->tag(Tag::ci, Tag::local);
};
41 changes: 37 additions & 4 deletions proofs/application.php
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ static function($assert, $random) {

$assert->true($result->successful());
},
);
)->tag(Tag::ci, Tag::local);
yield proof(
'BlackBox can run with a specified number of scenarii per proof',
given(Set\Integers::between(1, 10_000)), // limit to 10k so it doesn't take too mush time
Expand All @@ -50,7 +50,7 @@ static function($assert, $scenarii) {
->string($io->toString())
->contains("Scenarii: $scenarii");
},
);
)->tag(Tag::ci, Tag::local);
yield test(
'BlackBox can shrink the values of the proofs by default',
static function($assert) {
Expand Down Expand Up @@ -79,7 +79,7 @@ static function($assert) {
->contains('$a = 0') // as it is always the smallest value
->contains('$b = 0'); // as it is always the smallest value
},
);
)->tag(Tag::ci, Tag::local);
yield test(
'BlackBox can disable the shrinking mechanism',
static function($assert) {
Expand Down Expand Up @@ -113,5 +113,38 @@ static function($assert, $i) use (&$value) {
->not()
->contains('SF', 'The shrinking has not been disabled');
},
);
)->tag(Tag::ci, Tag::local);

yield test(
'BlackBox can disable the memory limit',
static function($assert) {
$io = Collect::new();

$assert
->expected('-1')
->not()
->same(\ini_get('memory_limit'));

$result = Application::new([])
->displayOutputVia($io)
->displayErrorVia($io)
->usePrinter(Standard::withoutColors())
->disableMemoryLimit()
->tryToProve(static function() {
yield test(
'example',
static fn($assert) => $assert->same(
'-1',
\ini_get('memory_limit'),
),
);
});

$assert->true($result->successful());
$assert
->expected('-1')
->not()
->same(\ini_get('memory_limit'));
},
)->tag(Tag::ci, Tag::local);
};
Loading

0 comments on commit 2aa9f57

Please sign in to comment.