Releases: maciejhirsz/logos
v0.15 - 2024 edition, improved callback, fixes and better docs!
This is a small release, but it contains two important fixes:
But it also contains a breaking change:
- MSRV has been bumped to 1.74
As well as a nice new feature for callback to support returning Result<Skip>
!
Moreover, the documentation has received very nice additions.
Many thanks to all the contributors!
What's Changed
- chore(deps): bump ariadne dependency by @LecrisUT in #425
- fix(book): link to common-regex from logos/attributes in book by @jonesetc in #430
- fix(lib): generate same code on big-endian platforms by @0x2a-42 in #431
- Support callbacks returning Result by @romamik in #432
- chore(deps): bump thollander/actions-comment-pull-request from 2 to 3 by @dependabot in #429
- chore(deps): bump codecov/codecov-action from 4 to 5 by @dependabot in #437
- Bump rstest to 0.23.0, the current release by @musicinmybrain in #438
- fix(lib): use precise capturing on 1.82+ by @indietyp in #435
- chore(book): add simple calculator example by @your-diary in #439
- Minor readme improvments by @tkr-sh in #442
- chore(release): bump to 0.14.3 by @jeertmans in #443
- chore(release): bump to 0.15 by @jeertmans in #451
New Contributors
- @LecrisUT made their first contribution in #425
- @jonesetc made their first contribution in #430
- @0x2a-42 made their first contribution in #431
- @romamik made their first contribution in #432
- @musicinmybrain made their first contribution in #438
- @indietyp made their first contribution in #435
- @your-diary made their first contribution in #439
- @tkr-sh made their first contribution in #442
Full Changelog: v0.14.2...v0.15
v0.14.3 - YANKED - 2024 edition, improved callback, fixes and better docs!
Important
This version was yanked as it bumped the MSRV, and we should have change the minor (not the patch) part of the version to reflect this.
This is a small release, but it contains two important fixes:
As well as a nice new feature for callback to support returning Result<Skip>
!
Moreover, the documentation has received very nice additions.
Many thanks to all the contributors!
What's Changed
- chore(deps): bump ariadne dependency by @LecrisUT in #425
- fix(book): link to common-regex from logos/attributes in book by @jonesetc in #430
- fix(lib): generate same code on big-endian platforms by @0x2a-42 in #431
- Support callbacks returning Result by @romamik in #432
- chore(deps): bump thollander/actions-comment-pull-request from 2 to 3 by @dependabot in #429
- chore(deps): bump codecov/codecov-action from 4 to 5 by @dependabot in #437
- Bump rstest to 0.23.0, the current release by @musicinmybrain in #438
- fix(lib): use precise capturing on 1.82+ by @indietyp in #435
- chore(book): add simple calculator example by @your-diary in #439
- Minor readme improvments by @tkr-sh in #442
- chore(release): bump to 0.14.3 by @jeertmans in #443
New Contributors
- @LecrisUT made their first contribution in #425
- @jonesetc made their first contribution in #430
- @0x2a-42 made their first contribution in #431
- @romamik made their first contribution in #432
- @musicinmybrain made their first contribution in #438
- @indietyp made their first contribution in #435
- @your-diary made their first contribution in #439
- @tkr-sh made their first contribution in #442
Full Changelog: v0.14.2...v0.14.3
v0.14.2 - Optional `forbid_unsafe` feature, fuzzing, book, and more!
What's Changed
- chore(book): added link to Rust's reference by @CommanderStorm in #411
- feat: impl Source for T: Deref in no_std by @yjhmelody in #406
- fix(codegen/regex): allow vec growth on parse by @LeoDog896 in #405
- test: basic fuzzing by @LeoDog896 in #407
- feat(lib): add
forbid_unsafe
feature to disable unsafe code by @davidkern in #413 - chore(version): release v0.14.2 by @jeertmans in #422
New Contributors
- @CommanderStorm made their first contribution in #411
- @yjhmelody made their first contribution in #406
- @davidkern made their first contribution in #413
Full Changelog: v0.14.1...v0.14.2
0.14.1 - Debug feature and fixes
What's Changed
- fix(doc): reset logos2 to logos by @jeertmans in #372
- chore(book): add JSON-borrowed parser example by @jeertmans in #373
- Add Rc and Arc sources by @InfiniteCoder01 in #340
- Fix unicode dot by @RustyYato in #376
- chore(docs): cleanup examples by @jeertmans in #381
- chore(lib): add debug feature by @jeertmans in #382
- Cleanup unused Source features by @kmicklas in #335
- chore(deps): bump peaceiris/actions-mdbook from 1 to 2 by @dependabot in #387
- Fix
Lexer::clone
leak and UB + tests by @Jakobeha in #390 - fix(lib): correctly handle miss for loop in loop by @lukas-code in #393
- chore(lib): remove error branch from LUT if it is unreachable by @RustyYato in #386
- fix(docs): typo by @joerivanruth in #396
- chore(docs): Adds graph debug documentation to book by @afreeland in #379
- chore: drop python linting frmo pre-commit-config by @LeoDog896 in #403
- refactor: don't use deprecated max_value() method by @LeoDog896 in #404
- chore(version): bump logos version to 0.14.1 by @jeertmans in #409
- fix(docs): change old 0.14.0 by @jeertmans in #410
New Contributors
- @InfiniteCoder01 made their first contribution in #340
- @RustyYato made their first contribution in #376
- @Jakobeha made their first contribution in #390
- @lukas-code made their first contribution in #393
- @joerivanruth made their first contribution in #396
- @afreeland made their first contribution in #379
- @LeoDog896 made their first contribution in #403
Full Changelog: v0.14...v0.14.1
0.14 - Let's make Logos active again!
After months without any new release, 0.14 is finally out!
Short summary
- Logos now provides a very useful handbook;
- Bug fixes and continuous-integration tools via GitHub workflows;
⚠️ BREAKING priority of classes was changed from 1 to 2, see #320, as a result of bumpregex-syntax
. This bump actually improves so performances, but only slightly (see below). The documentation about token disambiguation reflects that changes too.
Performance changes from #320:
group before changes
----- ------ -------
count_ok/identifiers 1.04 869.2±6.09ns 854.7 MB/sec 1.00 832.9±14.13ns 891.9 MB/sec
count_ok/keywords_operators_and_punctators 1.04 2.6±0.02µs 784.4 MB/sec 1.00 2.5±0.08µs 811.9 MB/sec
count_ok/strings 1.03 597.6±5.73ns 1389.9 MB/sec 1.00 582.6±6.94ns 1425.8 MB/sec
iterate/identifiers 1.05 883.7±23.22ns 840.7 MB/sec 1.00 838.2±12.23ns 886.3 MB/sec
iterate/keywords_operators_and_punctators 1.01 2.6±0.03µs 768.0 MB/sec 1.00 2.6±0.03µs 778.2 MB/sec
iterate/strings 1.02 595.7±7.48ns 1394.5 MB/sec 1.00 583.6±4.39ns 1423.3 MB/sec
The detailed list of patches can be found below, many thanks to all contributors!
Longer story and help needed
As mentioned in 0.13, the author of Logos, @maciejhirsz,
is reducing his time on GitHub. A few months ago, I was granted collaborator rights, so I can help to maintain this project by reviewing and merging PRs.
As a result, please tag me, @jeertmans, whenever you need help or else (if I did not give any sign of life for a few days).
From now on, I will be able to publish new versions to crates.io, which I hope to do more frequently than in the past months (or years) of this project.
However, I do not master this project at all, and I welcome any help from the community to make this project grow!
I tried to setup a nice working environment, with many tests and guides, to facilitate first time contributors' life!
What's Changed
- impl<T: Logos> CallbackResult for T by @ExoticMatter in #295
- chore(ci): automatically build handbook by @jeertmans in #293
- Add missing
Debug
error type requirement by @shilangyu in #298 - chore(book): adding examples by @jeertmans in #300
- chore(codegen): error to prevent undesired behavior by @jeertmans in #303
- chore(lib/book): move lib's doc to book by @jeertmans in #304
- chore(book): document extras by @jeertmans in #309
- it's -> its by @eniraa in #314
- chore(ci): improving CI by @jeertmans in #301
- chore(book): adding help and contributing pages by @jeertmans in #311
- feat(lib): impl Clone for SpannedIter by @moenie99 in #317
- chore(ci): print benchmarks to github step summary when fail to comment by @jeertmans in #323
- Use lifetime GAT for Source::Slice by @kmicklas in #332
- Fix ignored source attribute by @kmicklas in #333
- Update
Source
test for GATSlice
by @kmicklas in #334 - fix license violations by @iSaluki in #344
- Fix logos_path in generated code by @jannik4 in #348
- chore(book): first Logo's handbook version by @jeertmans in #319
- chore: fix typos by @a-rustacean in #361
- chore(deps): bump baptiste0928/cargo-install from 1 to 3 by @dependabot in #364
- chore(deps): bump actions/checkout from 2 to 4 by @dependabot in #357
- chore(deps): bump actions/upload-pages-artifact from 1 to 3 by @dependabot in #353
- chore(deps): bump actions/deploy-pages from 2 to 4 by @dependabot in #356
- Releasing Logos2 by @jeertmans in #365
- chore(release): add hooks by @jeertmans in #366
- chore(deps): release test versions by @jeertmans in #367
- chore(deps): bumping regex-syntax by @jeertmans in #320
- chore(deps): bump codecov/codecov-action from 2 to 4 by @dependabot in #363
- chore(deps): bump pretty-assertions to 1.4.0 by @jeertmans in #368
- chore(version): bump logos version to 0.14.0 by @jeertmans in #369
New Contributors
- @jeertmans made their first contribution in #293
- @shilangyu made their first contribution in #298
- @eniraa made their first contribution in #314
- @moenie99 made their first contribution in #317
- @kmicklas made their first contribution in #332
- @iSaluki made their first contribution in #344
- @a-rustacean made their first contribution in #361
- @dependabot made their first contribution in #364
Full Changelog: v0.13...v0.14
0.13
This is a long overdue release that includes a bunch of community provided PRs over the last months:
⚠️ BREAKING: @agluszak did the heroic feat of making theLexer
produce aResult<Token, Token::Error>
which removes, the need for the#[error]
variant. More on that below. (#273)- @jannik4 implemented support for
#[logos(crate = path::to::logos)]
. (#268) - @Plaba added the support for
{n,m}
regex ranges. (#278) SpannedIter
now derefs intoLexer
, so allLexer
methods are available on it, based on work started by @simvux. (#283, #231)- Added new
#[logos(skip ...)]
attribute to help declare whitespace now that#[error]
variant is gone. (#284) - Updated
syn
dependency to 2.0. (#289)
Migrating from 0.12
In 0.12 interacting with Logos would look like:
use logos::Logos;
#[derive(Logos, Debug, PartialEq)]
enum Token {
#[token("Hello")]
Hello,
#[token("Logos")]
Logos,
// This variant will be emitted if an error is encountered
#[error]
// By convention we also add an attribute to skip whitespace here
#[regex(r"[ \t\n\f]+", logos::skip)]
Error,
}
fn main() {
let mut lex = Token::lexer("Hello Logos");
// `Lexer` is an iterator of `Token`:
assert_eq!(lex.next(), Some(Token::Hello));
assert_eq!(lex.next(), Some(Token::Logos));
assert_eq!(lex.next(), None);
}
In 0.13 the #[error]
variant is gone, this is the updated code:
use logos::Logos;
#[derive(Logos, Debug, PartialEq)]
#[logos(skip r"[ \t\n\f]+")] // new way to annotate whitespace
enum Token {
#[token("Hello")]
Hello,
#[token("Logos")]
Logos,
}
fn main() {
let mut lex = Token::lexer("Hello Logos");
// `Lexer` is an iterator of `Result<Token, Token::Error>`:
assert_eq!(lex.next(), Some(Ok(Token::Hello)));
assert_eq!(lex.next(), Some(Ok(Token::Logos)));
assert_eq!(lex.next(), None);
}
By default the associated Token::Error
type is just ()
and holds no data. Originally I've put the #[error]
variant on the token because having one flat enum to make a match
on seemed like a performance win. Upon some scrutiny however there is no performance cost to matching on Result<Token, ()>
vs flat Token
due to Rust's ability to optimize enums: if your Token
is a simple enumeration with only unit variants (holds no data), and the number of variants doesn't exceed 254, then Result<Token, ()>
and Option<Result<Token, ()>>
will be represented by a single byte at runtime, and matching for pattern Ok(Token::Hello)
in 0.13 should compile to the same code as plain Token::Hello
would be in 0.12. See the full discussion in #104.
Future of Logos
If you've been using Logos for a while now and you've been fighting with some compile-time bugs, you might have been expecting a bunch of fixes here. Alas, as much as it pains me, there are none. There are two main reasons for this, one being me taking a really long coding break last year well into this year:
This is including public and private repos, I simply wrote no code at all in any capacity, not personal projects, not professional, nothing at all. I really needed that time.
While I'm back and active now, a lot of my time currently is spent on Kobold.
The derive macro hell
The elephant in the room is that the derive macro is simply unfixable in the current state and requires a complete rewrite. Last time this happened was 3 years ago in 0.10. That rewrite improved things considerably and allowed me to fix numerous outstanding bugs at the time. It took me about two weeks of pretty intense work, and I reckon this time it would be no smaller task.
The main issue is that I've chosen to implement different states of the state machine as separate functions. This made it impossible for the these functions to share common variables on stack, but being really clever I got it to work for the existing test suite. A fatal mistake. Douglas Crockford once said, paraphrasing from memory:
Debugging is harder than writing code. If you write code as cleverly as you possibly can, you are by definition not smart enough to debug it.
The good news is I know how to fix things in a way that basically avoids nearly all pitfalls of current implementation. Instead of functions all states of the state machine should be just unit variants of a single enum with some code in their corresponding match
expression arms stuck in a loop.
There are pros and cons to that. When it comes to performance lookup tables that currently use fn
pointers would become much smaller (by a factor of 8 for most projects compiled on 64bit architecture), making it much less demanding on CPU cache. On the flip side the generated code would lose automatic inlining of nested function calls, and while that can be done manually as an optimization rolled into the macro itself, it might be a new source of bugs if not done right, and it will almost definitely bloat out the amount of code produced (though it might be sensible to have some feature flags to disable this, particularly for targets like wasm32
).
The rewrite in principle should also make it possible to make Logos use ropes such as ropey.
Phasing out #[regex]
Not as much of a nightmare, but something I feel has been a problem for users is relying on a subset of regex syntax, particularly limiting backtracking and look-ahead features for the sake of both simplicity and performance. While it is possible to express pretty much any common programming language syntax necessary with that subset, often enough people find it confusing that you can't just do things like x*?
.
What I'd like to do eventually is expand on the #[token]
attribute with more DSL-y features, using string literals as means of escaping characters, backwards compatible with current use of the attribute to denote a simple byte sequence. Consider a simple block comment /* ... */
, I would like that to be declared as:
#[derive(Logos)]
enum Token {
#[token("/*" ... "*/")]
BlockComment,
// ...
}
The equivalent regex rules for that today would be #[regex(r"\/\*([^*]|\*[^/])*\*\/")]
... I think. The fact that I'm not able to write this out of the top of my head without being sure it works should be a testament enough of how clunky it can be. Not to mention this one example kind of breaks one of the main promises of Logos, and that is of making the generated code faster than anything you could write by hand. You do not want to see the codegen for that particular regex.
When?
I've no idea.
In the next few weeks I'd like to get Kobold to 1.0. I also want to get Ramhorns to 1.0 sometime soon, that carte honestly should have been marked 1.0 since last year. I've been guilty of not pushing my crates to 1.0 fast enough, and by not fast enough I mean I've been on noughty versions for most of my projects for years, even though most of them really are production ready and I'm just too much of a chicken to stabilize the API.
With the changes done in 0.13 now I believe the last two issues standing ahead of Logos going 1.0 are the two mentioned above: derive rewrite to fix bugs, and #[token]
syntax expansion / phasing out of #[regex]
to make the API nicer to use. The former is something I'm best suited to do alone, although the latter could be done by community (though realistically only after the rewrite is done).
If there is a living soul willing to undertake the big rewrite and help push this along faster, do reach out. I can at very least help explain what the hell current codegen is doing. I'd be also happy to give a person or two the ability to review and merge community PRs.
0.12.1
This is a long overdue patch release that includes a number of fixes from community:
- Subpattern definitions now accept nesting other subpatterns within them (#237 by @Jezza)
- Callbacks can now use a new
FilterResult
type where emitting an error or skipping would be necessary (#236 by @Jeremy-Stafford) Chunk
is now implemented for all array sizes viaconst
generics, which fixes issues with long regex patterns (#221 by @icewind1991)
0.12.0
- Added case insensitive flags via
ignore(case)
andignore(ascii_case)
(by @gymore-io #198). - Removed the
lookup!
macro. - Updated dependency in
logos-derive
that had a security alert on it.