feat(parser): the 1st version of a new combinator style parser #16876

TennyZhuang · 2024-05-21T12:40:04Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Introducing a new parser with winnow.

This is the first version of the new parser, with only the datatype parsing. However, it's also the most complicated part of our Parser. Due to our wrong design previously, we use < and > to define struct type, then introduce a necessary state angle_number during parsing. Fortunately, the new framework handle it perfectly.

winnow-rs is a parser-combinator crate, and it's forked by me for a simple patch winnow-rs/winnow#525.

The new framework can progressively replace the original parser, see Parser::parse_v2 for detail.

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

Signed-off-by: TennyZhuang <[email protected]>

src/sqlparser/src/parser_v2/data_type.rs

Signed-off-by: TennyZhuang <[email protected]>

TennyZhuang · 2024-05-23T09:51:38Z

parser_test and planner_test passed

wangrunji0408 · 2024-05-23T09:52:33Z

Can we have a better error display with this new parser framework? if it provides span in some way. 🤔

taken from #15091

TennyZhuang · 2024-05-23T10:02:07Z

Can we have a better error display with this new parser framework? if it provides span in some way. 🤔

It's possible, but must after we finish the whole refactor.

Signed-off-by: TennyZhuang <[email protected]>

BugenZhao

The attempt in this PR is truly exciting. The combinator style makes the code more readable and maintainable (although cut_err could be a bit subtle 🤯). Looking forward to seeing the user-facing benefits (like better error reporting) after the refactoring.

src/sqlparser/src/parser_v2/mod.rs

src/sqlparser/src/parser_v2/number.rs

BugenZhao · 2024-05-24T04:51:42Z

src/sqlparser/src/parser_v2/number.rs

+
+    delimited(
+        Token::LParen,
+        cut_err(literal_uint.try_map(move |v| {


Do we need cut_err here, or shall we add cut_err to all callers?

float is a valid data type, while float( not. So we must cut here.

src/sqlparser/src/parser_v2/impl_.rs

src/sqlparser/src/parser_v2/data_type.rs

BugenZhao · 2024-05-24T05:27:30Z

src/sqlparser/src/parser.rs

+            let msg = if let Some(e) = e.into_inner()
+                && let Some(cause) = e.cause()
+            {
+                format!(": {}", cause)
+            } else {
+                "".to_string()
+            };
+            ParserError::ParserError(format!(
+                "Unexpected {}{}",
+                if self.index + token_stream.location() >= self.tokens.len() {
+                    &"EOF" as &dyn std::fmt::Display
+                } else {
+                    &self.tokens[self.index + token_stream.location()] as &dyn std::fmt::Display
+                },
+                msg
+            ))


Can we use ParseError here for better error reporting?

Which ParserError? If you mean v2, I'd prefer give user a consistent user experience now.

Signed-off-by: TennyZhuang <[email protected]>

Co-authored-by: Bugen Zhao <[email protected]>

Signed-off-by: TennyZhuang <[email protected]>

xxchan · 2024-05-24T08:59:15Z

The combinator style makes the code more readable and maintainable

Why do you think the combinator is more maintainable? I think now everyone can work with the parser without learning, but not afterwards.

TennyZhuang · 2024-05-24T09:53:37Z

Why do you think the combinator is more maintainable? I think now everyone can work with the parser without learning, but not afterwards.

Let's use Golang to rewrite RisingWave 😄

Signed-off-by: TennyZhuang <[email protected]>

xxchan

rubber stamp

feat(parser): the 1st version of a new parser-combinator style parser

64132ed

Signed-off-by: TennyZhuang <[email protected]>

github-actions bot added the type/feature label May 21, 2024

TennyZhuang changed the title ~~feat(parser): the 1st version of a new parser-combinator style parser~~ feat(parser): the 1st version of a new combinator style parser May 21, 2024

add parse_v2

f68d97e

Signed-off-by: TennyZhuang <[email protected]>

TennyZhuang commented May 21, 2024

View reviewed changes

src/sqlparser/src/parser_v2/data_type.rs Outdated Show resolved Hide resolved

TennyZhuang added 19 commits May 21, 2024 22:51

Update src/sqlparser/src/parser_v2/data_type.rs

12bfadf

minor fix

b0a0afd

Signed-off-by: TennyZhuang <[email protected]>

minor improvement

8ea355a

Signed-off-by: TennyZhuang <[email protected]>

minor fix

332234b

Signed-off-by: TennyZhuang <[email protected]>

add some label

2ccd163

Signed-off-by: TennyZhuang <[email protected]>

introduce TokenStreamWrapper for better readability

5a3b34d

Signed-off-by: TennyZhuang <[email protected]>

fix parse_v2

6c9683a

Signed-off-by: TennyZhuang <[email protected]>

handle whitespace

905fa6a

Signed-off-by: TennyZhuang <[email protected]>

use preceed

80208ef

Signed-off-by: TennyZhuang <[email protected]>

remove dbg

9c9472f

Signed-off-by: TennyZhuang <[email protected]>

fix custom data type

2d6692c

Signed-off-by: TennyZhuang <[email protected]>

fix array parsing

7e8166d

Signed-off-by: TennyZhuang <[email protected]>

add a context

686217c

Signed-off-by: TennyZhuang <[email protected]>

fix unused import

51f9265

Signed-off-by: TennyZhuang <[email protected]>

fix struct parsing

89f67e5

Signed-off-by: TennyZhuang <[email protected]>

fix float parsing

6b62577

Signed-off-by: TennyZhuang <[email protected]>

fix precision parsing

32d29db

Signed-off-by: TennyZhuang <[email protected]>

add many comments

c76c314

Signed-off-by: TennyZhuang <[email protected]>

fix clippy

d00ec9f

Signed-off-by: TennyZhuang <[email protected]>

TennyZhuang marked this pull request as ready for review May 23, 2024 08:48

TennyZhuang requested a review from a team as a code owner May 23, 2024 08:48

TennyZhuang requested review from xiangjinwu, BugenZhao, st1page and wangrunji0408 May 23, 2024 08:49

TennyZhuang added 5 commits May 23, 2024 16:57

simplify error def

60f2019

Signed-off-by: TennyZhuang <[email protected]>

fix precision parsing

3b366ab

Signed-off-by: TennyZhuang <[email protected]>

fix double

26c7499

Signed-off-by: TennyZhuang <[email protected]>

fix custom type parsing

b26810d

Signed-off-by: TennyZhuang <[email protected]>

1..54

5cdab19

Signed-off-by: TennyZhuang <[email protected]>

TennyZhuang requested a review from xxchan May 23, 2024 09:52

TennyZhuang added 7 commits May 23, 2024 18:04

support TEXT

79ce4ea

Signed-off-by: TennyZhuang <[email protected]>

fix

df6137b

Signed-off-by: TennyZhuang <[email protected]>

fix ut

4504419

Signed-off-by: TennyZhuang <[email protected]>

refine error message

0e239cc

Signed-off-by: TennyZhuang <[email protected]>

fix stateful parsing

9ba9e97

Signed-off-by: TennyZhuang <[email protected]>

refine error

3cf20d1

Signed-off-by: TennyZhuang <[email protected]>

fix ut

88562ff

Signed-off-by: TennyZhuang <[email protected]>

BugenZhao reviewed May 24, 2024

View reviewed changes

TennyZhuang and others added 5 commits May 24, 2024 14:16

fix struct sep parsing

8452dc7

Signed-off-by: TennyZhuang <[email protected]>

Update src/sqlparser/src/parser_v2/number.rs

f9d93c7

Co-authored-by: Bugen Zhao <[email protected]>

address comments

162a663

Signed-off-by: TennyZhuang <[email protected]>

fix warning

cc9dbc3

Signed-off-by: TennyZhuang <[email protected]>

fix

d8b7d60

Signed-off-by: TennyZhuang <[email protected]>

revert a behavior

b7d79e1

Signed-off-by: TennyZhuang <[email protected]>

xxchan approved these changes May 24, 2024

View reviewed changes

TennyZhuang added this pull request to the merge queue May 24, 2024

Merged via the queue into main with commit 1aa8579 May 24, 2024
27 of 28 checks passed

TennyZhuang deleted the parser_v2_take_2 branch May 24, 2024 11:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(parser): the 1st version of a new combinator style parser #16876

feat(parser): the 1st version of a new combinator style parser #16876

TennyZhuang commented May 21, 2024 •

edited

Loading

TennyZhuang commented May 23, 2024

wangrunji0408 commented May 23, 2024 •

edited

Loading

TennyZhuang commented May 23, 2024

BugenZhao left a comment

BugenZhao May 24, 2024

TennyZhuang May 24, 2024

BugenZhao May 24, 2024

TennyZhuang May 24, 2024

xxchan commented May 24, 2024

TennyZhuang commented May 24, 2024

xxchan left a comment

feat(parser): the 1st version of a new combinator style parser #16876

feat(parser): the 1st version of a new combinator style parser #16876

Conversation

TennyZhuang commented May 21, 2024 • edited Loading

What's changed and what's your intention?

Checklist

Documentation

Release note

TennyZhuang commented May 23, 2024

wangrunji0408 commented May 23, 2024 • edited Loading

TennyZhuang commented May 23, 2024

BugenZhao left a comment

Choose a reason for hiding this comment

BugenZhao May 24, 2024

Choose a reason for hiding this comment

TennyZhuang May 24, 2024

Choose a reason for hiding this comment

BugenZhao May 24, 2024

Choose a reason for hiding this comment

TennyZhuang May 24, 2024

Choose a reason for hiding this comment

xxchan commented May 24, 2024

TennyZhuang commented May 24, 2024

xxchan left a comment

Choose a reason for hiding this comment

TennyZhuang commented May 21, 2024 •

edited

Loading

wangrunji0408 commented May 23, 2024 •

edited

Loading