Ensure Parser only reads tokens as necessary #135
Labels
semantics.parser
Any issue relating to changing the semantics of parser code.
semantics
Generic label for semantics issues. All semantics issues should have this tag.
One of the original goals of the design of the Parser was to make it able to work with streamed input, where the entirety of the input is not known immediately. The implementation around this says that every parsing method should only consume the tokens that it requires.
Something that came up through some initial work on implementing a REPL for Myst was that, in some cases, the parser will read one or two tokens ahead to determine which branch of the parser to descend into. This caused an issue where the REPL would ask the user for more input even when the expression they entered was complete.
A different possible solution for this is to somehow pad the input that the REPL passes to the parser so that any look-ahead is unambiguously dealt with. With the use-case of a REPL, padding the input with newlines should be enough to deal with this, as part of the grammar says that newlines act as expression delimiters, so the presence of a newline after a complete expression is enough to indicate the end of the expression (e.g., no methods are chained after the expression).
The text was updated successfully, but these errors were encountered: