Fix lex_state_beg_p

[haldun]

Closes #1587
Closes #1572

[haldun]

@kddnewton wdyt about this one?

[kddnewton]

I'm nervous about this change because it doesn't match up with parse.y, which makes me think we're missing some state change/detail. This line is equivalent to https://github.com/ruby/ruby/blob/486b674e2a8437bacb00c48038c04aec420c47a0/parse.y#L10801 which doesn't check if we're inside a keyword argument.

I did a little debugging and it looks like we're transitioning BEG -> CMDARG and they're transitioning BEG -> ARG. Maybe we're missing something here?

[haldun]

We are transitioning to CMDARG here:

prism/src/prism.c

Lines 8840 to 8848 in 2dc5320

	}
	pm_lex_state_t last_state = parser->lex_state;
	if (type == PM_TOKEN_IDENTIFIER || type == PM_TOKEN_CONSTANT || type == PM_TOKEN_METHOD_NAME) {
	if (lex_state_p(parser, PM_LEX_STATE_BEG_ANY | PM_LEX_STATE_ARG_ANY | PM_LEX_STATE_DOT)) {
	if (previous_command_start) {
	lex_state_set(parser, PM_LEX_STATE_CMDARG);
	} else {

because previous_command_start is true. An it is set to be true in

prism/src/prism.c

Lines 14345 to 14350 in 2dc5320

	} else {
	equal = not_provided(parser);
	if (lparen.type == PM_TOKEN_NOT_PROVIDED) {
	lex_state_set(parser, PM_LEX_STATE_BEG);
	parser->command_start = true;

[kddnewton]

Yeah I wonder if that's not correct or if we're missing something here. Because we should definitely be transitioning to ARG and not CMDARG.

[haldun]

It seemed quite unlikely but can lex_state_beg_p be just wrong? I attempted to fix it here and it seems like this bug gets resolved e854d47 Comparing with the Ruby implementation

https://github.com/ruby/ruby/blob/486b674e2a8437bacb00c48038c04aec420c47a0/parse.y#L8517C1-L8517C89

where IS_lex_state_all_for is defined as

https://github.com/ruby/ruby/blob/486b674e2a8437bacb00c48038c04aec420c47a0/parse.y#L347

[haldun]

Also seems to fix #1572

[haldun]

@kddnewton Can you take another look into this?

[kddnewton]

Great find!