Skip to content

Commit

Permalink
Fully preserve indented strings and match Nix parser
Browse files Browse the repository at this point in the history
This should make sure that:
- No sequences of characters get modified during parsing,
  indented strings are left exactly as is
- We won't have any parsing bugs

Also added a ton of docs to explain how it works and more tests
  • Loading branch information
infinisil committed Jun 26, 2024
1 parent c67a7b6 commit 086f0bc
Show file tree
Hide file tree
Showing 4 changed files with 80 additions and 11 deletions.
55 changes: 44 additions & 11 deletions src/Nixfmt/Parser.hs
Original file line number Diff line number Diff line change
Expand Up @@ -193,17 +193,50 @@ indentedStringPart :: Parser StringPart
indentedStringPart =
TextPart
<$> someText
( chunk "''\\n"
<|> chunk "''\\r"
<|> chunk "''\\t"
<|> chunk "''\\"
*> (Text.singleton <$> anySingle)
<|> chunk "''$"
<|> chunk "'''"
<|> chunk "$$"
<|> try (chunk "$" <* notFollowedBy (char '{'))
<|> try (chunk "'" <* notFollowedBy (char '\''))
<|> someP (\t -> t /= '\'' && t /= '$' && t /= '\n')
( {-
This should match https://github.com/NixOS/nix/blob/052f1320ddf72d617e337479ff1bf22cb4ee682a/src/libexpr/lexer.l#L182-L205
To understand that file, [this](https://westes.github.io/flex/manual/Matching.html#Matching) is important:
> If it finds more than one match, it takes the one matching the most text.
> If it finds two or more matches of the same length, the rule listed first in the flex input file is chosen.
There are some inconsequential differences though:
- We only parse one line at a time here, so we make sure to not process any newlines
- We don't want to transform the input in any way, e.g. don't turn ''' into ''
This is to preserve the input string as-is
-}

-- <IND_STRING>([^\$\']|\$[^\{\']|\'[^\'\$])+
-- While this rule doesn't have a fixed length, it's non-conflicting with the rules below,
-- so we can do it first without worrying about the length matching
someText
( Text.singleton
<$> satisfy (\t -> t /= '$' && t /= '\'' && t /= '\n')
<|> try (Text.snoc <$> chunk "$" <*> satisfy (\t -> t /= '{' && t /= '\'' && t /= '\n'))
<|> try (Text.snoc <$> chunk "'" <*> satisfy (\t -> t /= '\'' && t /= '$' && t /= '\n'))
)
-- These rules are of length 3, they need to come before shorter ones
-- <IND_STRING>\'\'\$
<|> chunk "''$"
-- <IND_STRING>\'\'\'
<|> chunk "'''"
-- <IND_STRING>\'\'\\{ANY} -> Note that ANY can match newlines as well, but we need to ignore those
<|> do
prefix <- chunk "''\\"
-- If we do have a newline
rest <-
-- If there's no newline, take the next character
(notFollowedBy (char '\n') *> (Text.singleton <$> anySingle))
-- Otherwise there's an unconsumed newline, which we don't need to handle,
-- it's consumed elsewhere
<|> pure ""
pure $ prefix <> rest
-- These are rules with length 2 and 1
-- <IND_STRING>\$\{ -> don't match, this will be an interpolation
-- <IND_STRING>\$ -> do match, just a dollar
<|> try (chunk "$" <* notFollowedBy (char '{'))
-- <IND_STRING>\'\' -> don't match, indented string ends
-- <IND_STRING>\' -> do match, just a quote
<|> try (chunk "'" <* notFollowedBy (char '\''))
)

indentedLine :: Parser [StringPart]
Expand Down
14 changes: 14 additions & 0 deletions test/stable/indented-string.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# These come from previous parser bugs in Nix
[
''
''\a
''\
'${true}'
$'\t'
''

''ending dollar $''
''$''
]
3 changes: 3 additions & 0 deletions test/stable/regression-197.nix
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
''
''\'''${A}
''
19 changes: 19 additions & 0 deletions test/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,22 @@ for file in test/diff/**/in.nix; do
exit 1
fi
done

# Verify "stable", files that don't change when formatted
for file in test/stable/*.nix; do
echo "Checking $file"
if ! out=$(nixfmt --verify < "$file"); then
echo "[ERROR] failed nixfmt verification"
exit 1
fi

if diff --color=always --unified "$file" <(echo "$out"); then
echo "[OK]"
elif [[ $* == *--update-diff* ]]; then
echo "$out" > "$file"
echo "[UPDATED] $file"
else
echo "[ERROR] Formatting not stable (run with --update-diff to update the diff)"
exit 1
fi
done

0 comments on commit 086f0bc

Please sign in to comment.