CSS Tokenizer: inconsistent handling of plus vs minus and % sign

[jrfnl]

Given the following testcode:

#div1 {
    left: +50px;
    right: -50px;
    width: calc(100% - 100px);
    max-width: calc(60% + 100px);
}

---

The tokenizer short output

0 :: L001 :: C1 :: T_OPEN_TAG :: (0) ::
1 :: L001 :: C1 :: T_HASH :: (1) :: #
2 :: L001 :: C2 :: T_STRING :: (4) :: div1
3 :: L001 :: C6 :: T_WHITESPACE :: (1) ::
4 :: L001 :: C7 :: T_OPEN_CURLY_BRACKET :: (1) :: {
5 :: L001 :: C8 :: T_WHITESPACE :: (0) ::

6 :: L002 :: C1 :: T_WHITESPACE :: (4) ::
7 :: L002 :: C5 :: T_STYLE :: (4) :: left
8 :: L002 :: C9 :: T_COLON :: (1) :: :
9 :: L002 :: C10 :: T_WHITESPACE :: (1) ::
10 :: L002 :: C11 :: T_PLUS :: (1) :: +
11 :: L002 :: C12 :: T_LNUMBER :: (2) :: 50
12 :: L002 :: C14 :: T_STRING :: (2) :: px
13 :: L002 :: C16 :: T_SEMICOLON :: (1) :: ;
14 :: L002 :: C17 :: T_WHITESPACE :: (0) ::

15 :: L003 :: C1 :: T_WHITESPACE :: (4) ::
16 :: L003 :: C5 :: T_STYLE :: (5) :: right
17 :: L003 :: C10 :: T_COLON :: (1) :: :
18 :: L003 :: C11 :: T_WHITESPACE :: (1) ::
19 :: L003 :: C12 :: T_LNUMBER :: (3) :: -50
20 :: L003 :: C15 :: T_STRING :: (2) :: px
21 :: L003 :: C17 :: T_SEMICOLON :: (1) :: ;
22 :: L003 :: C18 :: T_WHITESPACE :: (0) ::

23 :: L004 :: C1 :: T_WHITESPACE :: (4) ::
24 :: L004 :: C5 :: T_STYLE :: (5) :: width
25 :: L004 :: C10 :: T_COLON :: (1) :: :
26 :: L004 :: C11 :: T_WHITESPACE :: (1) ::
27 :: L004 :: C12 :: T_STRING :: (4) :: calc
28 :: L004 :: C16 :: T_OPEN_PARENTHESIS :: (1) :: (
29 :: L004 :: C17 :: T_LNUMBER :: (3) :: 100
30 :: L004 :: C20 :: T_MODULUS :: (1) :: %
31 :: L004 :: C21 :: T_WHITESPACE :: (1) ::
32 :: L004 :: C22 :: T_MINUS :: (1) :: -
33 :: L004 :: C23 :: T_WHITESPACE :: (1) ::
34 :: L004 :: C24 :: T_LNUMBER :: (3) :: 100
35 :: L004 :: C27 :: T_STRING :: (2) :: px
36 :: L004 :: C29 :: T_CLOSE_PARENTHESIS :: (1) :: )
37 :: L004 :: C30 :: T_SEMICOLON :: (1) :: ;
38 :: L004 :: C31 :: T_WHITESPACE :: (0) ::

39 :: L005 :: C1 :: T_WHITESPACE :: (4) ::
40 :: L005 :: C5 :: T_STYLE :: (9) :: max-width
41 :: L005 :: C14 :: T_COLON :: (1) :: :
42 :: L005 :: C15 :: T_WHITESPACE :: (1) ::
43 :: L005 :: C16 :: T_STRING :: (4) :: calc
44 :: L005 :: C20 :: T_OPEN_PARENTHESIS :: (1) :: (
45 :: L005 :: C21 :: T_LNUMBER :: (2) :: 60
46 :: L005 :: C23 :: T_MODULUS :: (1) :: %
47 :: L005 :: C24 :: T_WHITESPACE :: (1) ::
48 :: L005 :: C25 :: T_PLUS :: (1) :: +
49 :: L005 :: C26 :: T_WHITESPACE :: (1) ::
50 :: L005 :: C27 :: T_LNUMBER :: (3) :: 100
51 :: L005 :: C30 :: T_STRING :: (2) :: px
52 :: L005 :: C32 :: T_CLOSE_PARENTHESIS :: (1) :: )
53 :: L005 :: C33 :: T_SEMICOLON :: (1) :: ;
54 :: L005 :: C34 :: T_WHITESPACE :: (0) ::

55 :: L006 :: C1 :: T_CLOSE_CURLY_BRACKET :: (1) :: }
56 :: L006 :: C2 :: T_WHITESPACE :: (0) ::

57 :: L007 :: C1 :: T_CLOSE_TAG :: (0) ::

---

Note the following inconsistency:

Plus sign and number tokenized separately:

10 :: L002 :: C11 :: T_PLUS :: (1) :: +
11 :: L002 :: C12 :: T_LNUMBER :: (2) :: 50

versus minus sign tokenized as part of the T_LNUMBER token:

19 :: L003 :: C12 :: T_LNUMBER :: (3) :: -50

And the following unexpected result - the percent sign is tokenized as modulus:

29 :: L004 :: C17 :: T_LNUMBER :: (3) :: 100
30 :: L004 :: C20 :: T_MODULUS :: (1) :: %

[Beakerboy]

Several of the token names come from the PHP token_get_all function. Since, in PHP, the percent sign is the modulus operator, the CSS tokenizer just keeps that name. The name of the token is just a name, not any idication of what operation the token is expected to be performing in that context of the code.

Is the way the tokens are being defined affecting a sniff, or the results of a running your code through a standard, or are you just trying to understand how the tokenizer works?
Line 328 of src/Tokenizers/CSS.php is where the minus sign token is removed and instead added to the LNUMBER after it. You could add a new case to do something similar with plus signs, or create a sniff that says plus signs are not allowed since they are assumed.

[jrfnl]

Closing this ticket as support for CSS will be dropped in PHPCS 4.0 anyway.