Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception when last block in Pipfile is a [[source]] and no trailing newline at end of file #6244

Closed
gdiepen opened this issue Sep 16, 2024 · 2 comments

Comments

@gdiepen
Copy link

gdiepen commented Sep 16, 2024

Issue description

During a pipenv install pyodbc with a Pipfile that has multiple sources it seems pipenv reorganizes the Pipfile to have all [[sources]] blocks at the top.

However, this reordering will fail if the last line is missing a trailing new-line

If you use the following Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.11"


[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi2"

(note that I am using twice pypi.org, in our case this would be a private registry). Note that this Pipfile should NOT have a trailing newline after the last name = "pypi2" line!

Expected result

Using the above mentioned Pipfile in the statement

`pipenv install --verbose --extra-pip-args "--dry-run" pyodbc

should just work without a problem

Actual result

Executing the statement
`pipenv install --verbose --extra-pip-args "--dry-run" pyodbc

with the above mentioned Pipfile results in the following error:

pipenv install --verbose --extra-pip-args "--dry-run" pyodbc
Requirements file provided! Importing into Pipfile...
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pipenv/project.py", line 700, in _parse_pipfile
    return tomlkit.parse(contents)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/api.py", line 86, in parse
    return Parser(string).parse()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 154, in parse
    key, value = self._parse_table()
                 ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1038, in _parse_table
    result = self._parse_aot(result, full_key)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1079, in _parse_aot
    _, table = self._parse_table(name_first)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 1002, in _parse_table
    item = self._parse_item()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 238, in _parse_item
    return self._parse_key_value(True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 334, in _parse_key_value
    cws, comment, trail = self._parse_comment_trail()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pipenv/vendor/tomlkit/parser.py", line 280, in _parse_comment_trail
    raise self.parse_error(UnexpectedCharError, c)
pipenv.vendor.tomlkit.exceptions.UnexpectedCharError: Unexpected character: '[' at line 9 col 14

If you check the Pipfile after this command has finished, you will see that pipenv has modified the original contents:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi2"[packages]
pyodbc = "*"

[dev-packages]

[requires]
python_version = "3.11"

As you can see, the [packages] section is started right after the name = "pypi2" key-value line (that originally did not have the trailing new-line)

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

See the earlier mentioned Pipfile (note: make sure you remove any trailing newline, e.g. with command truncate -s -1 Pipfile) and command to execute.


Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

$ pipenv --support

Pipenv version: '2024.0.2'

Pipenv location: '/usr/local/lib/python3.11/site-packages/pipenv'

Python location: '/usr/local/bin/python'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.133.1-microsoft-standard-WSL2',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Oct 5 21:02:42 UTC 2023',
 'python_full_version': '3.11.10',
 'python_version': '3.11',
 'sys_platform': 'linux'}

System environment variables:

  • PYTHONUNBUFFERED
  • HOSTNAME
  • PYTHON_VERSION
  • PWD
  • PYTHON_SETUPTOOLS_VERSION
  • container
  • HOME
  • LANG
  • LS_COLORS
  • GPG_KEY
  • https_proxy
  • TERM
  • SHLVL
  • HTTPS_PROXY
  • HTTP_PROXY
  • PYTHON_PIP_VERSION
  • http_proxy
  • PYTHONDONTWRITEBYTECODE
  • PYTHON_GET_PIP_SHA256
  • PYTHON_GET_PIP_URL
  • PATH
  • _
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • LANG: C.UTF-8
  • PWD: /application

Contents of Pipfile ('/application/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.11"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi2"

Contents of Pipfile.lock ('/application/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "f68fbcf7316b462719ef6a182db46870756b9b585113d062202aea794b67e3d1"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.11"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": false
            },
            {
                "name": "pypi2",
                "url": "https://pypi.org/simple",
                "verify_ssl": false
            }
        ]
    },
    "default": {
        "pyodbc": {
            "hashes": [
                "sha256:02fe9821711a2d14415eaeb4deab471d2c8b7034b107e524e414c0e133c42248",
                "sha256:1c5e0cb79222aad4b31a3602e39b242683c29c6221a16ed43f45f18fd0b73659",
                "sha256:218bb75d4bc67075529a65ce8ec7daeed1d83c33dd7410450fbf68d43d184d28",
                "sha256:29425e2d366e7f5828b76c7993f412a3db4f18bd5bcee00186c00b5a5965e205",
                "sha256:2cbdbd019756285dc44bc35238a3ed8dfaa454e8c8b2c3462f1710cfeebfb290",
                "sha256:33f0f1d7764cefef6f787936bd6359670828a6086be67518ab951f1f7f503cda",
                "sha256:33f4984af38872e7bdec78007a34e4d43ae72bf9d0bae3344e79d9d0db157c0e",
                "sha256:3602136a936bc0c1bb9722eb2fbf2042b3ff1ddccdc4688e514b82d4b831563b",
                "sha256:397feee44561a6580be08cedbe986436859563f4bb378f48224655c8e987ea60",
                "sha256:3c36448322f8d6479d87c528cf52401a6ea4f509b9637750b67340382b4e1b40",
                "sha256:406b8fa2133a7b6a713aa5187dba2d08cf763b5884606bed77610a7660fdfabe",
                "sha256:735f6da3762e5856b5580be0ed96bb946948346ebd1e526d5169a5513626a67a",
                "sha256:84df3bbce9bafe65abd25788d55c9f1da304f6115d70f25758ff8c85f3ce0517",
                "sha256:92caed9d445815ed3f7e5a1249e29a4600ebc1e99404df81b6ed7671074c9227",
                "sha256:96b2a8dc27693a517e3aad3944a7faa8be95d40d7ec1eda51a1885162eedfa33",
                "sha256:a1bd14633e91b7a9814f4fd944c9ebb89fb7f1fd4710c4e3999b5ef041536347",
                "sha256:a2bbd2e75c77dee9f3cd100c3246110abaeb9af3f7fa304ccc2934ff9c6a4fa4",
                "sha256:aa4e02d3a9bf819394510b726b25f1566f8b3f0891ca400ad2d4c8b86b535b78",
                "sha256:aa6f46377da303bf79bcb4b559899507df4b2559f30dcfdf191358ee4b99f3ab",
                "sha256:af5282cc8b667af97d76f4955250619a53f25486cbb6b1f45a06b781006ffa0b",
                "sha256:b0df69e3a500791b70b5748c68a79483b24428e4c16027b56aa0305e95c143a4",
                "sha256:b19d7f44cfee89901e482f554a88177e83fae76b03c3f830e0023a195d840220",
                "sha256:be3b1c36c31ec7d73d0b34a8ad8743573763fadd8f2bceef1e84408252b48dce",
                "sha256:bed1c843565d3a4fd8c332ebceaf33efe817657a0505eacb97dd1b786a985b0b",
                "sha256:c3b65343557f4c7753204e06f4c82c97ed212a636501f4bc27c5ce0e549eb3e8",
                "sha256:c5bb4e43f6c72f5fa2c634570e0d761767d8ea49f39205229b812fb4d3fe05aa",
                "sha256:d3d9cc4af703c4817b6e604315910b0cf5dcb68056d52b25ca072dd59c52dcbc",
                "sha256:e71a51c252b503b4d753e21ed31e640015fc0d00202d42ea42f2396fcc924b4a",
                "sha256:e738c5eedb4a0cbab20cc008882f49b106054499db56864057c2530ff208cf32",
                "sha256:eae576b3b67d21d6f237e18bb5f3df8323a2258f52c3e3afeef79269704072a9",
                "sha256:f8488c3818f12207650836c5c6f7352f9ff9f56a05a05512145995e497c0bbb1"
            ],
            "index": "pypi",
            "markers": "python_version >= '3.8'",
            "version": "==5.1.0"
        }
    },
    "develop": {}
}
@oz123
Copy link
Contributor

oz123 commented Sep 18, 2024

The exception is happening at the tomlkit parse. I wonder if there is a bug upstream for that.

What is also surprise is that we are loading pipfile via tomlkit and not via plette.

@gdiepen
Copy link
Author

gdiepen commented Sep 20, 2024

Did some more debugging, and you are right, not a problem on the side of pipenv.

Created simple example that reproduces the problem and created an issue in the tomlkit repo: python-poetry/tomlkit#381

@gdiepen gdiepen closed this as completed Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants