Pipfile sections are sorted before adding packages with extras

[aradkdj]

Issue description

When sort_pipfile is set to ture, if you install a new package with extras, it gets put at the end of the respective section instead of being sorted. If you previously had a package with extras at the end of the section, that one will get sorted but the new package with extras you're installing will still be put at the end of the section.

Expected result

The recently installed package gets sorted within its section.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
psycopg = {extras = ["c"], version = "*"}
zenpy = "*"

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

Actual result

The recently installed package gets put at the end of the section.

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"
psycopg = {extras = ["c"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

Steps to replicate

1. Start with a sorted Pipfile such as

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

2. Add a new package with extras (pipenv install psycopg[c]). It will get put at the end of the section instead of between asgiref and zenpy.

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"
psycopg = {extras = ["c"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

3. (Optional) add another package with extras (pipenv install sqlalchemy[async]). The previous package with extras will be sorted and this new one will be put at the end of the section.

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
psycopg = {extras = ["c"], version = "*"}
zenpy = "*"
sqlalchemy = {extras = ["async"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

---

$ pipenv --support

Pipenv version: '2023.12.1'

Pipenv location: '/home/user/.local/lib/python3.12/site-packages/pipenv'

Python location: '/usr/bin/python3'

OS Name: 'posix'

User pip version: '23.3.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.8.11-300.fc40.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT_DYNAMIC Mon May 27 14:53:33 UTC 2024',
 'python_full_version': '3.12.3',
 'python_version': '3.12',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• SESSION_MANAGER
• SNAP_REVISION
• PYENV_SHELL
• HISTCONTROL
• XDG_MENU_PREFIX
• SNAP_REAL_HOME
• TERMINAL_EMULATOR
• SNAP_USER_COMMON
• HOSTNAME
• HISTSIZE
• SSH_AUTH_SOCK
• MEMORY_PRESSURE_WRITE
• TERM_SESSION_ID
• SNAP_INSTANCE_KEY
• XMODIFIERS
• DESKTOP_SESSION
• BAMF_DESKTOP_FILE_HINT
• EDITOR
• SNAP_EUID
• PWD
• XDG_SESSION_DESKTOP
• LOGNAME
• XDG_SESSION_TYPE
• SYSTEMD_EXEC_PID
• XAUTHORITY
• DESKTOP_STARTUP_ID
• SDL_VIDEO_MINIMIZE_ON_FOCUS_LOSS
• SNAP_CONTEXT
• GJS_DEBUG_TOPICS
• GDM_LANG
• HOME
• USERNAME
• LANG
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VIRTUAL_ENV
• MEMORY_PRESSURE_WATCH
• WAYLAND_DISPLAY
• SNAP_ARCH
• SNAP_INSTANCE_NAME
• SNAP_USER_DATA
• INVOCATION_ID
• MANAGERPID
• SNAP_REEXEC
• SNAP_UID
• STEAM_FRAME_FORCE_CLOSE
• GJS_DEBUG_OUTPUT
• GNOME_SETUP_DISPLAY
• XDG_ACTIVATION_TOKEN
• XDG_SESSION_CLASS
• TERM
• LESSOPEN
• USER
• SNAP
• SNAP_COMMON
• SNAP_VERSION
• DISPLAY
• SHLVL
• SNAP_LIBRARY_PATH
• SNAP_COOKIE
• QT_IM_MODULE
• VIRTUAL_ENV_PROMPT
• SNAP_DATA
• XDG_RUNTIME_DIR
• PS1
• PYENV_ROOT
• DEBUGINFOD_URLS
• SNAP_NAME
• JOURNAL_STREAM
• XDG_DATA_DIRS
• PATH
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• MAIL
• GIO_LAUNCHED_DESKTOP_FILE_PID
• GIO_LAUNCHED_DESKTOP_FILE
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/user/.pyenv/versions/3.10.14/bin:/home/user/.local/share/virtualenvs/pythonProject-bFLmJOh0/bin:/home/user/.pyenv/shims:/home/user/.pyenv/bin:/home/user/.local/bin:/home/user/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin
• SHELL: /bin/bash
• EDITOR: /usr/bin/nano
• LANG: en_US.UTF-8
• PWD: /home/user/Documents/learning/pythonProject
• VIRTUAL_ENV: /home/user/.local/share/virtualenvs/pythonProject-bFLmJOh0

---

Contents of Pipfile ('/home/user/Documents/learning/pythonProject/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[pipenv]
sort_pipfile = true

[packages]
asgiref = "*"
zenpy = "*"
psycopg = {extras = ["c"], version = "*"}

[dev-packages]

[requires]
python_version = "3.10"
python_full_version = "3.10.14"

Contents of Pipfile.lock ('/home/user/Documents/learning/pythonProject/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "f89b0e17c8d001893b901878be733a0705afc39a6cd04041a6ddffdfee693e5b"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_full_version": "3.10.14",
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "asgiref": {
            "hashes": [
                "sha256:3e1e3ecc849832fe52ccf2cb6686b7a55f82bb1d6aee72a58826471390335e47",
                "sha256:c343bd80a0bec947a9860adb4c432ffa7db769836c64238fc34bdc3fec84d590"
            ],
            "index": "pypi",
            "markers": "python_version >= '3.8'",
            "version": "==3.8.1"
        },
        "cachetools": {
            "hashes": [
                "sha256:0abad1021d3f8325b2fc1d2e9c8b9c9d57b04c3932657a72465447332c24d945",
                "sha256:ba29e2dfa0b8b556606f097407ed1aa62080ee108ab0dc5ec9d6a723a007d105"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==5.3.3"
        },
        "certifi": {
            "hashes": [
                "sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516",
                "sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2024.6.2"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:06435b539f889b1f6f4ac1758871aae42dc3a8c0e24ac9e60c2384973ad73027",
                "sha256:06a81e93cd441c56a9b65d8e1d043daeb97a3d0856d177d5c90ba85acb3db087",
                "sha256:0a55554a2fa0d408816b3b5cedf0045f4b8e1a6065aec45849de2d6f3f8e9786",
                "sha256:0b2b64d2bb6d3fb9112bafa732def486049e63de9618b5843bcdd081d8144cd8",
                "sha256:10955842570876604d404661fbccbc9c7e684caf432c09c715ec38fbae45ae09",
                "sha256:122c7fa62b130ed55f8f285bfd56d5f4b4a5b503609d181f9ad85e55c89f4185",
                "sha256:1ceae2f17a9c33cb48e3263960dc5fc8005351ee19db217e9b1bb15d28c02574",
                "sha256:1d3193f4a680c64b4b6a9115943538edb896edc190f0b222e73761716519268e",
                "sha256:1f79682fbe303db92bc2b1136016a38a42e835d932bab5b3b1bfcfbf0640e519",
                "sha256:2127566c664442652f024c837091890cb1942c30937add288223dc895793f898",
                "sha256:22afcb9f253dac0696b5a4be4a1c0f8762f8239e21b99680099abd9b2b1b2269",
                "sha256:25baf083bf6f6b341f4121c2f3c548875ee6f5339300e08be3f2b2ba1721cdd3",
                "sha256:2e81c7b9c8979ce92ed306c249d46894776a909505d8f5a4ba55b14206e3222f",
                "sha256:3287761bc4ee9e33561a7e058c72ac0938c4f57fe49a09eae428fd88aafe7bb6",
                "sha256:34d1c8da1e78d2e001f363791c98a272bb734000fcef47a491c1e3b0505657a8",
                "sha256:37e55c8e51c236f95b033f6fb391d7d7970ba5fe7ff453dad675e88cf303377a",
                "sha256:3d47fa203a7bd9c5b6cee4736ee84ca03b8ef23193c0d1ca99b5089f72645c73",
                "sha256:3e4d1f6587322d2788836a99c69062fbb091331ec940e02d12d179c1d53e25fc",
                "sha256:42cb296636fcc8b0644486d15c12376cb9fa75443e00fb25de0b8602e64c1714",
                "sha256:45485e01ff4d3630ec0d9617310448a8702f70e9c01906b0d0118bdf9d124cf2",
                "sha256:4a78b2b446bd7c934f5dcedc588903fb2f5eec172f3d29e52a9096a43722adfc",
                "sha256:4ab2fe47fae9e0f9dee8c04187ce5d09f48eabe611be8259444906793ab7cbce",
                "sha256:4d0d1650369165a14e14e1e47b372cfcb31d6ab44e6e33cb2d4e57265290044d",
                "sha256:549a3a73da901d5bc3ce8d24e0600d1fa85524c10287f6004fbab87672bf3e1e",
                "sha256:55086ee1064215781fff39a1af09518bc9255b50d6333f2e4c74ca09fac6a8f6",
                "sha256:572c3763a264ba47b3cf708a44ce965d98555f618ca42c926a9c1616d8f34269",
                "sha256:573f6eac48f4769d667c4442081b1794f52919e7edada77495aaed9236d13a96",
                "sha256:5b4c145409bef602a690e7cfad0a15a55c13320ff7a3ad7ca59c13bb8ba4d45d",
                "sha256:6463effa3186ea09411d50efc7d85360b38d5f09b870c48e4600f63af490e56a",
                "sha256:65f6f63034100ead094b8744b3b97965785388f308a64cf8d7c34f2f2e5be0c4",
                "sha256:663946639d296df6a2bb2aa51b60a2454ca1cb29835324c640dafb5ff2131a77",
                "sha256:6897af51655e3691ff853668779c7bad41579facacf5fd7253b0133308cf000d",
                "sha256:68d1f8a9e9e37c1223b656399be5d6b448dea850bed7d0f87a8311f1ff3dabb0",
                "sha256:6ac7ffc7ad6d040517be39eb591cac5ff87416c2537df6ba3cba3bae290c0fed",
                "sha256:6b3251890fff30ee142c44144871185dbe13b11bab478a88887a639655be1068",
                "sha256:6c4caeef8fa63d06bd437cd4bdcf3ffefe6738fb1b25951440d80dc7df8c03ac",
                "sha256:6ef1d82a3af9d3eecdba2321dc1b3c238245d890843e040e41e470ffa64c3e25",
                "sha256:753f10e867343b4511128c6ed8c82f7bec3bd026875576dfd88483c5c73b2fd8",
                "sha256:7cd13a2e3ddeed6913a65e66e94b51d80a041145a026c27e6bb76c31a853c6ab",
                "sha256:7ed9e526742851e8d5cc9e6cf41427dfc6068d4f5a3bb03659444b4cabf6bc26",
                "sha256:7f04c839ed0b6b98b1a7501a002144b76c18fb1c1850c8b98d458ac269e26ed2",
                "sha256:802fe99cca7457642125a8a88a084cef28ff0cf9407060f7b93dca5aa25480db",
                "sha256:80402cd6ee291dcb72644d6eac93785fe2c8b9cb30893c1af5b8fdd753b9d40f",
                "sha256:8465322196c8b4d7ab6d1e049e4c5cb460d0394da4a27d23cc242fbf0034b6b5",
                "sha256:86216b5cee4b06df986d214f664305142d9c76df9b6512be2738aa72a2048f99",
                "sha256:87d1351268731db79e0f8e745d92493ee2841c974128ef629dc518b937d9194c",
                "sha256:8bdb58ff7ba23002a4c5808d608e4e6c687175724f54a5dade5fa8c67b604e4d",
                "sha256:8c622a5fe39a48f78944a87d4fb8a53ee07344641b0562c540d840748571b811",
                "sha256:8d756e44e94489e49571086ef83b2bb8ce311e730092d2c34ca8f7d925cb20aa",
                "sha256:8f4a014bc36d3c57402e2977dada34f9c12300af536839dc38c0beab8878f38a",
                "sha256:9063e24fdb1e498ab71cb7419e24622516c4a04476b17a2dab57e8baa30d6e03",
                "sha256:90d558489962fd4918143277a773316e56c72da56ec7aa3dc3dbbe20fdfed15b",
                "sha256:923c0c831b7cfcb071580d3f46c4baf50f174be571576556269530f4bbd79d04",
                "sha256:95f2a5796329323b8f0512e09dbb7a1860c46a39da62ecb2324f116fa8fdc85c",
                "sha256:96b02a3dc4381e5494fad39be677abcb5e6634bf7b4fa83a6dd3112607547001",
                "sha256:9f96df6923e21816da7e0ad3fd47dd8f94b2a5ce594e00677c0013018b813458",
                "sha256:a10af20b82360ab00827f916a6058451b723b4e65030c5a18577c8b2de5b3389",
                "sha256:a50aebfa173e157099939b17f18600f72f84eed3049e743b68ad15bd69b6bf99",
                "sha256:a981a536974bbc7a512cf44ed14938cf01030a99e9b3a06dd59578882f06f985",
                "sha256:a9a8e9031d613fd2009c182b69c7b2c1ef8239a0efb1df3f7c8da66d5dd3d537",
                "sha256:ae5f4161f18c61806f411a13b0310bea87f987c7d2ecdbdaad0e94eb2e404238",
                "sha256:aed38f6e4fb3f5d6bf81bfa990a07806be9d83cf7bacef998ab1a9bd660a581f",
                "sha256:b01b88d45a6fcb69667cd6d2f7a9aeb4bf53760d7fc536bf679ec94fe9f3ff3d",
                "sha256:b261ccdec7821281dade748d088bb6e9b69e6d15b30652b74cbbac25e280b796",
                "sha256:b2b0a0c0517616b6869869f8c581d4eb2dd83a4d79e0ebcb7d373ef9956aeb0a",
                "sha256:b4a23f61ce87adf89be746c8a8974fe1c823c891d8f86eb218bb957c924bb143",
                "sha256:bd8f7df7d12c2db9fab40bdd87a7c09b1530128315d047a086fa3ae3435cb3a8",
                "sha256:beb58fe5cdb101e3a055192ac291b7a21e3b7ef4f67fa1d74e331a7f2124341c",
                "sha256:c002b4ffc0be611f0d9da932eb0f704fe2602a9a949d1f738e4c34c75b0863d5",
                "sha256:c083af607d2515612056a31f0a8d9e0fcb5876b7bfc0abad3ecd275bc4ebc2d5",
                "sha256:c180f51afb394e165eafe4ac2936a14bee3eb10debc9d9e4db8958fe36afe711",
                "sha256:c235ebd9baae02f1b77bcea61bce332cb4331dc3617d254df3323aa01ab47bd4",
                "sha256:cd70574b12bb8a4d2aaa0094515df2463cb429d8536cfb6c7ce983246983e5a6",
                "sha256:d0eccceffcb53201b5bfebb52600a5fb483a20b61da9dbc885f8b103cbe7598c",
                "sha256:d965bba47ddeec8cd560687584e88cf699fd28f192ceb452d1d7ee807c5597b7",
                "sha256:db364eca23f876da6f9e16c9da0df51aa4f104a972735574842618b8c6d999d4",
                "sha256:ddbb2551d7e0102e7252db79ba445cdab71b26640817ab1e3e3648dad515003b",
                "sha256:deb6be0ac38ece9ba87dea880e438f25ca3eddfac8b002a2ec3d9183a454e8ae",
                "sha256:e06ed3eb3218bc64786f7db41917d4e686cc4856944f53d5bdf83a6884432e12",
                "sha256:e27ad930a842b4c5eb8ac0016b0a54f5aebbe679340c26101df33424142c143c",
                "sha256:e537484df0d8f426ce2afb2d0f8e1c3d0b114b83f8850e5f2fbea0e797bd82ae",
                "sha256:eb00ed941194665c332bf8e078baf037d6c35d7c4f3102ea2d4f16ca94a26dc8",
                "sha256:eb6904c354526e758fda7167b33005998fb68c46fbc10e013ca97f21ca5c8887",
                "sha256:eb8821e09e916165e160797a6c17edda0679379a4be5c716c260e836e122f54b",
                "sha256:efcb3f6676480691518c177e3b465bcddf57cea040302f9f4e6e191af91174d4",
                "sha256:f27273b60488abe721a075bcca6d7f3964f9f6f067c8c4c605743023d7d3944f",
                "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5",
                "sha256:fb69256e180cb6c8a894fee62b3afebae785babc1ee98b81cdf68bbca1987f33",
                "sha256:fd1abc0d89e30cc4e02e4064dc67fcc51bd941eb395c502aac3ec19fab46b519",
                "sha256:ff8fa367d09b717b2a17a052544193ad76cd49979c805768879cb63d9ca50561"
            ],
            "markers": "python_full_version >= '3.7.0'",
            "version": "==3.3.2"
        },
        "idna": {
            "hashes": [
                "sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc",
                "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.7"
        },
        "psycopg": {
            "extras": [
                "c"
            ],
            "hashes": [
                "sha256:92d7b78ad82426cdcf1a0440678209faa890c6e1721361c2f8901f0dccd62961",
                "sha256:dca5e5521c859f6606686432ae1c94e8766d29cc91f2ee595378c510cc5b0731"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==3.1.19"
        },
        "psycopg-c": {
            "hashes": [
                "sha256:8e90f53c430e7d661cb3a9298e2761847212ead1b24c5fb058fc9d0fd9616017"
            ],
            "version": "==3.1.19"
        },
        "python-dateutil": {
            "hashes": [
                "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3",
                "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==2.9.0.post0"
        },
        "pytz": {
            "hashes": [
                "sha256:2a29735ea9c18baf14b448846bde5a48030ed267578472d8955cd0e7443a9812",
                "sha256:328171f4e3623139da4983451950b28e95ac706e13f3f2630a879749e7a8b319"
            ],
            "version": "==2024.1"
        },
        "requests": {
            "hashes": [
                "sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760",
                "sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==2.32.3"
        },
        "six": {
            "hashes": [
                "sha256:1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
                "sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.16.0"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:6024b58b69089e5a89c347397254e35f1bf02a907728ec7fee9bf0fe837d203a",
                "sha256:915f5e35ff76f56588223f15fdd5938f9a1cf9195c0de25130c627e4d597f6d1"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.12.1"
        },
        "urllib3": {
            "hashes": [
                "sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d",
                "sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==2.2.1"
        },
        "zenpy": {
            "hashes": [
                "sha256:39303ce90a5cf8ed71e753fa5f0ae429ad3e52324e633c57e62cd8f987e3ef3f"
            ],
            "index": "pypi",
            "version": "==2.0.49"
        }
    },
    "develop": {}
}

[oz123]

Is this a bug report or a feature request?

[matteius]

I believe its a bug report with the sort_pipfile = true feature and dependencies with extras

[aradkdj]

It's a bug report, I already have a (currently failing) test case to validate it. I also could try and tackle it

[sanspareilsmyn]

@aradkdj @matteius @oz123 I had a dive on this issue and encountered several problems.

1. In pipenv/project.py, there's a logic like this. This handles sort_pipfile flag.

    def _sort_category(self, category):
        # toml tables won't maintain sorted dictionary order
        # so construct the table in the order that we need
        table = tomlkit.table()
        for key, value in sorted(category.items()):
            table.add(key, value)

        return table

2. tomlkit.table() has its own appending logic (append and _raw_append), which is implemented in vendor/tomlkit/container.py.

3. No matter how we sort the order before adding to tomlkit.table(), table.add() ruins sorted order.

4. I had some troubles figuring out which logic blocks us from sorting package keys in alphabetical orders.

[oz123]

@Naofal-Helal thank you for your contribution fixing this issue.