Python packaging

3.14

https://docs.python.org/3.14/whatsnew/3.14.html#id7

In the limited C API 3.14 and newer, Py_TYPE() and Py_REFCNT() are now implemented as an opaque function call to hide implementation details. (Contributed by Victor Stinner in gh-120600 and gh-124127.)

Freethreading

https://py-free-threading.github.io/porting/

Cleanups

  • drop python-pytest7
  • PEP517 MR's https://gitlab.archlinux.org/groups/archlinux/packaging/packages/-/merge_requests?scope=all&state=opened&search=PEP+517
  • drop python-nose
  • drop python-eventlet like Fedora
  • drop python-six
  • drop python-future
  • drop python-async-timeout https://github.com/Bluetooth-Devices/dbus-fast/blob/0a9e4c5a2140f20f580c67bed2a68ff0ac524b62/CHANGELOG.md?plain=1#L1497
    • python-redis
    • python-aioesphomeapi
    • python-zeroconf
    • python-aiopg
    • python-dbus-fast
    • esphome
    • python-asyncpg
    • python-jeepney
    • python-bleak
  • drop using python-pytest-cov in our tests
  • drop python-importlib-metdata
  • drop python-importlib_resources
  • fix packages with checkdepends without running check()
  • build with PEP517
  • backport package python-typing_extensions
  • backport package python-unicodedata2 => https://fonttools.readthedocs.io/en/latest/optional.html#fonttools-unicode
  • update packaging guidelines and remove nose

remove python-py

  • python-pytest-forked https://github.com/pytest-dev/pytest-forked/blob/master/setup.py#L22 - create bug report https://github.com/pytest-dev/pytest-forked/issues/88

nose

Convert packages to use either pytest or python -m unittest

  • run nose doctests with pytest

List generated with pacquery python-nose | jq -r '.[].required_by | join("\n* [ ] ")'

  • python-j2cli
  • vigra
  • python-repoze.lru
  • python-ipython-genutils
  • shadowsocks
  • python-hglib
  • python-xmltodict
  • python-agate
  • python-lockfile
  • python-sure
  • python-zope-deprecation
  • python-jaconv
  • python-vigra
  • yapf
  • python-passlib
  • python-flask-restful
  • python-pyqrcode
  • lutris
  • python-astor
  • python-xlib

Drop pytest-coverage as checkdepends

Either use -o addopts='' when possible or sed it out.

Major release

Arch's Python modules store the version number in the module path meaning they won't be picked up by a new Python release for example 3.11 => 3.12.

  • bump Python and rebuild it in a separate branch
  • bootstrap
  • find incompatible packages upfront?

Package: python-bootstrap-evil

Rebuild order

The python package repo has a script called genrebuild this should include all packages required for the rebuild:

Figuring out the order: (TODO: exclude bootstraped build packages)

./genrebuild > rebuild-list.txt
cat rebuild-list.txt | xargs expac -Sv %n | sort | uniq > final.txt

For some reason our files.db include old packages which are no longer in the repos, arch-rebuild-order hard fails on missing packages so we clean those out with an ugly expac hack.

We can use arch-rebuild-order, it does not handle cyclic depenendencies but should be good enough (tm):

arch-rebuild-order --no-reverse-depends $(cat ./final.txt)

Python bootstrapping

Custom repository:

https://pkgbuild.com/~jelle/python3.11

cp /usr/share/devtools/pacman-staging.conf /usr/share/devtools/pacman-python.conf

Edit the config file and add above [staging]

[python]
SigLevel = Optional
Server = https://pkgbuild.com/~jelle/python3.11

sudo ln -s /usr/bin/archbuild /usr/bin/python-x86_64-build

repo-add python.db.tar.gz *.pkg.tar.zst

sudo python-x86_64-build --  -- --nocheck

Bootstrappping

  1. First build python-bootstrap (from svn-packages) with Python 3.X
  2. Yeet the packages into a pacman repository
  3. Build flit-core with bootstrapped build and installer
  4. Build python-installer comment out the sphinx build and repo-add it
  5. Build python-packaging (requires build,installer,flit-core). HACK: PYTHONPATH=src python -m build -nw required by python-build!
  6. Build python-build comment out the sphinx build and repo-add it
  7. Build python-pyproject-hooks and repo-add it
  8. build python-jaraco.text (requirement for bootstrap build of setuptools)
  9. build python-setuptools => bootstrap python-jaraco.text and tons more...
  10. Or build python-setuptools with export PYTHONPATH=/usr/lib/python3.10/site-packages/
  11. Wheel needs jaraco.functools and shit..

Parsing metadata

Parsing METADATA from Python package

import sys

from packaging.metadata import parse_email, Metadata

raw, unparsed = parse_email(metadata)
parsed = Metadata.from_raw(raw)

pyversion = f"{sys.version_info.major}.{sys.version_info.minor}"
environment = {'python_version': pyversion}
deps = []
for dep in parsed.requires_dist:
    if dep.marker is None:
         deps.append(dep.name)
	 continue
    if 'python_version' in str(dep.marker) and dep.marker.evaluate(environment)
         deps.append(dep.name)
    # do something with extra, by filling in the "extra" env

But we also need to detect:

https://github.com/abravalheri/validate-pyproject/blob/5ea862ffb5a31f4611813f223a1f1c0977661196/src/validate_pyproject/remote.py#L14