Skip to content

Recreate venv for each build + install from pylock.toml when available - #345

Merged
hugovk merged 7 commits into
python:mainfrom
hugovk:clear-and-pylock
Jul 30, 2026
Merged

Recreate venv for each build + install from pylock.toml when available#345
hugovk merged 7 commits into
python:mainfrom
hugovk:clear-and-pylock

Conversation

@hugovk

@hugovk hugovk commented Jul 28, 2026

Copy link
Copy Markdown
Member

Re: python/cpython#153227 (comment)

python/cpython#153228 switched to use a dev version of pygments from GitHub, but it's not being picked up on the doc server.

This is because:

  • We re-use a venv per Python version on the docs server
  • pip install --upgrade -rrequirements.txt still sees pygments with the same version number in the venv, so doesn't need to upgrade it

This PR does two things:

  • Clear the venv for each build. We still have a pip cache on the server, so it's not downloading everything every time. This means we'll always a fresh venv matching what we expect.
  • We now have pylock.toml on newer branches (3.15+), which locks the actual required version, so this explicitly pin the dev version from GitHub. Let's use pylock.toml when when available.

Comment thread build_docs.py
venv_path,
symlinks=os.name != "nt",
with_pip=True,
clear=True,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensures a fresh venv.

Comment thread build_docs.py
symlinks=os.name != "nt",
with_pip=True,
clear=True,
upgrade_deps=True,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgrade pip, because otherwise we use the bundled pip, and we need at least 25.1 for pylock.toml support.

Could do this just for the pylock.toml part, but might as well update for both.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a little clarification for anyone reading this in the future, it needs 26.1 for pylock.toml support.

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm definitely not the most qualified reviewer here, but nothing jumps out at me as wrong (and I really want the highlighting 😃)

@hugovk

hugovk commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

You'll be happy to hear that Discourse (which uses highlight.js) will also get lazy highlighting at some point :)

highlightjs/highlight.js#4450

@ZeroIntensity

Copy link
Copy Markdown
Member

Hugo, do you realize I was the one who implemented that 😅: highlightjs/highlight.js#4398

But yeah, I think GitHub uses hljs too. Hopefully we see adoption before the 3.15 release.

Comment thread build_docs.py Outdated
Comment thread build_docs.py Outdated

So we can reuse them from builds to builds, while they contain
different Sphinx versions.
The venv is recreated from scratch for every build: pip considers

@StanFromIreland StanFromIreland Jul 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll now be build[ing]_venv[s] for every version, but also every language. That's quite a lot considering the amount of translations we have. I'd rather avoid the rebuilds, I think running it once per version will keep the venv fresh enough?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, good idea, updated to create each venv at most once per run.

It also now deletes the venvs at the end of the run, we don't need to keep those hanging around, especially as they're recreated at the start of a run.

How does it look now?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks great!

Comment thread build_docs.py Outdated
Co-authored-by: Stan Ulbrych <stan@python.org>
@StanFromIreland

Copy link
Copy Markdown
Member

Our Integration Test currently runs on 3.14, any objections to also running it for main (maybe this was tried previously, but was found brittle)?

@hugovk

hugovk commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

Our Integration Test currently runs on 3.14, any objections to also running it for main (maybe this was tried previously, but was found brittle)?

I don't think we tried it before, it was added last year: #275.

No objection, let's see how it goes.

@hugovk
hugovk merged commit 9822eee into python:main Jul 30, 2026
13 checks passed
@hugovk
hugovk deleted the clear-and-pylock branch July 30, 2026 20:30
@StanFromIreland

Copy link
Copy Markdown
Member

I'll write a patch.

@hugovk

hugovk commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

I've confirmed this is creating new venv dirs for 3.13-3.16, like venv-3.15-only-html-en below, when it's rebuilding:

docsbuild@docs:/srv/docsbuild$ du -sh venv*
32M	venv
299M	venv-3.10
297M	venv-3.11
294M	venv-3.12
299M	venv-3.13
299M	venv-3.14
299M	venv-3.15
300M	venv-3.15-only-html-en
299M	venv-3.16

And then tidying up afterwards:

docsbuild@docs:/srv/docsbuild$ du -sh venv*
32M	venv
299M	venv-3.10
297M	venv-3.11
294M	venv-3.12
299M	venv-3.13
299M	venv-3.14
299M	venv-3.15
299M	venv-3.16

I've moved venv-3.13 - venv-3.16 to /tmp for now, and we can delete them later, and also the older ones.

@hugovk

hugovk commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

Translations also working:

https://docs.python.org/pl/3.16/whatsnew/3.15.html#whatsnew315-lazy-imports
https://docs.python.org/fr/3.16/whatsnew/3.15.html#whatsnew315-lazy-imports

I've now deleted all the old venv*s.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants