Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make BOLT as stable feature #101525

Open
5 of 7 tasks
corona10 opened this issue Feb 2, 2023 · 7 comments
Open
5 of 7 tasks

Make BOLT as stable feature #101525

corona10 opened this issue Feb 2, 2023 · 7 comments
Assignees
@corona10
Labels
build The build process and cross-build performance Performance or resource usage type-feature A feature request or enhancement

Comments

@corona10
Copy link
Member

corona10 commented Feb 2, 2023 •
edited by zanieb
Loading

Using a BOLT still not guarantee the unexpected behavior changes.
For example, I observed that #53093 is raised if the binary is optimized by BOLT.
We can reduce several too aggressive optimization.

cc @aaupov

Linked PRs
@corona10 corona10 added type-feature A feature request or enhancement performance Performance or resource usage labels Feb 2, 2023
@corona10 corona10 self-assigned this Feb 2, 2023
@corona10
Copy link
Member Author

corona10 commented Feb 2, 2023

I am analyzing several effects of BOLT.

@corona10
Copy link
Member Author

corona10 commented Feb 8, 2023

Need to update get_build_info for the BOLT binary.

@corona10
Copy link
Member Author

corona10 commented Feb 24, 2023 •
edited
Loading

Consider a way to detect post-bolted binary: llvm/llvm-project#60253

corona10 added a commit to corona10/cpython that referenced this issue Apr 2, 2023
@corona10
pythongh-101525: Disable peephole optimization process of BOLT
16f15a1
@bedevere-bot bedevere-bot mentioned this issue Apr 2, 2023
Merged
corona10 added a commit that referenced this issue Apr 5, 2023
@corona10
gh-101525: Disable peephole optimization process of BOLT (gh-103187)
a62ff97 
Co-authored-by: Dong-hee Na <donghee.na@linecorp.com>
gaogaotiantian pushed a commit to gaogaotiantian/cpython that referenced this issue Apr 8, 2023
@corona10 @gaogaotiantian
pythongh-101525: Disable peephole optimization process of BOLT (pytho…
1614139 
…ngh-103187)


Co-authored-by: Dong-hee Na <donghee.na@linecorp.com>
warsaw pushed a commit to warsaw/cpython that referenced this issue Apr 11, 2023
@corona10 @warsaw
pythongh-101525: Disable peephole optimization process of BOLT (pytho…
9ca78d7 
…ngh-103187)


Co-authored-by: Dong-hee Na <donghee.na@linecorp.com>
corona10 added a commit to corona10/cpython that referenced this issue Apr 16, 2023
@corona10
pythongh-101525: Fix make test if the --enable-bolt enabled
a9a36d5
@bedevere-bot bedevere-bot mentioned this issue Apr 16, 2023
Merged
corona10 added a commit that referenced this issue Apr 17, 2023
@corona10
gh-101525: Fix make test if the --enable-bolt enabled (gh-103574)
5d9762e
@corona10 corona10 mentioned this issue May 5, 2023
Open
indygreg added a commit to indygreg/cpython that referenced this issue May 20, 2023
@indygreg
pythongh-101282: rework the BOLT build process
f7851cd 
(This change is a quick and dirty way to merge some of the build system
improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze.
I wanted to scope bloat myself to fix some longstanding deficiencies in
the build system around profile-guided builds. But I'm getting soft
resistance to the reviews so close to the freeze deadline and it is
obvious that we need a simpler solution to hit the 3.12 deadline. While
this change is quick and dirty, it attempts to not make things worse.)

Before this change, we only applied bolt to the main python binary.
After this change, we apply bolt to libpython if it is configured. In
shared library builds, most of the C code is in libpython so it is
critical to apply bolt to libpython to realize bolt benefits.

This change also reworks how bolt instrumentation is applied. It
effectively removes the readelf based logic added in pythongh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform
bolt optimizations without having to manually delete the output binary
to force a new bolt run.

We also add a new make target for purging bolt files and hook it up
to `clean` so bolt state is purged when appropriate.

`.gitignore` rules have been added to ignore files related to bolt.

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need
to `rm profile-bolt-stamp` to force a bolt run because there aren't
appropriate non-phony targets for bolt's make target to depend on.
Fixing this is a non-trivial amount of work that will likely have to
wait until the 3.13 window.

To make it easier to iterate on custom BOLT settings, the flags to
pass to instrumentation and application are now defined in configure
and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and
`BOLT_APPLY_FLAGS`.
@indygreg indygreg mentioned this issue May 20, 2023
Merged
indygreg added a commit to indygreg/cpython that referenced this issue May 21, 2023
@indygreg
pythongh-101282: rework the BOLT build process
555686d 
(This change is a quick and dirty way to merge some of the build system
improvements I'm proposing in pythongh-101093 before the 3.12 feature freeze.
I wanted to scope bloat myself to fix some longstanding deficiencies in
the build system around profile-guided builds. But I'm getting soft
resistance to the reviews so close to the freeze deadline and it is
obvious that we need a simpler solution to hit the 3.12 deadline. While
this change is quick and dirty, it attempts to not make things worse.)

Before this change, we only applied bolt to the main python binary.
After this change, we apply bolt to libpython if it is configured. In
shared library builds, most of the C code is in libpython so it is
critical to apply bolt to libpython to realize bolt benefits.

This change also reworks how bolt instrumentation is applied. It
effectively removes the readelf based logic added in pythongh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform
bolt optimizations without having to manually delete the output binary
to force a new bolt run.

We also add a new make target for purging bolt files and hook it up
to `clean` so bolt state is purged when appropriate.

`.gitignore` rules have been added to ignore files related to bolt.

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need
to `rm profile-bolt-stamp` to force a bolt run because there aren't
appropriate non-phony targets for bolt's make target to depend on.
Fixing this is a non-trivial amount of work that will likely have to
wait until the 3.13 window.

To make it easier to iterate on custom BOLT settings, the flags to
pass to instrumentation and application are now defined in configure
and can be overridden by passing `BOLT_INSTRUMENT_FLAGS` and
`BOLT_APPLY_FLAGS`.
erlend-aasland pushed a commit that referenced this issue May 22, 2023
@indygreg
gh-101282: Apply BOLT optimizations to libpython for shared builds (#…
5360cb3 
…104709)

Apply BOLT optimizations to libpython for shared builds. Most of the C
code is in libpython so it is critical to apply BOLT there fully realize
BOLT benefits.

This change also reworks how BOLT instrumentation is applied. It
effectively removes the readelf based logic added in gh-101525 and
replaces it with a mechanism that saves a copy of the pre-bolt binary
and restores that copy when necessary. This allows us to perform BOLT
optimizations without having to manually delete the output binary to
force a new bolt run.

Also:
- add a clean-bolt target for purging BOLT files and hook that up to the
  clean target
- .gitignore BOLT related files

Before and after this refactor, `make` will no-op after a previous run.
Both versions should also share common make DAG deficiencies where
targets fail to trigger as often as they need to or can trigger
prematurely in certain scenarios. e.g. after this change you may need to
`rm profile-bolt-stamp` to force a BOLT run because there aren't
appropriate non-phony targets for BOLT's make target to depend on.

To make it easier to iterate on custom BOLT settings, the flags to pass
to instrumentation and application are now defined in configure and can
be overridden by passing BOLT_INSTRUMENT_FLAGS and BOLT_APPLY_FLAGS.
@corona10
Copy link
Member Author

faster-cpython/ideas#598

@iritkatriel iritkatriel added the build The build process and cross-build label Nov 29, 2023
corona10 added a commit to corona10/cpython that referenced this issue May 4, 2024
@corona10
pythongh-101525: Skip test_gdb if the binary is BOLTed.
28b67ad
@bedevere-app bedevere-app bot mentioned this issue May 4, 2024
Merged
@corona10
Copy link
Member Author

corona10 commented May 4, 2024

Need to use cdsort instead of hfsort+
https://github.com/llvm/llvm-project/blob/cb015b9ec9446b3a1303980c095fa442d5e46fbf/bolt/lib/Passes/ReorderFunctions.cpp#L57-L58

corona10 added a commit to corona10/cpython that referenced this issue Aug 31, 2024
@corona10
pythongh-101525: Skip test_gdb if the binary is BOLTed.
8755316
corona10 added a commit that referenced this issue Sep 2, 2024
@corona10
gh-101525: Skip test_gdb if the binary is relocated by BOLT. (gh-118572)
f95fc4d
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Sep 2, 2024
@corona10 @miss-islington
pythongh-101525: Skip test_gdb if the binary is relocated by BOLT. (p…
da1b359 
…ythongh-118572)

(cherry picked from commit f95fc4d)

Co-authored-by: Donghee Na <donghee.na@python.org>
@bedevere-app bedevere-app bot mentioned this issue Sep 2, 2024
Merged
corona10 added a commit that referenced this issue Sep 2, 2024
@miss-islington @corona10
[3.13] gh-101525: Skip test_gdb if the binary is relocated by BOLT. (g…
3b3a1a8 
…h-118572) (#123601)

gh-101525: Skip test_gdb if the binary is relocated by BOLT. (gh-118572)
(cherry picked from commit f95fc4d)

Co-authored-by: Donghee Na <donghee.na@python.org>
corona10 added a commit to corona10/cpython that referenced this issue Sep 2, 2024
@corona10
[3.12] pythongh-101525: Skip test_gdb if the binary is relocated by B…
1ba223c 
…OLT. (pythongh-118572)

(cherry picked from commit f95fc4d)

Co-authored-by: Donghee Na <donghee.na@python.org>
@bedevere-app bedevere-app bot mentioned this issue Sep 2, 2024
Merged
corona10 added a commit that referenced this issue Sep 2, 2024
@corona10
[3.12] gh-101525: Skip test_gdb if the binary is relocated by BOLT. (… (
4514998 
#123603)

[3.12] gh-101525: Skip test_gdb if the binary is relocated by BOLT. (gh-118572)
(cherry picked from commit f95fc4d)
This was referenced Jan 5, 2025
Open
Closed
@zanieb
Copy link
Contributor

zanieb commented Feb 9, 2025

Let's talk about next steps for stabilization here? We have testing in CI now and I think x86-64 can be considered stable?

@corona10
Copy link
Member Author

Yeah, I think that we can close this issue at this moment :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@corona10 corona10

Labels
build The build process and cross-build performance Performance or resource usage type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@iritkatriel @zanieb @corona10