Skip to content
This repository was archived by the owner on May 24, 2024. It is now read-only.

Commit dda0db0

Browse files
liuxueyangbeeenderhiguoxing
authored
[main] Add support for checking ABI compatibility (#16763)
- Migrate changes in PR #16298 here - Some fields in the struct are removed or renamed. We add those structs to the ignore list as they are very internally specific and not considered an ABI break. - If you want to ignore a struct, the `struct` keyword should be added before the struct name in the ignore file. Co-authored-by: Chen Mulong <chenmulong@gmail.com> Co-authored-by: Xing Guo <higuoxing@gmail.com>
1 parent fb5eb8d commit dda0db0

File tree

5 files changed

+272
-0
lines changed

5 files changed

+272
-0
lines changed

‎.abi-check/.images/download-report-from-gh-action.png

125 KB
Loading

‎.abi-check/7.0.0/postgres.symbols.ignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
ConfigureNamesInt_gp
2+
ConfigureNamesBool_gp
3+
ConfigureNamesEnum_gp
4+
UpdateOrAddAttributeEncodings

‎.abi-check/7.0.0/postgres.types.ignore

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
struct workfile_set
2+
struct HashJoinTableData
3+
struct AppendOnlyBlockDirectory
4+
struct AOCSFetchDescData
5+
struct AOCSInsertDescData
6+
struct AOCSWriteColumnDescData
7+
struct AppendOnlyFetchDescData
8+
struct AppendOnlyInsertDescData

‎.abi-check/README.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
# Check the compatibility of Greenplum ABI.
2+
3+
## Introduction
4+
5+
We use the [`abi-dumper`](https://github.com/lvc/abi-dumper) and [`abi-compliance-checker`](https://github.com/lvc/abi-compliance-checker/) to check the Greenplum's ABI. We also use the [GitHub action](../.github/workflows/greenplum-abi-tests.yml) to automate this job.
6+
7+
## Requirements
8+
9+
`abi-dumper` requires the binary being compiled with `-Og -g3`, hence the `CFLAGS` for configuration looks like:
10+
11+
```bash
12+
## GCC's maybe-uninitialized checker may produce false positives with different
13+
## levels of optimizations. To prevent building failures, we append the '-Wno-maybe-uninitialized'
14+
## to the $CFLAGS as well.
15+
CFLAGS='-Og -g3 -Wno-maybe-uninitialized' ./configure --with-xxx --with-yyy --with-zzz
16+
```
17+
18+
## Check the ABI's compatibility
19+
20+
Several binaries are shipped in Greenplum, e.g., `$GPHOME/bin/postgres`, `$GPHOME/lib/libpq.so`, etc. Since the `postgres` binary are referenced by many extensions, the ABI compatibility of it is the most important. The following steps illustrate how to check the ABI compatibility of the `postgres` binary.
21+
22+
1. Dump the ABI information of one `postgres` binary.
23+
```
24+
abi-dumper $GPHOME/bin/postgres -lver <version of the binary> -o <output file>
25+
```
26+
- `<version of the binary>`: The version of the binary. You can give it some reasonable name, e.g., `7.0.0` to indicate the binary is built from '7.0.0' tag.
27+
- `<output file>`: The file path for dumping the ABI information, e.g., `greenplum-7.0.0.dump`
28+
29+
2. Dump the ABI information of another `postgres` binary (same as the step 1).
30+
31+
3. Compare the ABI between these two binaries with `abi-compliance-checker`.
32+
```
33+
abi-compliance-checker \
34+
-lib <library name> \
35+
-old <output file from step1> \
36+
-new <output file from step2>
37+
```
38+
- `<library name>`: The name of the library, e.g., `postgres`.
39+
40+
4. By default, the `abi-compliance-checker` will produce an HTML web page and there will be detailed information about ABI changes.
41+
42+
## Ignore the "Safe ABI breaking change"
43+
44+
There might be "safe ABI breaking changes", e.g., some symbol being removed and not referenced by any extensions or programs. Here are steps on how to suppress such errors.
45+
46+
1. Add ignored symbols to `gpdb_src/.abi-check/<base version>/postgres.symbols.ignore` (one symbol per line).
47+
- `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `7.0.0` release and the latest `main`. The baseline version of Greenplum is `7.0.0`. See: [./7.0.0/postgres.symbols.ignore](./7.0.0/postgres.symbols.ignore)
48+
49+
2. Add ignored types to `gpdb_src/.abi-check/<base version>/postgres.types.ignore` (one type per line).
50+
- `<base version>`: The baseline version of Greenplum. If we want to ensure the ABI isn't broken between the `7.0.0` release and the latest `6X_STABLE`. The baseline version of Greenplum is `7.0.0`. See: [./7.0.0/postgres.types.ignore](./7.0.0/postgres.types.ignore). Add the `struct` keyword in front of the struct name if we need to ignore a struct type.
51+
52+
3. Pass these two files to `abi-compliance-checker` and it will produce a report in HTML format.
53+
```
54+
abi-compliance-checker -skip-symbols gpdb_src/.abi-check/<base version>/postgres.symbols.ignore \
55+
-skip-types gpdb_src/.abi-check/<base version>/postgres.types.ignore \
56+
-lib postgres \
57+
-old greenplum-<base version>.dump
58+
-new greenplum-new.dump
59+
```
60+
It will produce a ABI report in `./compat_reports/postgres/X_to_Y/compat_report.html`.
61+
62+
## View the ABI compatibility report
63+
64+
### View the report locally
65+
66+
You can either open the HTML report in your browser or dump it to stdout using `lynx -dump compat_reports/postgres/X_to_Y/compat_report.html`.
67+
68+
## View the report from GitHub Action
69+
70+
1. Navigate to the "Summary" page of the test.
71+
2. Click the report and download it.
72+
3. View the report as above.
73+
74+
![./.images/download-report-from-gh-action.png](./.images/download-report-from-gh-action.png)

‎.github/workflows/greenplum-abi-tests.yml

+186
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
name: Greenplum ABI Tests
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
paths:
7+
- 'concourse/scripts/**'
8+
- 'src/**'
9+
- '.github/workflows/**'
10+
- '.github/scripts/**'
11+
- '.abi-check/**'
12+
13+
push:
14+
branches:
15+
- main
16+
paths:
17+
- 'concourse/scripts/**'
18+
- 'src/**'
19+
- '.github/workflows/**'
20+
- '.github/scripts/**'
21+
- '.abi-check/**'
22+
23+
jobs:
24+
abi-dump-setup:
25+
runs-on: ubuntu-latest
26+
outputs:
27+
BASELINE_REF: ${{ steps.vars.outputs.BASELINE_REF }}
28+
BASELINE_VERSION: ${{ steps.vars.outputs.BASELINE_VERSION }}
29+
ABI_LIBS: ${{ steps.vars.outputs.ABI_LIBS }}
30+
ABI_HEADERS: ${{ steps.vars.outputs.ABI_HEADERS }}
31+
EXCEPTION_LISTS_COUNT: ${{ steps.check_exception_lists.outputs.EXCEPTION_LISTS_COUNT }}
32+
steps:
33+
- name: Fetch source
34+
uses: actions/checkout@v3
35+
36+
- name: Get Greenplum version variables
37+
id: vars
38+
run: |
39+
remote_repo='https://github.com/greenplum-db/gpdb.git'
40+
git ls-remote --tags --refs --sort='v:refname' $remote_repo '7.*' | grep -v '-' | tail -n 1 > baseline_version_ref
41+
baseline_ref=$(cat baseline_version_ref | awk '{print $1}')
42+
baseline_version=$(cat baseline_version_ref | awk '{print $2}')
43+
echo "BASELINE_REF=${baseline_ref}" | tee -a $GITHUB_OUTPUT
44+
echo "BASELINE_VERSION=${baseline_version#'refs/tags/'}" | tee -a $GITHUB_OUTPUT
45+
echo "ABI_LIBS=postgres" | tee -a $GITHUB_OUTPUT
46+
echo "ABI_HEADERS=." | tee -a $GITHUB_OUTPUT
47+
48+
- name: Check if exception list exists
49+
id: check_exception_lists
50+
run: |
51+
exception_lists_count=$(ls .abi-check/${{ steps.vars.outputs.BASELINE_VERSION }}/ 2> /dev/null | wc -l)
52+
echo "EXCEPTION_LISTS_COUNT=${exception_lists_count}" | tee -a $GITHUB_OUTPUT
53+
54+
- name: Upload symbol/type checking exception list
55+
if: steps.check_exception_lists.outputs.EXCEPTION_LISTS_COUNT != '0'
56+
uses: actions/upload-artifact@v3
57+
with:
58+
name: exception_lists
59+
path: '.abi-check/${{ steps.vars.outputs.BASELINE_VERSION }}/'
60+
61+
abi-dump:
62+
needs: abi-dump-setup
63+
runs-on: ubuntu-latest
64+
container: gcr.io/data-gpdb-public-images/gpdb7-rocky8-build
65+
strategy:
66+
matrix:
67+
name:
68+
- build-baseline
69+
- build-latest
70+
include:
71+
- name: build-baseline
72+
repo: greenplum-db/gpdb
73+
ref: ${{ needs.abi-dump-setup.outputs.BASELINE_VERSION }}
74+
- name: build-latest
75+
repo: ${{ github.repository }}
76+
ref: ${{ github.sha }}
77+
78+
steps:
79+
## FIXME: abi-dumper requires 'Universal Ctags' but the package manager only provides
80+
## 'Exuberant Ctags'.
81+
- name: Install universal-ctags.
82+
run: |
83+
wget 'https://github.com/universal-ctags/ctags-nightly-build/releases/download/2023.07.05%2Bafdae39c0c2e508d113cbc570f4635b96159840c/uctags-2023.07.05-linux-x86_64.tar.xz'
84+
tar -xf uctags-2023.07.05-linux-x86_64.tar.xz
85+
cp uctags-2023.07.05-linux-x86_64/bin/* /usr/bin/
86+
which ctags
87+
88+
- name: Download Greenplum source code
89+
uses: actions/checkout@v3
90+
with:
91+
repository: ${{ matrix.repo }}
92+
ref: ${{ matrix.ref }}
93+
submodules: recursive
94+
fetch-depth: 0 # Specify '0' to fetch all history for all branches and tags.
95+
path: gpdb_src
96+
97+
- name: Install abi-dumper
98+
run: |
99+
yum install -y epel-release
100+
yum install -y abi-dumper
101+
102+
- name: Build Greenplum
103+
run: |
104+
## TODO: Since abi-dumper requires debug info and it's hard to inject CFLAGS via the script for
105+
## releasing Greenplum, we have to manually configure it here. Probably we can improve it in future.
106+
pushd gpdb_src
107+
CC='gcc -m64' \
108+
CFLAGS='-Og -g3 -Wno-maybe-uninitialized' LDFLAGS='-Wl,--enable-new-dtags -Wl,--export-dynamic' \
109+
./configure --with-quicklz --disable-gpperfmon --with-gssapi --enable-mapreduce --enable-orafce --enable-ic-proxy \
110+
--enable-orca --with-libxml --with-pythonsrc-ext --with-uuid=e2fs --with-pgport=5432 --enable-tap-tests \
111+
--enable-debug-extensions --with-perl --with-python --with-openssl --with-pam --with-ldap --with-includes="" \
112+
--with-libraries="" --disable-rpath \
113+
--prefix=/usr/local/greenplum-db-devel \
114+
--mandir=/usr/local/greenplum-db-devel/man
115+
make -j`nproc` && make install
116+
117+
- name: Dump ABI
118+
run: |
119+
abi-dumper -lver ${{ matrix.ref }} -skip-cxx -public-headers /usr/local/greenplum-db-devel/include/${{ needs.abi-dump-setup.outputs.ABI_HEADERS }} -o postgres-${{ matrix.ref }}.abi /usr/local/greenplum-db-devel/bin/postgres
120+
121+
- name: Upload ABI files
122+
uses: actions/upload-artifact@v3
123+
with:
124+
name: ${{ matrix.name }}
125+
path: '*${{ matrix.ref }}.abi'
126+
127+
abi-compare:
128+
needs:
129+
- abi-dump-setup
130+
- abi-dump
131+
runs-on: ubuntu-latest
132+
container: gcr.io/data-gpdb-public-images/gpdb7-rocky8-build
133+
steps:
134+
- name: Download baseline
135+
uses: actions/download-artifact@v3
136+
with:
137+
name: build-baseline
138+
path: build-baseline/
139+
- name: Download latest
140+
uses: actions/download-artifact@v3
141+
with:
142+
name: build-latest
143+
path: build-latest/
144+
145+
- name: Download exception lists
146+
if: needs.abi-dump-setup.outputs.EXCEPTION_LISTS_COUNT != '0'
147+
uses: actions/download-artifact@v3
148+
with:
149+
name: exception_lists
150+
path: exception_lists/
151+
152+
- name: Install abi-compliance-checker and report viewer (lynx)
153+
run: |
154+
yum install -y epel-release
155+
yum install -y abi-compliance-checker
156+
yum install -y --enablerepo=powertools lynx
157+
158+
- name: Compare ABI
159+
run: |
160+
SKIP_POSTGRES_SYMBOLS_LIST="exception_lists/postgres.symbols.ignore"
161+
SKIP_POSTGRES_SYMBOLS_OPTION=""
162+
if [[ -f "$SKIP_POSTGRES_SYMBOLS_LIST" ]]; then
163+
SKIP_POSTGRES_SYMBOLS_OPTION="-skip-symbols ${SKIP_POSTGRES_SYMBOLS_LIST}"
164+
fi
165+
SKIP_POSTGRES_TYPES_LIST="exception_lists/postgres.types.ignore"
166+
SKIP_POSTGRES_TYPES_OPTION=""
167+
if [[ -f "$SKIP_POSTGRES_TYPES_LIST" ]]; then
168+
SKIP_POSTGRES_TYPES_OPTION="-skip-types ${SKIP_POSTGRES_TYPES_LIST}"
169+
fi
170+
abi-compliance-checker ${SKIP_POSTGRES_SYMBOLS_OPTION} \
171+
${SKIP_POSTGRES_TYPES_OPTION} \
172+
-lib postgres \
173+
-old build-baseline/postgres*.abi \
174+
-new build-latest/postgres*.abi
175+
176+
- name: Print out ABI report
177+
if: always()
178+
run: |
179+
lynx -dump $(find compat_reports/ | grep html)
180+
181+
- name: Upload ABI Comparison
182+
if: always()
183+
uses: actions/upload-artifact@v3
184+
with:
185+
name: compat-report-${{ github.sha }}
186+
path: compat_reports/

0 commit comments

Comments
 (0)