[metasrv] feature: exchange protocol version with client #5645

drmingdrmer · 2022-05-28T07:28:14Z

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

[metasrv] feature: exchange protocol version with client

Before transfer any data, MetaGrpcClient must ensure it is compatible
with the metasrv it is connecting to, in the handshake RPC.

Add compatibility cross test:

latest query -- latest metasrv
old    query -- latest metasrv
latest query -- old    metasrv

Features

Compatibility test: download test suite for query.

When testing an old query with latest metasrv,
download the test suite for that version of query instead of using
the latest suite, which may not pass if new feature test is added
into the latest suite but not yet impl-ed in the old query.
Compatibility test:

Test the compatibility between:
- The latest query and the latest metasrv.
- The oldest compatible query with latest metasrv.
- The latest query with oldest compatible metasrv.
Protocol version exchange when meta-client handshake with metasrv.

meta-client(query) keeps track of the min-metasrv version it is compatible
with.

metasrv keeps track of the min-meta-client version it is compatible
with.

Client handshake check these versions to ensure compatibility.
Otherwise an error is returned and no further data will be transferred.
Add admin CLI command --cmd ver.

databend-query --cmd ver outputs its build version and the minimal
compatible metasrv version, such as:
```
version: 0.7.61-nightly
min-compatible-metasrv-version: 0.7.59
```
databend-meta --cmd ver outputs its build version and the minimal
compatible meta-client(query) version, such as:
```
version: 0.7.61-nightly
min-compatible-client-version: 0.7.57
```

Refactors

Add a new error InvalidArgument to indicate protocol-error
MetaGrpcClient::try_new() does not need to be async.
Metasrv config does not check() if --cmd is non-empty:

When cmd is specified, it does not mean to run it.

Docs

Add doc explain how to check versions and how to upgrade.

Changelog

New Feature

Related Issues

vercel · 2022-05-28T07:28:20Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated
databend	✅ Ready (Inspect)	Visit Preview	May 29, 2022 at 2:09AM (UTC)

mergify · 2022-05-28T07:28:45Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

Before transfer any data, `MetaGrpcClient` must ensure it is compatible with the metasrv it is connecting to, in the handshake RPC. Add compatibility cross test: ``` latest query -- latest metasrv old query -- latest metasrv latest query -- old metasrv ``` Features - Compatibility test: download test suite for query. When testing an old query with latest metasrv, download the test suite for that version of query instead of using the latest suite, which may not pass if new feature test is added into the latest suite but not yet impl-ed in the old query. - Compatibility test: Test the compatibility between: - The latest query and the latest metasrv. - The oldest compatible query with latest metasrv. - The latest query with oldest compatible metasrv. - Protocol version exchange when meta-client `handshake` with metasrv. meta-client(query) keeps track of the min-metasrv version it is compatible with. metasrv keeps track of the min-meta-client version it is compatible with. Client handshake check these versions to ensure compatibility. Otherwise an error is returned and no further data will be transferred. - Add admin CLI command `--cmd ver`. `databend-query --cmd ver` outputs its build version and the minimal compatible metasrv version, such as: ``` version: 0.7.61-nightly min-compatible-metasrv-version: 0.7.59 ``` `databend-meta --cmd ver` outputs its build version and the minimal compatible meta-client(query) version, such as: ``` version: 0.7.61-nightly min-compatible-client-version: 0.7.57 ``` Refactors - Add a new error `InvalidArgument` to indicate protocol-error - `MetaGrpcClient::try_new()` does not need to be `async`. - Metasrv config does not `check()` if `--cmd` is non-empty: When `cmd` is specified, it does not mean to run it. Docs - Add doc explain how to check versions and how to upgrade. --- - Fix: databendlabs#5627 - Fix: databendlabs#4948

BohuTANG · 2022-05-28T08:09:40Z

Cool.
If we build from source with out git information, what happenes during metasrv and query?

drmingdrmer · 2022-05-28T09:02:08Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

The commit-version will be missing and the query won't be able to connect to metasrv.

drmingdrmer · 2022-05-28T09:38:47Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

BTW I've once considered using the version defined in Cargo.toml for building a release.

[package]
name = "databend-query"
version = "0.7.0"

E.g., if the version in Cargo is 0.7.0, then a nightly build version could be 0.7.0+20220528.

Then the git-tag triggered release action only is used for formal releases.

BohuTANG · 2022-05-28T10:48:30Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

The commit-version will be missing and the query won't be able to connect to metasrv.

I think this is OK.

drmingdrmer · 2022-05-28T13:35:23Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

The commit-version will be missing and the query won't be able to connect to metasrv.

I think this is OK.

Hmm... looks like CI failed due to this issue. In the CI it just checkout one last commit without any tags. The version can not be determined.

It can be worked around by using a special protocol-version == 0, which I do not like. 🤔

BohuTANG · 2022-05-28T13:39:10Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

The commit-version will be missing and the query won't be able to connect to metasrv.

I think this is OK.

Hmm... looks like CI failed due to this issue. In the CI it just checkout one last commit without any tags. The version can not be determined.

It can be worked around by using a special protocol-version == 0, which I do not like. 🤔

@everpcpc fixed it, may help here

everpcpc · 2022-05-28T14:30:40Z

.github/actions/test_compat/action.yml

@@ -15,6 +15,10 @@ runs:
    - name: Maximize build space
      uses: ./.github/actions/cleanup

+    - name: Rust setup


dev_setup is only needed on macos.
on linux ci, setup_build_tool already makes it working.

It complains in test-compat.sh that wget is not found.
What did I do wrongly?

I was trying to run it with build-tool:
build-tool bash ./tests/compat/test-compat.sh

wget should be already provided by github-action.

ah, it's not in the build-tool image now.
build-tool command will run command in container with image datafuselabs/build-tool:dev
maybe you can use curl instead.

That's all right! Thanks!

everpcpc · 2022-05-28T14:36:37Z

Cool. If we build from source with out git information, what happenes during metasrv and query?

The commit-version will be missing and the query won't be able to connect to metasrv.

I think this is OK.

Hmm... looks like CI failed due to this issue. In the CI it just checkout one last commit without any tags. The version can not be determined.
It can be worked around by using a special protocol-version == 0, which I do not like. 🤔

@everpcpc fixed it, may help here

should be fixed with: https://github.com/datafuselabs/databend/pull/5520/files#r884142502

drmingdrmer · 2022-05-28T14:39:02Z

should be fixed with: https://github.com/datafuselabs/databend/pull/5520/files#r884142502

Is it appropriate to fetch the full history?
The fetching time would be longer then.

everpcpc · 2022-05-28T14:39:14Z

.github/actions/test_compat/action.yml

+    - name: Rust setup
+      run: |
+        bash ./scripts/setup/dev_setup.sh -yb
+
    - name: Setup Build Tool
      uses: ./.github/actions/setup_build_tool



installing python libs is also not need, and already done by build tool image

everpcpc · 2022-05-28T14:42:57Z

should be fixed with: https://github.com/datafuselabs/databend/pull/5520/files#r884142502

Is it appropriate to fetch the full history? The fetching time would be longer then.

we are already doing this with release build, and the checkout takes around 10s which seems fine.

… query and latest meta, it is covered by other tests

… that are already included in build-tool

databend-bot added the need-review label May 28, 2022

mergify bot added the pr-feature this PR introduces a new feature to the codebase label May 28, 2022

drmingdrmer marked this pull request as ready for review May 28, 2022 07:29

drmingdrmer force-pushed the 30-protocol-ver branch from 8cd102a to 262575f Compare May 28, 2022 07:30

drmingdrmer requested review from Xuanwo, ariesdevil, lichuang, flaneur2020 and ZhiHanZ May 28, 2022 07:30

vercel bot deployed to Preview May 28, 2022 07:33 View deployment

drmingdrmer added 2 commits May 28, 2022 15:35

fixup: lint

c9fef8f

drmingdrmer force-pushed the 30-protocol-ver branch from 262575f to c9fef8f Compare May 28, 2022 07:56

lichuang approved these changes May 28, 2022

View reviewed changes

BohuTANG approved these changes May 28, 2022

View reviewed changes

fixup: default_config test and print semver parsing error

efbc3d3

Merge branch 'main' into 30-protocol-ver

f1e356c

everpcpc reviewed May 28, 2022

View reviewed changes

drmingdrmer added 6 commits May 29, 2022 10:06

fixup: test-compat does not need to test compatibility between latest…

1eed356

… query and latest meta, it is covered by other tests

fixup: run dev_setup before test-compat

55fbd88

fixup: use curl instead of wget in image build-tool

e6c9722

fixup: fetch all history in CI to get the version for a build

31ccc57

fixup: according to @everpcpc suggestion, remove steps installing lib…

739e789

… that are already included in build-tool

Merge remote-tracking branch 'ups/main' into 30-protocol-ver

346c102

drmingdrmer force-pushed the 30-protocol-ver branch from 1f62512 to 346c102 Compare May 29, 2022 02:07

vercel bot deployed to Preview May 29, 2022 02:09 View deployment

mergify bot merged commit 7cf4440 into databendlabs:main May 29, 2022

drmingdrmer deleted the 30-protocol-ver branch May 29, 2022 02:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metasrv] feature: exchange protocol version with client #5645

[metasrv] feature: exchange protocol version with client #5645

drmingdrmer commented May 28, 2022

vercel bot commented May 28, 2022 •

edited

Loading

mergify bot commented May 28, 2022

BohuTANG commented May 28, 2022

drmingdrmer commented May 28, 2022

drmingdrmer commented May 28, 2022

BohuTANG commented May 28, 2022

drmingdrmer commented May 28, 2022

BohuTANG commented May 28, 2022

everpcpc May 28, 2022 •

edited

Loading

drmingdrmer May 28, 2022

everpcpc May 28, 2022 •

edited

Loading

drmingdrmer May 28, 2022

everpcpc commented May 28, 2022

drmingdrmer commented May 28, 2022

everpcpc May 28, 2022

everpcpc commented May 28, 2022

[metasrv] feature: exchange protocol version with client #5645

[metasrv] feature: exchange protocol version with client #5645

Conversation

drmingdrmer commented May 28, 2022

Summary

[metasrv] feature: exchange protocol version with client

Changelog

Related Issues

vercel bot commented May 28, 2022 • edited Loading

mergify bot commented May 28, 2022

BohuTANG commented May 28, 2022

drmingdrmer commented May 28, 2022

drmingdrmer commented May 28, 2022

BohuTANG commented May 28, 2022

drmingdrmer commented May 28, 2022

BohuTANG commented May 28, 2022

everpcpc May 28, 2022 • edited Loading

Choose a reason for hiding this comment

drmingdrmer May 28, 2022

Choose a reason for hiding this comment

everpcpc May 28, 2022 • edited Loading

Choose a reason for hiding this comment

drmingdrmer May 28, 2022

Choose a reason for hiding this comment

everpcpc commented May 28, 2022

drmingdrmer commented May 28, 2022

everpcpc May 28, 2022

Choose a reason for hiding this comment

everpcpc commented May 28, 2022

vercel bot commented May 28, 2022 •

edited

Loading

everpcpc May 28, 2022 •

edited

Loading

everpcpc May 28, 2022 •

edited

Loading