Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: metadata error #4890

Closed
1 of 2 tasks
oslet opened this issue Apr 17, 2022 · 8 comments · Fixed by #4901
Closed
1 of 2 tasks

bug: metadata error #4890

oslet opened this issue Apr 17, 2022 · 8 comments · Fixed by #4901
Assignees
@drmingdrmer
Labels
C-bug Category: something isn't working

Comments

@oslet
Copy link

oslet commented Apr 17, 2022

Search before asking

  • I had searched in the issues and found no similar issues.

Version

main:
commitid: d6fe2d1
tag: v0.7.20-nightly

What's Wrong?

2022-04-17T00:07:53.557980Z ERROR openraft::core: quit RaftCore::main on error err=StorageError(IO { source: StorageIOError { subject: Logs, verb: Read, source: common_meta_types::meta_storage_errors::MetaStorageError: serde json se/de error: Error("missing field `if_not_exists`", line: 1, column: 228), backtrace: "<disabled>" } })

How to Reproduce?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@oslet oslet added the C-bug Category: something isn't working label Apr 17, 2022
@BohuTANG
Copy link
Member

BohuTANG commented Apr 17, 2022 •
edited
Loading

Hi, @oslet

Thank you for the information.
What's your databend-query configuration and when your metadata created first time?

Is there any way to get the metadata version and time in the disk? cc @drmingdrmer

@drmingdrmer
Copy link
Member

It must be upgraded and the data format has been changed in:

@drmingdrmer drmingdrmer self-assigned this Apr 17, 2022
@drmingdrmer
Copy link
Member

This upgrade added field if_xxx_exist to metadata.
Let me find a way to solve this issue 🤔

@oslet
Copy link
Author

oslet commented Apr 17, 2022

It is recommended to add scheduled backup functions, such as hourly backup configuration to the object storage directory

@drmingdrmer
Copy link
Member

If you are using a single-node metasvr a backup may be necessary.

But a backup should be considered only for data durability. It may not help in case data format is changed.:(

@BohuTANG
Copy link
Member

@oslet

Agreed.
The metadata is small now(only table schema, user and fuse engine root snapshot), but it's very very important, I think the metasrv cluster for HA and Data Reliability, but this is not enough, the data maybe corruption by the metasrv bug.

I agree with you to backup metadata to object or other places with version, this is a solid last resort for the metasrv.

@BohuTANG
Copy link
Member

BohuTANG commented Apr 17, 2022 •
edited
Loading

metasrv also need to add some tests to check the version compatibly for the current version with the previous version if metadata changed.
cc @drmingdrmer

drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Apr 17, 2022
@drmingdrmer
feature: metasrv has to be compatible with 20220413-34e89c99e4e356327…
acc7dcb 
…18e9227f6549b3090eb0fb9

Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

- fix: databendlabs#4890
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Apr 17, 2022
@drmingdrmer
feature: metasrv has to be compatible with 20220413-34e89c99e4e356327…
9508495 
…18e9227f6549b3090eb0fb9

Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

- fix: databendlabs#4890
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Apr 17, 2022
@drmingdrmer
feature: metasrv has to be compatible with 20220413-34e89c99e4e356327…
223e6cf 
…18e9227f6549b3090eb0fb9

Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

To solve th compatible issue, an intermedia type is introduced, which is
a super set of the old and new type.
When loading a record, first load it into the superset type, then reduce
to the latest type.

- fix: databendlabs#4890
@drmingdrmer drmingdrmer mentioned this issue Apr 17, 2022
Merged
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Apr 17, 2022
@drmingdrmer
feature: metasrv has to be compatible with 20220413-34e89c99e4e356327…
78097ba 
…18e9227f6549b3090eb0fb9

Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

To solve th compatible issue, an intermedia type is introduced, which is
a super set of the old and new type.
When loading a record, first load it into the superset type, then reduce
to the latest type.

- fix: databendlabs#4890
drmingdrmer added a commit to drmingdrmer/databend that referenced this issue Apr 18, 2022
@drmingdrmer
[metasrv] feature: metasrv has to be compatible with 20220413-34e89c99
62b501a 
Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

To solve th compatible issue, an intermedia type is introduced, which is
a super set of the old and new type.
When loading a record, first load it into the superset type, then reduce
to the latest type.

- fix: databendlabs#4890
@BohuTANG
Copy link
Member

Hi @oslet

We have made databend-meta forward compatible from v0.7.22, please use https://github.com/datafuselabs/databend/releases/tag/v0.7.22-nightly

ygf11 pushed a commit to ygf11/databend that referenced this issue Apr 20, 2022
@drmingdrmer @ygf11
[metasrv] feature: metasrv has to be compatible with 20220413-34e89c99
a538935 
Let metasvr be able to load the log-entry format before commit
34e89c9,
in which the `Cmd` format changed.

To solve th compatible issue, an intermedia type is introduced, which is
a super set of the old and new type.
When loading a record, first load it into the superset type, then reduce
to the latest type.

- fix: databendlabs#4890
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@drmingdrmer drmingdrmer

Labels
C-bug Category: something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@drmingdrmer @BohuTANG @oslet