Skip to content

Latest commit

 

History

History
83 lines (65 loc) · 3.95 KB

pip-360.md

File metadata and controls

83 lines (65 loc) · 3.95 KB
Raw

PIP-360: Admin API to display Schema metadata

Background knowledge

Broker loads and initializes Schema of the topic during the topic loading. However, we have seen large number of instances and issues when broker fails to load the topic when topic schema is broken due to missing or corrupt schema ledger, index ledger or even schema data. Therefore, if broker is not able to load the topic for any reason then it is not possible to fetch schema metadata and identify which schema ledger is causing the issue because broker is storing schema metadata into binary format and there is no such API exists which shows schema metadata into readable format. So, it is very important to have an API to read schema metadata with complete information to help system admin to understand topic unavailability issues. It is also very useful to get schema metadata to build various schema related external tools which can be used by system administrator. We already have APIs for managed-ledger and bookkeeper-ledgers which are used by external tools and CLI to read binary data from metadata store and display in readable format.

Motivation

Schema is one of the important part of the topic because it also plays important part in topic availability and required to successfully load the topic, and if schema initialization failure is causing issue in topic loading then it is very important to get schema metadata information to understand schema related issues and perform appropriate actions to mitigate that issue to successfully load the topic and make it available for users. Therefore, similar to ledger metadata and managed-ledger metadata, Pulsar should have API to show schema metadata and related ledger info which can be used by tools or users to perform appropriate actions during topic availability issues or any other troubleshooting.

Goals

Add an .admin API under schema resource which returns schema metadata into readable format

High Level Design

This PIP will introduce REST api which will accept the topic name and return schema metadata along with ledger information of schema-ledgers and index entries. It will also add CLI support to print schema metadata for users to see it in human readable format.

Public API

This PIP will add a new REST endpoint under Schema resource path.

Path: schema/{tenant}/{namespace}/{topic}/metadata
Response code:
307, message = Current broker doesn't serve the namespace of this topic
401, message = Client is not authorized or Don't have admin permission
403, message = Client is not authenticated
404, message = Tenant or Namespace or Topic doesn't exist; or Schema is not found for 
412, message = Failed to find the ownership for the topic

This admin API will return below schema metadata response.

@Data
public class SchemaMetadata {
    public Entry info;
    public List<Entry> index;
    @Data
    @AllArgsConstructor
    @NoArgsConstructor
    static class Entry {
        private long ledgerId;
        private long entryId;
        private long version;
        @Override
        public String toString() {
            return MoreObjects.toStringHelper(this)
                    .add("ledgerId", ledgerId)
                    .add("entryId", entryId)
                    .add("version", version)
                    .toString();
        }
    }
}

CLI

This PIP will also add appropriate CLI command under Schema command to get schema metadata.

bin/pulsar-admin schemas get-metadata <topic>

Links

Sample PR: #22938

  • Mailing List discussion thread:
  • Mailing List voting thread: