Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload schema from a secure schema registry #1299

Closed
2 tasks done
yruslan opened this issue Apr 23, 2020 · 1 comment · Fixed by #1405 or #1430
Closed
2 tasks done

Upload schema from a secure schema registry #1299

yruslan opened this issue Apr 23, 2020 · 1 comment · Fixed by #1405 or #1430
Assignees
@dk1844
Labels
feature New feature priority: high Critical to the health of the project request External feature request

Comments

@yruslan
Copy link
Contributor

yruslan commented Apr 23, 2020 •
edited by dk1844
Loading

Background

This is requested by our users as something that would be very helpful to them.

Feature

Upload schema from a secure schema registry.

@kevinwallimann gave a great idea how we can simplify all these things.

We should add support for specifying the subject of a schema registry.
The base URL of schema server should be configured in Menas in application.properties

User should provide:

  • A subject in the schema registry. For the naming strategy based on the topic name it will be topic_name-value.
  • [ ] An optional key subject might be specified (e.g. topic_name-key). If it is specified, the schema of the key should be merged with the value schema.
  • if the provided topic_name does not exist, there should be an automatic (behind the scenes) fallback to topic_name-value. If even that does not succeed, the schema loading fails.

Menas should:

  • Upload the schema from the schema registry.
@yruslan yruslan added feature New feature request External feature request priority: high Critical to the health of the project labels Apr 23, 2020
dk1844 added a commit that referenced this issue Jun 15, 2020
@dk1844
#1299 schema upload by topic-name - UI with -value, -key selection wi…
65dd4c7 
…th liveUpdate
dk1844 added a commit that referenced this issue Jun 17, 2020
@dk1844
#1299 schema upload routine outline, backend missing
1e71722
dk1844 added a commit that referenced this issue Jun 17, 2020
@dk1844
#1299 schema loading from schema registry by topicNameStem,
e351748 
 secure SR works by supplying keyStore & trustStore config values.
 Optionally, value+key can get combined, nontrivial combination for struct-based schemas added -> TODO add tests!
 menas template update with SR stuff

 SecureKafka.scala moved to utils and renamed to SecureConfig.

 Smart Pathjoining added via /.
dk1844 added a commit that referenced this issue Jun 17, 2020
@dk1844
#1299 schema combining unit test added.
66a23c5
dk1844 added a commit that referenced this issue Jun 17, 2020
@dk1844
#1299 schema combining unit test added.
00a3b4b 
typos/comments updated
dk1844 added a commit that referenced this issue Jun 17, 2020
@dk1844
#1299 schema combining: duplicates disallowed withing single fieldLis…
460e7da 
…t + unitTest
dk1844 added a commit that referenced this issue Jun 18, 2020
@dk1844
#1299 UI update: vbox with collapsed/ible panels and formatted text w…
06d58c4 
…ith descriptions
dk1844 added a commit that referenced this issue Jun 18, 2020
@dk1844
#1299 UI update: remoteUrl loading/by topicNameStem loading handling …
2d84986 
…accounted for,

latest version of schema allowed to get fixed, e.g. `https://mydomain1.com/subjects/myTopic1/versions/latest` -> `https://mydomain1.com/subjects/myTopic1/versions/latest/schema`
dk1844 added a commit that referenced this issue Jun 18, 2020
@dk1844
#1299 Schema registry "byTopicStemName" load from SR - integration te…
4fb045c 
…st added.

UI reset-after-schema-update fix (topicMergeWithKey)
dk1844 added a commit that referenced this issue Jun 19, 2020
@dk1844
#1299 combining schemas - integTest
420260e
@dk1844 dk1844 linked a pull request Jun 19, 2020 that will close this issue
Feature/1299 schema from secure schema registry #1405
Merged
@dk1844
Copy link
Contributor

dk1844 commented Jun 19, 2020 •
edited
Loading

Release notes
Schema can be updated by specifying subjectName (if the subjectName is not found, subjectName-value will be tried). A secure schema registry is supported, too.

dk1844 added a commit that referenced this issue Jun 22, 2020
@dk1844
#1299 PR updates: api/schema/topic -> api/schema/registry, scaladoc
8df37f6
dk1844 added a commit that referenced this issue Jun 22, 2020
@dk1844
#1299 Schema Availability API = mainly shows if SR support is enabled.
7799a17 
ConfigReader/ConfigImplicits consolidated a bit + unit tests added.
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 Schema Availability API => only registryAvailable is part of th…
1a72028 
…e api (upload, remoteUrl are always available for the user)
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 Requirement changed: -value + -key combination removed, now onl…
228b086 
…y subject is loaded + fallback subject-value. (unit-tests removed)

 - topicName -> subject rewording
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 schema update availability tests added
fc94d8e
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 cleanup, UI update
a111a9c
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 cleanup, UI update
1b81819
dk1844 added a commit that referenced this issue Jun 24, 2020
@dk1844
#1299 cleanup, UI update, test update, template wording, update
b535549
@dk1844 dk1844 mentioned this issue Jun 24, 2020
Merged
dk1844 added a commit that referenced this issue Jun 26, 2020
@dk1844
#1299 mergefix (Config reader import changes mostly)
b55aa10
dk1844 added a commit that referenced this issue Jun 29, 2020
@dk1844
#1299 SchemaApi -Availability -> -Features, /features REST endpoint
fff00e6
dk1844 added a commit that referenced this issue Jun 30, 2020
@dk1844
#1299 menas configuration: secure schema registry
722088e
dk1844 added a commit that referenced this issue Jul 7, 2020
@dk1844 @Zejnilovic
Feature/1299 schema from secure schema registry (#1405)
3e3db4a 
* #1299 schema loading from schema registry by subjectName (or fallbacks to subjectName-value)
-  secure SR works by supplying keyStore & trustStore config values.
- SecureKafka.scala moved to utils and renamed to SecureConfig.
- Smart Pathjoining added via /.
-  UI update: vbox with collapsed/ible panels and formatted text with descriptions
- SchemaApiFeature endpoint drives the SR integration enabling on UI
- integration test added, unit tests added

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
dk1844 added a commit that referenced this issue Jul 7, 2020
@dk1844 @Zejnilovic
#1299 menas configuration: secure schema registry (#1419)
a2ed7b7 
* #1299 menas configuration: secure schema registry

Co-authored-by: Saša Zejnilović <zejnils@gmail.com>
dk1844 added a commit that referenced this issue Jul 7, 2020
@dk1844
#1299 application.properties.template cleanup
f80e916
@dk1844 dk1844 linked a pull request Jul 7, 2020 that will close this issue
#1299 application.properties.template cleanup #1430
Merged
dk1844 added a commit that referenced this issue Jul 7, 2020
@dk1844
#1299 application.properties.template cleanup
85a8c20
dk1844 added a commit that referenced this issue Jul 8, 2020
@dk1844
#1299 application.properties.template cleanup (#1430)
ad78dbd 
* #1299 application.properties.template cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dk1844 dk1844

Labels
feature New feature priority: high Critical to the health of the project request External feature request
Projects
None yet
Milestone
No milestone
2 participants
@yruslan @dk1844