Implement light subscriptions stats and refactor #861

lmontrieux · 2018-04-15T07:51:03Z

Implement light subscriptions stats and refactor

Zalando ticket : ARUHA-1551

Description

light subscription stats in /subscriptions endpoint
refactor Zk subscription client to not acquire locks to read
subscription data
remove ServiceUnavailableException from the project, preferring the
unchecked ServiceTemporarilyUnavailableException instead
update the swagger definition

Review

Tests
Documentation
CHANGELOG

- light subscription stats in /subscriptions endpoint - refactor Zk subscription client to not acquire locks to read subscription data - remove ServiceUnavailableException from the project, preferring the unchecked ServiceTemporarilyUnavailableException instead - update the swagger definition

codecov-io · 2018-04-15T07:55:36Z

Codecov Report

Merging #861 into master will decrease coverage by 0.03%.
The diff coverage is 62.9%.

@@             Coverage Diff              @@
##             master     #861      +/-   ##
============================================
- Coverage     53.59%   53.56%   -0.04%     
- Complexity     1680     1691      +11     
============================================
  Files           310      309       -1     
  Lines          9327     9359      +32     
  Branches        837      841       +4     
============================================
+ Hits           4999     5013      +14     
- Misses         4029     4046      +17     
- Partials        299      300       +1

Impacted Files	Coverage Δ	Complexity Δ
...alando/nakadi/service/CursorOperationsService.java	`59.8% <ø> (ø)`	`21 <0> (ø)`	⬇️
...org/zalando/nakadi/repository/TopicRepository.java	`0% <ø> (ø)`	`0 <0> (ø)`	⬇️
...nakadi/service/converter/VersionZeroConverter.java	`29.31% <ø> (ø)`	`5 <0> (ø)`	⬇️
.../nakadi/service/converter/CursorConverterImpl.java	`51.42% <ø> (ø)`	`7 <0> (ø)`	⬇️
...ava/org/zalando/nakadi/service/CursorsService.java	`10.3% <ø> (ø)`	`2 <0> (ø)`	⬇️
.../service/subscription/zk/ZkSubscriptionClient.java	`31.7% <ø> (ø)`	`0 <0> (ø)`	⬇️
...va/org/zalando/nakadi/service/CursorConverter.java	`87.5% <ø> (ø)`	`0 <0> (ø)`	⬇️
...alando/nakadi/controller/PartitionsController.java	`79.22% <ø> (-7.8%)`	`12 <0> (ø)`
...g/zalando/nakadi/controller/ExceptionHandling.java	`52.5% <ø> (ø)`	`12 <0> (ø)`	⬇️
...a/org/zalando/nakadi/service/BlacklistService.java	`13.04% <0%> (ø)`	`3 <0> (ø)`	⬇️
... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25e9497...e06f28d. Read the comment docs.

adyach · 2018-04-16T07:52:30Z

docs/_data/nakadi-event-bus-api.yaml

@@ -964,6 +964,11 @@ paths:
          required: false
          default: 0
          minimum: 0
+        - name: show_status


do you think it is better to have separate endpoint for that ?

@adyach I followed the requirements in our internal ticket:
GET /subscriptions endpoint should have additional parameter show_status = true/false (false by default).

To be honest I don't really have a preference one way or another. Do you think it would be better as a separate endpoint?

v-stepanov · 2018-04-16T09:40:28Z

docs/_data/nakadi-event-bus-api.yaml

+            type: object
+            description: statistics of partition within a subscription context
+            properties:
+              partition:


there's no description for partition field

v-stepanov · 2018-04-16T09:53:06Z

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

            throws InconsistentStateException, ServiceTemporarilyUnavailableException {
+        final List<EventType> eventTypes = getEventTypesForSubscription(subscription);
+        final List<PartitionEndStatistics> topicPartitions = loadPartitionEndStatistics(eventTypes);


As I see we do this call not looking at includeDistance value. But this method will actually go to Kafka and do this expensive seek operation. I thought that in case of light-stats we will not check the offsets in Kafka.

That is correct, I thought of doing it then forgot to do so. Looks like there will be some more refactoring.

@v-stepanov 9436dff

v-stepanov · 2018-04-16T10:49:19Z

src/main/java/org/zalando/nakadi/service/subscription/zk/AbstractZkSubscriptionClient.java

            throws SubscriptionNotInitializedException, NakadiRuntimeException {
        if (!isSubscriptionCreatedAndInitialized()) {
            return Optional.empty();
        }

-        return Optional.of(runLocked(() -> new ZkSubscriptionNode(


Why have you removed running under lock?

Because it isn't necessary anymore, since we migrated to new zk subscription format (I double checked with @antban )

v-stepanov · 2018-04-17T14:40:57Z

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

+        if (includeDistance) {
+            return loadStats(eventTypes, zkSubscriptionNode, subscriptionClient);
+        }
+        else {


I used to put else on a new line like that but the team told me not to do that :(

@v-stepanov faaee92. What the team wants, the team gets :)

v-stepanov · 2018-04-18T18:32:51Z

@lmontrieux code looks good for me now.
Are there any tests for the new functionality?

v-stepanov · 2018-04-18T19:21:27Z

deploy validation please

v-stepanov · 2018-04-19T04:39:50Z

src/main/java/org/zalando/nakadi/domain/Subscription.java

@@ -18,6 +20,8 @@ public Subscription(final String id, final DateTime createdAt, final Subscriptio

    private DateTime createdAt;

+    private List<SubscriptionEventTypeStats> stats;


maybe you can add @JsonInclude(Include.NON_NULL) here? Because in other case it will show up for all endpoints where we return subscription.

v-stepanov · 2018-04-19T04:41:49Z

src/main/java/org/zalando/nakadi/domain/Subscription.java

@@ -34,6 +38,14 @@ public void setCreatedAt(final DateTime createdAt) {
        this.createdAt = createdAt;
    }

+    public List<SubscriptionEventTypeStats> getStats() {


probably it should be marked as nullable as it is null in most of the cases

v-stepanov · 2018-04-19T04:43:37Z

src/main/java/org/zalando/nakadi/service/subscription/SubscriptionService.java

@@ -157,10 +157,16 @@ public Result listSubscriptions(@Nullable final String owningApplication, @Nulla
                    subscriptionRepository.listSubscriptions(eventTypesFilter, owningAppOption, offset, limit);
            final PaginationLinks paginationLinks = SubscriptionsUriHelper.createSubscriptionPaginationLinks(
                    owningAppOption, eventTypesFilter, offset, limit, subscriptions.size());


new field show_status should also be included to pagination link.

lmontrieux · 2018-04-19T10:21:15Z

@v-stepanov there are 2 acceptance tests in 2ffeb20

lmontrieux · 2018-04-19T11:05:14Z

👍

v-stepanov · 2018-04-19T14:11:44Z

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+    public void whenLightStatsOnNotInitializedSubscriptionThenCorrectResponse() throws IOException {
+        final String et = createEventType().getName();
+        final Subscription s = createSubscriptionForEventType(et);
+        final Response response = when().get("/subscriptions?show_status=true").thenReturn();


it's an acceptance test, what if there will be other subscriptions created by other tests? I think it's better to differ the subscription you created (e.g. by owning_application)

v-stepanov · 2018-04-19T14:12:59Z

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+            if (subscription.getId().equals(s.getId())) {
+                Assert.assertNotNull(subscription.getStats());
+                Assert.assertEquals("assigned", subscription.getStats().get(0).getPartitions().get(0).getState());
+                Assert.assertNotEquals("", subscription.getStats().get(0).getPartitions().get(0).getStreamId());


you can actually take streamId from TestStreamingClient and use it here for assertion.

v-stepanov · 2018-04-19T14:14:10Z

src/acceptance-test/java/org/zalando/nakadi/webservice/hila/SubscriptionAT.java

+                .start();
+        waitFor(() -> assertThat(client.getBatches(), hasSize(15)));
+
+        final Response response = when().get("/subscriptions?show_status=true").thenReturn();


here it's also possible to have the same problem as I described several lines ago in another test (subscriptions left by other tests).

v-stepanov · 2018-04-19T14:22:29Z

👍

lmontrieux · 2018-04-19T14:23:50Z

👍

lmontrieux · 2018-04-19T14:29:24Z

Deploy validation please

This is to avoid confusion, since the flag is called show_status, and the result is the status of the subscription, not some stats

lmontrieux · 2018-04-20T10:11:45Z

👍

lmontrieux · 2018-04-20T14:08:35Z

👍

v-stepanov · 2018-04-20T14:41:55Z

👍

v-stepanov · 2018-04-23T08:53:08Z

deploy validation please

…stats # Conflicts: # CHANGELOG.md

v-stepanov · 2018-04-25T14:11:06Z

👍

antban · 2018-04-25T14:31:06Z

👍

Lionel Montrieux added 2 commits April 15, 2018 09:48

Checkstyle

3134284

adyach reviewed Apr 16, 2018

View reviewed changes

Only get light stats for subscriptions in the returned page

da1c25c

v-stepanov reviewed Apr 16, 2018

View reviewed changes

Lionel Montrieux added 2 commits April 16, 2018 17:09

Do not get last position from topic repository when getting light stats

9436dff

description for partition fields

f650f43

v-stepanov reviewed Apr 17, 2018

View reviewed changes

formatting

faaee92

v-stepanov reviewed Apr 19, 2018

View reviewed changes

Lionel Montrieux added 2 commits April 19, 2018 11:19

post review changes

c2ddcd0

acceptance tests

2ffeb20

Lionel Montrieux added 2 commits April 19, 2018 13:24

Merge branch 'master' into light-subscription-stats

7e6de87

Changelog

cf0e8de

v-stepanov reviewed Apr 19, 2018

View reviewed changes

Improve acceptance tests

42210eb

lmontrieux changed the title ~~[WIP] Implement light subscriptions stats and refactor~~ Implement light subscriptions stats and refactor Apr 19, 2018

Lionel Montrieux added 2 commits April 20, 2018 11:28

Change wording in documentation from light stats to status

a735d48

This is to avoid confusion, since the flag is called show_status, and the result is the status of the subscription, not some stats

Update changelog for release

5bd55fd

change stats to status

74fb913

Merge remote-tracking branch 'origin/master' into light-subscription-…

e06f28d

…stats # Conflicts: # CHANGELOG.md

v-stepanov merged commit f56de8a into master Apr 25, 2018

		@@ -18,6 +20,8 @@ public Subscription(final String id, final DateTime createdAt, final Subscriptio

		private DateTime createdAt;

		private List<SubscriptionEventTypeStats> stats;

Implement light subscriptions stats and refactor #861

Implement light subscriptions stats and refactor #861

Conversation

lmontrieux commented Apr 15, 2018

Implement light subscriptions stats and refactor

Description

Review

codecov-io commented Apr 15, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

lmontrieux Apr 16, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

v-stepanov commented Apr 18, 2018

v-stepanov commented Apr 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmontrieux commented Apr 19, 2018

lmontrieux commented Apr 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

v-stepanov Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

v-stepanov commented Apr 19, 2018

lmontrieux commented Apr 19, 2018

lmontrieux commented Apr 19, 2018

lmontrieux commented Apr 20, 2018

lmontrieux commented Apr 20, 2018

v-stepanov commented Apr 20, 2018

v-stepanov commented Apr 23, 2018

v-stepanov commented Apr 25, 2018

antban commented Apr 25, 2018

codecov-io commented Apr 15, 2018 •

edited

Loading

lmontrieux Apr 16, 2018 •

edited

Loading

v-stepanov Apr 19, 2018 •

edited

Loading