ES upgrade to 2.4.1 #808

ChandraAddala · 2017-05-11T20:50:26Z

Upgrading ES to 2.x
 - Upgrading ES to 2.4.4 and lucene to 5.5.2
 - Changes to mapping document: Change from index_analyzer to analyzer
 - Changing max limit from 100,000 to 10,000; Externalized this limit.
 - Adding ESIntegTestCase for integration tests
 - Changes to ingest ES client to ElasticIO and ElasticTokensIO
 - Remove references to shaded libraries of old ES
 - Removed old test library, elasticsearch-test
 - Common logic for ES setup is placed in HttpESIntegrationBase

- Upgrading ES to 2.4.4 and lucene to 5.5.2 - Changes to mapping document: Change from index_analyzer to analyzer - Changing max limit from 100,000 to 10,000; Externalized this limit. - Adding ESIntegTestCase for integration tests - Changes to ingest ES client to ElasticIO and ElasticTokensIO - Remove references to shaded libraries of old ES

- Removed old test library, elasticsearch-test - Common logic for ES setup is placed in HttpESIntegrationBase

coveralls · 2017-05-11T21:02:41Z

Coverage decreased (-20.5%) to 54.613% when pulling 1b2649f on ChandraAddala:es-upgrade into be582d1 on rackerlabs:master.

shintasmith

some questions

shintasmith · 2017-05-12T13:23:06Z

blueflood-core/pom.xml

@@ -122,6 +122,7 @@
              <!-- do not require zookeeper for these tests. -->
              <argLine>-DZOOKEEPER_CLUSTER=NONE</argLine>
              <argLine>-Dlog4j.configuration=file://${basedir}/src/test/resources/log4j-test.properties ${jacoco.agent.it.argLine} -Xmx1024m -XX:MaxPermSize=256m</argLine>
+              <argLine>-Duser.language=en</argLine>


What's wrong with French or Korean??? 😂

shintasmith · 2017-05-12T13:24:21Z

...lood-core/src/integration-test/java/com/rackspacecloud/blueflood/io/IntegrationTestBase.java

@@ -37,7 +37,7 @@

    private static final char[] STRING_SEEDS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890_".toCharArray();
    protected static final Random RAND = new Random(System.currentTimeMillis());
-    protected static final ConcurrentHashMap<Locator, String> locatorToUnitMap = new ConcurrentHashMap<Locator, String>();
+    public static final ConcurrentHashMap<Locator, String> locatorToUnitMap = new ConcurrentHashMap<Locator, String>();


Just curious, why are these methods and the above variable all made public?

This class is in blueflood-core. It was earlier extended in blueflood-http test classes. Now instead of extending they contain this class (has-a relationship). I have to make them public to be able to call them.

shintasmith · 2017-05-12T13:29:24Z

blueflood-elasticsearch/pom.xml

@@ -33,8 +33,101 @@


  <build>
+    <testResources>
+      <testResource>
+        <directory>src/integration-test/java</directory>


Why is this directory specified as a testResource? If there are some files in that directory needed to be copied to classpath, then those files should be moved to src/integration-test/resources directory.

shintasmith · 2017-05-12T13:44:22Z

blueflood-elasticsearch/pom.xml

+            <id>add-integration-test-sources</id>
+            <phase>generate-test-sources</phase>
+            <goals>
+              <goal>add-test-source</goal>


Do we need to do this? I know we did this in blueflood-core but I'm wondering if it's just left over cruft there. Adding test sources should be done within the failsafe-plugin or surefire-plugin. There is a

<testSourceDirectory>src/integration-test/java</testSourceDirectory>

configuration element. Does that not work?

This is not available in the version we are using 2.19. It is available from 2.2 onwards.

shintasmith · 2017-05-12T14:27:27Z

...arch/src/integration-test/java/com/rackspacecloud/blueflood/io/ElasticIOIntegrationTest.java

 public class ElasticIOIntegrationTest extends BaseElasticTest {

    protected ElasticIO elasticIO;
-    protected ElasticTokensIO elasticTokensIO;


So is this test changing from testing the Token indexes to just general ElasticSearch test?

testing token indexes is in its own class ElasticTokensIOIntegrationTest

shintasmith · 2017-05-12T14:41:35Z

...rc/integration-test/java/com/rackspacecloud/blueflood/io/ElasticTokensIOIntegrationTest.java

+ * -Dtests.security.manager=false (https://github.com/elastic/elasticsearch/issues/16459)
+ *
+ */
+public class ElasticTokensIOIntegrationTest extends BaseElasticTest {


👍 for separating this

shintasmith · 2017-05-12T14:47:06Z

...ttp/src/integration-test/java/com/rackspacecloud/blueflood/http/HttpIntegrationTestBase.java

+        Configuration.getInstance().setProperty(CoreConfig.DISCOVERY_MODULES.name(), "com.rackspacecloud.blueflood.io.ElasticIO");
+        Configuration.getInstance().setProperty(CoreConfig.EVENTS_MODULES.name(), "com.rackspacecloud.blueflood.io.EventElasticSearchIO");
+        Configuration.getInstance().setProperty(CoreConfig.TOKEN_DISCOVERY_MODULES.name(), "com.rackspacecloud.blueflood.io.ElasticTokensIO");
+        Configuration.getInstance().setProperty(CoreConfig.ENABLE_TOKEN_SEARCH_IMPROVEMENTS.name(), "true");


Do we not load a blueflood config for these tests? Should we be putting these in the config file?

Thats correct. We dont load a separate config file. I just used the way it was being done before. It mostly takes defaults for other properties except a few of them.

shintasmith · 2017-05-12T14:55:40Z

@ChandraAddala do these changes work with ElasticSearch 1.x ?

Once we merge this PR, we will essentially move our tests to ElasticSearch 2.x, while our Prod is still 1.x. I'm wondering what should be the order of things?

shintasmith · 2017-05-12T14:57:07Z

Also, coverage seems to decrease 20% with this PR. Can you look to see if that's just a bug in coverall? Or was there something big we missed testing?

coveralls · 2017-05-23T04:25:39Z

Coverage decreased (-20.4%) to 54.675% when pulling da6bbf4 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-24T16:33:18Z

Coverage decreased (-20.3%) to 54.736% when pulling d8c88a7 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-25T17:04:43Z

Coverage decreased (-20.4%) to 54.655% when pulling 9e8b15c on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-25T17:26:05Z

Coverage decreased (-20.4%) to 54.675% when pulling b83a659 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-25T19:11:47Z

Coverage decreased (-20.4%) to 54.655% when pulling b7e5f5b on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-25T20:35:30Z

Coverage decreased (-20.5%) to 54.614% when pulling 9d7c34e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-25T21:02:01Z

Coverage decreased (-20.5%) to 54.614% when pulling 6fea98e on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

coveralls · 2017-05-26T01:52:35Z

Coverage decreased (-20.5%) to 54.604% when pulling 8ca95d2 on ChandraAddala:es-upgrade into 358242e on rackerlabs:master.

Chandra Addala added 4 commits April 13, 2017 21:58

Separating token search tests

fe482c5

Merge remote-tracking branch 'upstream/master' into es-upgrade

c554b1d

Upgrading ES to 2.x

1b2649f

- Removed old test library, elasticsearch-test - Common logic for ES setup is placed in HttpESIntegrationBase

ChandraAddala closed this May 11, 2017

ChandraAddala reopened this May 11, 2017

shintasmith reviewed May 12, 2017

View reviewed changes

ChandraAddala closed this May 17, 2017

ChandraAddala reopened this May 17, 2017

ChandraAddala force-pushed the es-upgrade branch 2 times, most recently from a01fe21 to 1b2649f Compare May 18, 2017 16:16

corrected test resources; adding log4j test directory

567b20b

ChandraAddala closed this May 19, 2017

ChandraAddala reopened this May 19, 2017

Chandra Addala added 4 commits May 22, 2017 10:26

Removed ShardStateIntegrationTest

3cd3c56

Merge remote-tracking branch 'upstream/master' into es-upgrade

62ed1bc

Fixing a integration test

bed30ed

Fixing some timing issue with a test

da6bbf4

Chandra Addala added 3 commits May 23, 2017 13:35

Adding back ShardStateIntegrationTest

067af04

Adding some debugging

4451cad

More debugging info

d8c88a7

Some more debug info

4906a80

ChandraAddala mentioned this pull request May 25, 2017

Elasticsearch version #811

Open

Change one of test to run for 5 secs instead of 15 secs

9e8b15c

Removed debug statements

b83a659

Moving ES to 2.4.1 instead of 2.4.4

b7e5f5b

ChandraAddala changed the title ~~ES upgrade to 2.4.4~~ ES upgrade to 2.4.1 May 25, 2017

ChandraAddala closed this May 25, 2017

ChandraAddala reopened this May 25, 2017

upgrading to jackson 2.6.2 as ES requires it

9d7c34e

upgraded jackson to 2.8.1

6fea98e

Adding jackson-dataformat-smile

8ca95d2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES upgrade to 2.4.1 #808

ES upgrade to 2.4.1 #808

ChandraAddala commented May 11, 2017

coveralls commented May 11, 2017

shintasmith left a comment

shintasmith May 12, 2017

shintasmith May 12, 2017

ChandraAddala May 17, 2017

shintasmith May 12, 2017

shintasmith May 12, 2017

ChandraAddala May 17, 2017

shintasmith May 12, 2017

ChandraAddala May 17, 2017

shintasmith May 12, 2017

shintasmith May 12, 2017

ChandraAddala May 17, 2017

shintasmith commented May 12, 2017

shintasmith commented May 12, 2017

coveralls commented May 23, 2017

coveralls commented May 24, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 26, 2017

ES upgrade to 2.4.1 #808

Are you sure you want to change the base?

ES upgrade to 2.4.1 #808

Conversation

ChandraAddala commented May 11, 2017

coveralls commented May 11, 2017

shintasmith left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shintasmith commented May 12, 2017

shintasmith commented May 12, 2017

coveralls commented May 23, 2017

coveralls commented May 24, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 25, 2017

coveralls commented May 26, 2017