Elasticsearch K mer Database Setup

K-Mer Database Setup

In order to setup the k-mer database, we first must setup Elasticsearch.

Acquire Elasticsearch

At the time of writing, Elasticsearch v5.1.1 is the most current version and thus will be what we are using. You can view the most current version at Download Elasticsearch

Download and Install Elasticsearch

We are using Ubuntu Server as our operating system. This allows us to install Elasticsearch via a Debian package. More information is available at Installing Elasticsearch

Commands

(Optional) Remove Previous Version

sudo apt-get --purge autoremove elasticsearch
sudo rm -rf /var/lib/elasticsearch/
sudo rm -rf /etc/elasticsearch
sudo rm -rf /home/elasticsearch/*

Java JDK 8 Requirement

Followed Digital Ocean's How To Install Java Guide

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Download and Install On Ubuntu

sudo su

# Create directory to install and use as data source
mkdir -p /home/elasticsearch
cd /home/elasticsearch
mkdir configs
mkdir logs

# Download ES v5.1.1
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.deb
dpkg -i elasticsearch-5.1.1.deb

# Create a configs folder and download templates
cd configs 
# /etc/elasticsearch/elasticsearch.yml
wget https://gist.githubusercontent.com/rpetit3/f0e0c565bd30c6c079b26811dcbdffa1/raw/b7929b66b0fc2b6d18a929aa067e93ad406896cc/elasticsearch.yml
mv /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
ln -s /home/elasticsearch/configs/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml

# /etc/default/elasticsearch
wget https://gist.githubusercontent.com/rpetit3/e1b32ba74345c29d6b5998010fa6d231/raw/860803ba554d96601e5acdce978fa65ef76df439/elasticsearch.default
mv /etc/default/elasticsearch /etc/default/elasticsearch.bak
ln -s /home/elasticsearch/configs/elasticsearch.default /etc/default/elasticsearch

# Give ownership to elasticsearch user
cd ..
chown -R elasticsearch /home/elasticsearch
chgrp -R elasticsearch /home/elasticsearch

exit

Edit /home/elasticsearch/configs/elasticsearch.yml

Change the node.name variable to the server name.

# Change none.name
node.name: NODE_NAME

Edit /etc/elasticsearch/jvm.options

See the following ES JVM Options

# CHANGE 
-Xms2g
-Xmx2g

# TO THIS
-Xms30g
-Xmx30g

Edit /etc/security/limits.conf

# Add the following to /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited

Ubuntu 16.04 Changes

Make the following changes Systemd configutation

sudo emacs /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

# WHILE IN EMACS (OR TEXT EDITOR)
Add the following to /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

[Service]
LimitMEMLOCK=infinity

### EXIT TEXT EDITOR

sudo systemctl daemon-reload

Start Elasticsearch and Test

sudo service elasticsearch start
curl -XGET "http://localhost:9200/_cat/nodes?pretty"