Doradus Configuration Files

[Table of Contents](https://github.com/dell-oss/Doradus/wiki/Doradus Administration: Table-of-Contents) | [Previous](https://github.com/dell-oss/Doradus/wiki/Runtime Memory) | [Next](https://github.com/dell-oss/Doradus/wiki/Managing Doradus via JMX)
[Doradus Configuration and Operation](https://github.com/dell-oss/Doradus/wiki/Doradus Configuration and Operation): Doradus Configuration Files

---

The following are the primary Doradus configuration files within the {doradus_home}/config folder:

• doradus.yaml: This file provides global configuration parameters. Typically only a few parameters merit modification from their defaults.

• log4j.properties: This file controls logging features of the Doradus instance.

Setting Doradus Logging Options

Doradus logs messages about its operation using the log4j logging facility. Logging options are defined in the file log4j-server.properties. The high-level logging options are shown below:

log4j.rootLogger=DEBUG, console, file, memory
log4j.logger.org.eclipse.jetty=INFO
log4j.appender.console.Threshold=INFO
log4j.appender.file.Threshold=INFO
log4j.appender.memory.Threshold=DEBUG`

This sets the global logging level to DEBUG and defines multiple “appenders”: console (stdout), a disk file, and memory. Although DEBUG-level log records are generated, only INFO-level entries (and above) are sent to the console and file appenders. DEBUG entries are only collected by the memory appender. This minimizes the output to the console- and file-based logs, but it allows the most recent DEBUG records to be retrieved via the REST command: GET /_logs.

If Doradus is run as a service, the output to console is unnecessary, so this parameter should be removed. Additionally, the lines that begin with log4j.appender.console should be deleted or commented-out.

Because the memory appender only retains the most recent log entries, in diagnostic situations, you might want to capture DEBUG entries in the file log as well. Note that this causes log files to grow more quickly. DEBUG entries can be captured in the file appender by changing the following line:

log4j.appender.console.Threshold=DEBUG

The log4j-server.properties file also defines the following option:

log4j.appender.file.File=doradus.log

This option sends the file log appender to a log file called doradus.log in the Doradus server’s runtime directory. If you wish log files to be stored with another name and/or in another location, modify this option with a new relative or absolute file name.

Two other options to consider:

log4j.appender.file.maxBackupIndex=50
log4j.appender.file.maxFileSize=20MB

These options cause the log file to grow up to 20MB, at which point it is renamed with a numeric extension (.1, .2, etc.) and a new file is started. Up to 50 files are retained (up to 1GB total log data), at which point the oldest file is removed. Modify these options to increase or decrease the number and size of log files stored.

Doradus Security Options

Doradus does not store any application data; all data is stored by the Cassandra database service. Securing Doradus from unauthorized access requires securing its configuration files at the network protocols.

Securing the Doradus REST API

By default, the Doradus REST API uses unencrypted HTTP. Because Doradus provides no application-level security, any process that connect to the Doradus REST port is allowed to perform all schema, update, and query commands. The REST API can be secured by enabling TLS (SSL), which encrypts all traffic and uses mutual authentication to restrict access to specific clients. Optionally, client authentication can be enabled to restrict connections to only those whose certificates have been registered at the server. The process for securing the REST port with TLS is defined below:

1. Enable TLS by setting the tls parameter in the doradus.yaml file to true. Example:

   tls: true

2. Create a certificate for use by the Doradus server and store it in a keystore file. You can use the keytool utility included with the JRE. An overview of the process to create a self-signed certificate is outline here:

   http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#CreateKeystore

3. Set the keystore parameter in the doradus.yaml file to the location of the keystore file, and set the keystorepassword parameter to the file’s password. Example:

    keystore: config/keystore
    keystorepassword: mykspassword
   

4. If client authentication will be used, create a certificate for each client application and import them into a truststore file. (See the same article referenced above.) Set the truststore parameter in the doradus.yaml file to the location of the keystore file, and set the truststorepassword parameter to the file’s password. Example:

    truststore: config/truststore
    truststorepassword: mytspassword
   

5. To require client authentication, set the clientauthentication parameter in the doradus.yaml file. This requires REST API connections to use mutual authentication. Example:

   clientauthentication: true

6. The cipher algorithms allowed by the REST API when TLS is enabled is controlled via the tls_cipher_suites parameter. The default list includes the algorithms recommended for FIPS compliance. The actual algorithms allowed by REST API is a subset of the listed algorithms and those actually available to the JVM in which Doradus is running. The cipher algorithm list can be tailored, for example, to only allow 256-bit symmetrical encryption. Example:

   tls_cipher_suites: - TLS_DHE_DSS_WITH_AES_256_CBC_SHA - TLS_DHE_RSA_WITH_AES_256_CBC_SHA - TLS_RSA_WITH_AES_256_CBC_SHA - TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA - TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA - TLS_ECDH_RSA_WITH_AES_256_CBC_SHA

Custom algorithms can also be used as long as installed with the JVM used to run Doradus.

With the steps above, Doradus will use TLS for its REST API port, optionally requiring mutual authentication. Clients must connect to the REST port using TLS. If client authentication is enabled, they must submit a certificate that was added to the truststore. Each client must also support one of the configured cipher algorithms.

Securing the Doradus JMX API

Doradus supports the Java Management Extensions (JMX) protocol for monitoring Doradus and performing certain administrative commands. JMX can be secured with authorization and/or encryption, however, security JMX security is disabled in the {doradus_home}/bin/doradus-server.bat file included with Doradus. The common JMX options are defined in variables at the top of the file as shown below:

set JMX_PORT=9999
set JMX_SSL=false
set JMX_AUTHENTICATE=false

These options are summarized below:

• JMX_PORT: This variable sets the com.sun.management.jmxremote.port define in the JVM, which is the port number that the remote JMX clients must use. As shown, port 9999 is the default used for Doradus.

• JMX_SSL: This variable sets the com.sun.management.jmxremote.ssl define in the JVM. When set to true, it requires remote clients to use SSL (TLS) to connect to the JMX port. When SSL is enabled, additional options are available to require remote clients to have a client-side certificate.

• JMX_AUTHENTICATE: This variable sets the com.sun.management.jmxremote.authenticate define in the JVM. When set to true, it requires remote clients to provide a user ID and password in order to connect. Additional parameters are required to define allowed user IDs and passwords and the locations of the corresponding files.

Because JMX is a standard and external documentation is available for securing JMX, details for using SSL and/or authentication are not covered here. See for example the following:

http://docs.oracle.com/javase/6/docs/technotes/guides/management/agent.html

Though not shown above, here is another useful option: On multi-homed systems, java.rmi.server.hostname can be set to a specific IP address, causing JMX to bind to that address instead of the default one. For example:

set JAVA_OPTS=
...
 -Djava.rmi.server.hostname=10.1.82.121^
...

This causes the JMX port to bind to address 10.1.82.121.

Using a Secured CQL API

As described in the section [Securing the Cassandra API](https://github.com/dell-oss/Doradus/wiki/Cassandra Configuration Files#SecuringCassandraAPI), by default Cassandra’s CQL and Thrift APIs are not secured. Encrypted communication can be enabled as described in that section.

Configuring Doradus for Clusters

Multiple instances of Cassandra and/or Doradus can be used in the same cluster. As described in the section [Deployment Guidelines](https://github.com/dell-oss/Doradus/wiki/Deployment Guidelines), a recommended practice is to deploy an instance of both Doradus and Cassandra on each node. Then, each Doradus instance should be configured to contact all Cassandra nodes. This provides load balancing and fault tolerance in case a Cassandra node fails.

The Cassandra instance(s) to which each Doradus instance connects is defined in the dbhost parameter in its doradus.yaml file. This parameter should be set to a comma-separated list of host names or (preferably) IP addresses of Cassandra instance hosts. For example:

dbhost: 10.1.82.146, 10.1.82.147, 10.1.82.148

This example tells the corresponding Doradus instance to use three Cassandra instances. Doradus will distribute connections across these nodes for load balancing.

Configuration Parameters in doradus.yaml

This section describes the parameters that can be set in the doradus.yaml file. Except as discussed in the previous sections, most of these parameters can be left at their default value. However, there are some values that you should change for clustered configurations, to enable OLAP caching, etc.

Default Services Parameter

Default services define the internal Doradus services that will be started. These are in addition to storage services, which are defined separately. DBService and SchemaService are required and will always be started even if not listed here. Other services are optional and can be disabled when not needed. Custom services can also be added to this list. Note that the default services parameter is only used by Doradus in stand-alone mode. When Doradus is started as an embedded application, all services (include storage services) must be passed to the startEmbedded() method. Each line must contain the full package name of the service class. The optional services are:

com.dell.doradus.mbeans.MBeanService

Provides monitoring and management services via custom Doradus MBeans. Can be disabled if custom JMX commands are not needed.

com.dell.doradus.service.rest.RESTService

Provides the REST API via an embedded Jetty server, which listens to the server and port defined by restaddr and restport. It only makes sense to disable this when running Doradus as an embedded service.

com.dell.doradus.service.taskmanager.TaskManagerService

Provides background task management for data aging, statistics refreshing (Spider), and other scheduled tasks. Also manages on-demand tasks such as removing obsolete data (Spider). Can be disabled if background tasks are not needed for this Doradus instance.

default_services:
 -com.dell.doradus.mbeans.MBeanService
 -com.dell.doradus.service.db.DBService
 -com.dell.doradus.service.rest.RESTService
 -com.dell.doradus.service.schema.SchemaService
 -com.dell.doradus.service.taskmanager.TaskManagerService`

Doradus Service Parameters

The storage_services parameter defines which storage services are initialized when Doradus is started as a standalone application (e.g., as a service or console app). At least one storage service must be defined. The first storage service listed becomes the default for new applications created without explicitly declaring a storage service. Example:

storage_services:
- com.dell.doradus.service.spider.SpiderService
- com.dell.doradus.service.olap.OLAPService

This initializes both the Spider and OLAP services, making the Spider service the default for new applications.

Cassandra Keyspace Parameters

These parameters control the initial creation of the Cassandra Keyspace used by Doradus. A single Keyspace is created by Doradus and used to hold all ColumnFamilies used by Doradus applications. When the Doradus server starts, if the configured Keyspace does not exist, it is created using the parameters configured below.

keyspace: 'Doradus'

ks_defaults:
    - strategy_class: SimpleStrategy
    - strategy_options:
        - replication_factor: "1"   # enclose this value in quotes
    - durable_writes: true

The keyspace parameter is the keyspace name and defaults to 'Doradus'. The ks_defaults parameter defines parameters passed to Cassandra to create the Keyspace. All parameters accepted by Cassandra are allowed and are passed "as is". Note that parameters should be indented and begin with a dash (-). When used, sub-parameters such as the replication_factor passed to strategy_options, should be further indented. Numeric values should be enclosed in quotes.

When the Doradus server starts, if its configured Keyspace already exists, its parameters are not modified to match values in doradus.yaml. It is safe to use Cassandra tools such as cassandra-cli to modify Keyspace parameters when needed.

See Cassandra documentation for information about Keyspace parameters.

Cassandra ColumnFamily Parameters

The cf_defaults parameter is used when Doradus creates a new ColumnFamily. All values recognized by Cassandra are allowed and are passed "as is". Parameters must begin with a dash (-), and sub-parameters should be further indented. Empty values should be specified with a pair of quotes. Example:

cf_defaults:
    - compression_options:
        - sstable_compression: ""   # use empty string for "none"
    - gc_grace_seconds: 3600

After a ColumnFamily is created, Doradus does not modify its parameters to match values in doradus.yaml. It is safe to use Cassandra tools such as cassandra-cli to modify ColumnFamily parameters when needed.

See Cassandra documentation for information about ColumnFamily parameters.

Cassandra Connection Parameters

These parameters control connections to the Cassandra database using the Thrift or CQL APIs:

`dbhost: 'localhost'

This parameter configures the host name(s) or address(es) of the Cassandra node(s) that the Doradus server connect to. Static IP addresses are preferred over host names. If a comma-separated list of hosts is provided, Doradus will connect to each one in round-robin fashion, allowing load balancing and failover.

dbport: 9160

This is the Cassandra Thrift or CQL API port number. Change this value if Cassandra’s port is using something other than 9160. (Cassandra should use the same port on all nodes.)

jmxport: 7199

Change this value if Cassandra’s JMX port is using something other than 7199. (Cassandra should use the same port on all nodes.)

db_timeout_millis: 60000

This is the socket-level Cassandra database connection timeout in milliseconds. It is applicable to both the Thrift and CQL APIs. This time is used both when connecting to Cassandra and when performing read/write operations. If a connect, read, or write request is not received within this time, the connection is closed and a retry is initiated. Note, however, that Cassandra has its own RPC timeout value, which defaults to 10 seconds. This means that read and write operations will typically based on Cassandra’s value and not this one.

db_connect_retry_wait_millis: 30000

This is the Cassandra database (Thrift) initial connection retry wait in milliseconds. It is applicable to both the Thrift and CQL APIs. If an initial connection to Cassandra times-out (see db_timeout_millis above), Doradus waits this amount of time before trying again. Doradus keeps retrying indefinitely under the assumption that Cassandra may take a while to start.

max_commit_attempts: 10

This is the maximum number of times Doradus will attempt to perform a Cassandra update operation before aborting the operation. (See also retry_wait_millis).

max_read_attempts: 3

This is the maximum number of times Doradus will attempt to perform a read operation before aborting the operation. (See also retry_wait_millis).

max_reconnect_attempts: 3

This is the maximum number of times Doradus will attempt to reconnect to Cassandra after a failure. A reconnect is performed if a read or update call fails. (See also retry_wait_millis).

retry_wait_millis: 5000

This is the time in milliseconds that Doradus waits after a failed update, read, or reconnect request call to Cassandra. This time is multiplied by the attempt number, which means Doradus waits a little longer after each successive failed call. (See also max_commit_attempts, max_read_attempts, and max_reconnect_attempts.)

use_cql: false

By default, Doradus uses the Thrift API to communicate with Cassandra. When this option is set to true, it uses the CQL API instead. CQL must be enabled in each Cassandra instance that Doradus connects via the cassandra.yaml file option start_native_transport: true. Additionally, the doradus.yaml file option dbport must be set to the CQL API’s port (the default is 9042).

Note: A Doradus database created with the v2.1 or prior release must continue to use the Thrift API. New databases created with the v2.2 release can use either Thrift or CQL. Tests show that the Thrift API is a little faster for bulk data loading by about 3%. But over time, CQL may be faster due to the advantages of prepared statements. CQL is the recommended Cassandra API, and over time the Thrift API may be phased-out.

Doradus REST API Parameters

These parameters configure the Doradus REST API.

restaddr: 0.0.0.0

By default, the Doradus REST API binds to address 0.0.0.0, which means it accepts connections directed to any of the host’s IP addresses (including "localhost"). Change this value if you want the REST API to accept connections on a specific IP address instead.

restport: 1123

This is the REST port that Doradus listens to. Change this to accept REST API connections on a different port.

maxconns: 200

This is the maximum number of connections that the Doradus server will service at one time. If more than this number of connections is received, each additional connection is queued but not serviced until an existing connection is closed. This value also controls the maximum size of the REST server thread pool.

tls: false
tls_cipher_suites:
   - TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
   ...
clientauthentication: false
keystore: config/keystore
keystorepassword: changeit
truststore: config/truststore
truststorepassword: password

As described in the section **4.2.2 **Doradus Security Options, these parameters should be set to configure TLS/SSL for the Doradus REST API and to restrict connections to authenticated clients.

max_request_size: 52428800

This parameter determines the maximum size of an input requested that is accepted. Requests larger than this are rejected and the socket is closed. The default is 50MB.

Doradus Query Parameters

These parameters configure defaults used for Doradus object and aggregate query parsing and execution.

search_default_page_size: 100

This value is the default query page size used when an object query does not specify the &s (page size) parameter.

aggr_separate_search: false
aggr_concurrent_threads: 0
dbesoptions_entityBuffer: 1000
dbesoptions_linkBuffer: 1000
dbesoptions_initialLinkBuffer: 10
dbesoptions_initialLinkBufferDimension: 1000
dbesoptions_initialScalarBuffer: 30
l2r_enable: true

These parameters are set by the Doradus team and normally should not be changed.

OLAP Parameters

These parameters control the operation of the OLAP service:

olap_cache_size_mb: 100

This parameter controls the memory size of the most-recently-used field cache. If this value is exceeded, older fields will be un-loaded from memory.

olap_loaded_segments: 0

This parameter controls the number of most-recently-used segment statistics structures cached. Statistics structures are needed to load segment fields; they are small and do not occupy much memory. If this value is zero, the default value is 1 million, which is sufficient even if your application performs many merges.

olap_file_cache_size_mb: 100

When this value is non-zero, it enables OLAP "file" caching and sets the total size, in megabytes, of the cached data. OLAP uses virtual files to hold raw scalar values such as text values. Caching this data prevents round-trips to the database for certain types of queries. This value does not affect shard caching defined by other parameters.

olap_query_cache_size_mb: 100

When this value is non-zero, it enables recent query result caching and sets the total size, in megabytes, of the cached search results. Each cached result takes 1 bit per each object in the table.

olap_cf_defaults:
    - compression_options:
        - sstable_compression:  "" # use empty string for "none"
    - gc_grace_seconds: 3600

This parameter is used to create the OLAP ColumnFamily. All parameters recognized by Cassandra are accepted and passed "as is". Parameters must be indented and begin with a “-“; sub-parameters must be further indented. The OLAP ColumnFamily is created when the server is first started for a new database. If the OLAP ColumnFamily already exists, it is not modified to match these parameters. See Cassandra documentation for details about these values.

One parameter worth highlighting is the OLAP ColumnFamily’s gc_grace_seconds. This value controls how long deleted rows called "tombstones" are retained with data tables (called sstables) before they are truly deleted during a compaction cycle. The default is 864,000 seconds or 10 days, which provides lots of time for a failed node to recover and learn about deletions it may have missed. However, the OLAP service deletes many rows when it merges a shard, and these rows consume disk space. Moreover, they consume memory because active sstables are memory-mapped (mmap-ed) by Cassandra. This can cause excessive memory usage.

A much smaller gc_grace_seconds value is recommended for the OLAP CF, somewhere between 3600 (a hour) and 86400 (1 day). This causes tombstones to be deleted more quickly, freeing-up disk space and reducing memory usage.

Spider Parameters

These parameters control the operation of the Spider server:

batch_mutation_threshold: 10000

Defines the maximum number of mutations a batch update will queue before flushing to the database. After the mutations are generated for each object in a batch update, Spider checks to see if this threshold has been met and flushes the current mutation batch if it has. Larger values improve performance by reducing Cassandra round trips, but they require larger API buffer sizes. The default is 10,000.

Background Tasks Parameters

max_node_tasks: 2

Defines the maximum number of tasks that can be launched concurrently on one Doradus node. If there is no node that can launch a task since all the nodes are busy with max_node_tasks each, then the task is skipped, and may be scheduled next time.

# task_exec_delay: 5

Defines the minimum interval (in seconds) between the scheduled time and actual start of the task. This delay is used for Doradus nodes to synchronize and to decide which node would perform the task. After this initial delay, the task may be delayed once more for maximum task_exec_delay seconds, so actual time interval between scheduled time and actual task start time may be between task_exec_delay and 2 * task_exec_delay seconds.

task_restarts: 2

Number of attempts to restart a task if it was interrupted due to an internal error. The task will not be restarted if an external error (like server node fault) occurs.

check_interrupting_interval: 10

Defines the interval (in seconds) between consequent analysis of whether a task that is executing by this node was interrupted by user request.

default_schedule:

Global default schedule used for all tasks for all applications and tables. The value is a cron expression (example: "0 0 31 10 *"). The default is null, which means no tasks are automatically scheduled.