Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the hardware calculator, and add storage calculator #1318

Merged
merged 1 commit into from
Dec 18, 2024
Merged

Improve the hardware calculator, and add storage calculator #1318

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,13 @@

This site contains documentation for Gravwell, plus other resources such as [Downloads](quickstart/downloads) and [Release Notes](changelog/list).

If you're just starting out with Gravwell, we recommend reading the [Quick Start](quickstart/quickstart) first, then moving on to the [Search pipeline](search/search) documentation to learn more. If you're confused, the [Glossary](glossary/glossary) defines some of the technical terminology used throughout the documentation.
If you're just starting out with Gravwell, we recommend reading the [Quick Start](quickstart/quickstart) first, then moving on to the [Search pipeline](search/search) documentation to learn more.

Gravwell is pleased to announce our free [Community Edition](https://www.gravwell.io/download)!
If you're confused, the [Glossary](glossary/glossary) defines some of the technical terminology used throughout the documentation.

```{note}
Gravwell is pleased to announce our new [no-licensed-required tier](https://www.gravwell.io/blog/gravwell-5.6.0-new-license-tiers) (and the Community Advanced option for even more free ingest)!
```

```{toctree}
---
Expand Down
58 changes: 51 additions & 7 deletions quickstart/quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,37 @@
<script>
function calculate(){
var ingest = document.getElementById("ingest").value;
if (ingest == 0) { return; }
var cores = Math.ceil(ingest / 30);
var memory = Math.ceil(ingest / 6);
if (cores < 2) { cores = 2; }
if (memory < 4) { memory = 4; }
var indexers = Math.ceil(ingest / 250);
cores = Math.ceil(cores / indexers);
memory = Math.ceil(memory / indexers);
memory = Math.pow(2, Math.ceil(Math.log(memory)/Math.log(2)));
cores = Math.pow(2, Math.ceil(Math.log(cores)/Math.log(2)));
document.getElementById("corecount").innerHTML = cores;
document.getElementById("memsize").innerHTML = memory;
document.getElementById("indexerCount").innerHTML = indexers;
}
function calculateStorage(){
var daily = document.getElementById("dailydata").value;
var retention = document.getElementById("retention").value;
var compressionFactor = document.getElementById("compressionFactor").value;
var accelRatio = 2.0
if (document.getElementById('ftIndex').checked) {
accelRatio = 2.0
}
if (document.getElementById('ftBloom').checked) {
accelRatio = 1.1
}
if (document.getElementById('none').checked) {
accelRatio = 1.0
}
var storage = Math.ceil(accelRatio * daily * retention * compressionFactor);
document.getElementById("storageSize").innerHTML = storage;
}
calculate()
</script>

# Quick Start
Expand Down Expand Up @@ -53,21 +76,42 @@ Open-Source Library Licenses </open_source>
(system_requirements)=
## System Requirements

### Hardware
(hardware_calculator)=
### Hardware Estimator

We strongly recommend at least 4GB of RAM and 2 CPU cores to run Gravwell Community Edition; this should be sufficient up to the CE ingest limit. If you have a paid license, you should scale up your hardware as your daily data ingest increases. In general, we recommend the following rule of thumb:

* One CPU core per 30 GB/day ingest.
* 1 GB of RAM per 6 GB/day ingest.

Note that as your ingest scales to the hundreds or thousands of gigabytes per day, you'll usually want more cores and memory per indexer, as queries simply tend to become more "expensive" -- and you're often running more of them in such a large deployment!

We provide a basic calculator below; just enter your expected daily ingest in gigabytes:

| | |
| -------------------------------- | -------------------------------------------------------------------------------- |
| **Expected Ingest Per Day (GB)** | <input type='number' id='ingest' onInput='calculate()' placeholder="Gigabytes"/> |
| **Recommended CPU Core Count** | <span id="corecount">--</span> |
| **Recommended RAM** | <span id="memsize">--</span> GB |
| | |
|----------------------------------------------------------------|----------------------------------------------------------------------------------|
| **Expected Ingest Per Day (GB)** | <input type='number' id='ingest' onInput='calculate()' placeholder="Gigabytes"/> |
| <span style="color:blue">**Recommended # of Indexers**</span> | <span style="color:blue"><span id="indexerCount">--</span></span> |
| <span style="color:blue">**Recommended Per-Indexer Core Count**</span> | <span style="color:blue"><span id="corecount">--</span></span> |
| <span style="color:blue">**Recommended Per-Indexer RAM**</span> | <span style="color:blue"><span id="memsize">--</span> GB</span> |

```{note}
These are only very rough estimates, and of course performance will vary based on the hardware you select: eight Ivy Bridge cores are not comparable to eight AMD Epyc cores!
```

You can estimate your storage requirements using the following calculator:

| | |
|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| **Daily Data Volume (GB)** | <input type='number' id='dailydata' onInput='calculateStorage()' placeholder="Gigabytes"/> |
| **Required Retention (days)** | <input type='number' id='retention' onInput='calculateStorage()' placeholder="Days"/> |
| **Estimated Compression Factor†** | <input type='number' id='compressionFactor' onInput='calculateStorage()' value="0.2"> |
| **Acceleration Storage Type‡** | <div><input type="radio" id="ftIndex" name="accel" value="index" onInput='calculateStorage()' checked/><label for="ftIndex">Indexing</label><br><input type="radio" id="ftBloom" name="accel" value="bloom" onInput='calculateStorage()'/><label for="ftBloom">Bloom</label><br><input type="radio" id="none" name="accel" value="none" onInput='calculateStorage()'/><label for="none">None</label></br></div> |
| <span style="color:blue">**Recommended Storage Size**</span> | <span style="color:blue"><span id="storageSize">--</span> GB</span> |

**†** Value between 0 and 1.0 representing how compressed the raw data will be on disk. Most text-based logs compress to about 20% of their size, so we use a default of 0.2. If you intend to disable compression entirely, or if you'll be ingesting lots of binary data, set this to 1.0.

**‡** We assume fulltext acceleration for this calculator. Storing the acceleration data using true indexing is the most expensive; using a bloom filter instead can reduce storage overhead significantly at the cost of potentially worse performance. Disabling acceleration entirely (by selecting "None") is usually not recommended, as it makes queries very disk-intensive.

## Installation
Installing Gravwell on a single machine is quite simple--just follow the instructions in this section. For more advanced environments involving multiple systems, review the Advanced Topics section.
Expand Down
Loading