Skip to content

Commit

Permalink
Update benchmarks and README (#462)
Browse files Browse the repository at this point in the history
Competition benchmarks are outdated, and benchmarks vs. AoS highly depend on the hardware (see #414). Therefore, the benchmarks are removed from the README.

Also, meanwhile Arche is not the fasted Go ECS anymore (if code generation is permitted, see #412). Therefore, the feature list is rewritten.

Closes #329
Closes #414
Closes #461

* Go version upgrade for benchmarks
* use Go 1.23 in CI
* make benchmarking table functionality public
* remove ArrayOfStructs benchmarks from README
* print CPU name in benchmark tables
* upgrade testify
* print arche version above benchmarks
* cleanup README
* remove readme ref from benchmarks
* update user guide and changelog
* remove table benchmarks for deprecated methods
* add a second field to benchmark comps, document it
* add world creation and ID access to benchmarks
* addWorld.Reset to benchmarks
* rework design section ofthe README
* remove statement on OESA models
* update user guide
* remove plotting code for competition benchmarks
* remove competition benchmarks code
* link go-ecs-benchmarks in README and user guide
  • Loading branch information
mlange-42 authored Jan 8, 2025
1 parent 301c6d9 commit b5e4acc
Showing 56 changed files with 221 additions and 2,534 deletions.
103 changes: 3 additions & 100 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
@@ -17,7 +17,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Run internal benchmarks
@@ -36,7 +36,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Run internal benchmarks (tiny)
@@ -47,103 +47,6 @@ jobs:
cd benchmark
go test -tags tiny -benchmem -run=^$ -bench ^.*$ ./arche/...
relations:
name: Entity relations
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
- name: Install dependencies
run: go get .
- name: Benchmark entity relations
run: |
cd benchmark
go test -benchmem -run=^$ -bench ^.*$ ./competition/relations/...

relations_tiny:
name: Entity relations (tiny)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
- name: Install dependencies
run: go get .
- name: Benchmark entity relations (tiny)
run: |
cd benchmark
go test -tags tiny -benchmem -run=^$ -bench ^.*$ ./competition/relations/...
other_methods:
name: vs. Array of Structs
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
- name: Install dependencies
run: go get .
- name: Benchmark vs. Array of Structs
run: |
cd benchmark
go test -benchmem -run=^$ -bench ^.*$ ./competition/array_of_structs/...
competition:
name: ECS competition
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
- name: Install dependencies
run: go get .
- name: Benchmark Pos/Vel
run: |
cd benchmark
go test -benchmem -run=^$ -bench ^.*$ ./competition/pos_vel/... --count 10
- name: Benchmark Access 10 Components
run: |
cd benchmark
go test -benchmem -run=^$ -bench ^.*$ ./competition/many_components/... --count 10
- name: Benchmark Add/Remove Components
run: |
cd benchmark
go test -benchmem -run=^$ -bench ^.*$ ./competition/add_remove/... --count 10
competition_tiny:
name: ECS competition (tiny)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
- name: Install dependencies
run: go get .
- name: Benchmark Pos/Vel (tiny)
run: |
cd benchmark
go test -tags tiny -benchmem -run=^$ -bench ^.*$ ./competition/pos_vel/... --count 5
- name: Benchmark Access 10 Components (tiny)
run: |
cd benchmark
go test -tags tiny -benchmem -run=^$ -bench ^.*$ ./competition/many_components/... --count 5
- name: Benchmark Add/Remove Components (tiny)
run: |
cd benchmark
go test -tags tiny -benchmem -run=^$ -bench ^.*$ ./competition/add_remove/... --count 5
tables:
name: Benchmark Tables
runs-on: ubuntu-latest
@@ -152,7 +55,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Benchmark tables
2 changes: 1 addition & 1 deletion .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.21.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get ./...
- name: Run doc tests
16 changes: 8 additions & 8 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Build Linux
@@ -39,7 +39,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Check out code
uses: actions/checkout@v2
- name: Install dependencies
@@ -61,7 +61,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Check out code
uses: actions/checkout@v2
- name: Install dependencies
@@ -79,7 +79,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Check out code
uses: actions/checkout@v2
- name: Install dependencies
@@ -98,7 +98,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: |
go get .
@@ -128,7 +128,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: |
go get .
@@ -147,7 +147,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Run examples
@@ -177,7 +177,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v3
with:
go-version: '1.22.x'
go-version: '1.23.x'
- name: Install dependencies
run: go get .
- name: Run examples
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -4,6 +4,11 @@

* Optimizes entity creation by an altered growth policy for archetypes, entity list and entity pool (#464)

### Documentation

* Rewrites features and benchmarks sections of the README (#462)
* Adds version and CPU information to the benchmark tables (#462)

### Other

* Use mask pointers in all tests and benchmarks (#460)
97 changes: 11 additions & 86 deletions README.md
Original file line number Diff line number Diff line change
@@ -10,10 +10,6 @@

*Arche* is an [archetype](https://mlange-42.github.io/arche/background/architecture/)-based [Entity Component System](https://en.wikipedia.org/wiki/Entity_component_system) for [Go](https://go.dev/).

*Arche* is designed for the use in simulation models of the
[Department of Ecological Modelling](https://www.ufz.de/index.php?en=34213) at the
[Helmholtz Centre for Environmental Research](https://www.ufz.de).

<div align="center" width="100%">

&mdash;&mdash;
@@ -23,13 +19,11 @@

## Features

* Simple [core API](https://pkg.go.dev/github.com/mlange-42/arche/ecs). See the [API docs](https://pkg.go.dev/github.com/mlange-42/arche).
* Optional logic [filter](https://pkg.go.dev/github.com/mlange-42/arche/filter) and type-safe [generic](https://pkg.go.dev/github.com/mlange-42/arche/generic) API.
* Entity relations as first-class feature. See the [User Guide](https://mlange-42.github.io/arche/guide/relations/).
* World serialization and deserialization with [arche-serde](https://github.com/mlange-42/arche-serde).
* Designed for performance and highly optimized. See the [Benchmarks](#benchmarks).
* Well-documented [API](https://pkg.go.dev/github.com/mlange-42/arche) and comprehensive [User Guide](https://mlange-42.github.io/arche/).
* No systems. Just queries. Use your own structure (or the [Tools](#tools)).
* No dependencies. Except for unit tests ([100% coverage](https://coveralls.io/github/mlange-42/arche)).
* Probably the fastest Go ECS out there. See the [Benchmarks](#benchmarks).
* No dependencies. Except for unit tests (100% [test coverage](https://coveralls.io/github/mlange-42/arche)).
* World serialization and deserialization with [arche-serde](https://github.com/mlange-42/arche-serde).

## Installation

@@ -136,10 +130,10 @@ There is neither an update loop nor systems.
These should be implemented by the user.
For a batteries-included implementation, see module [arche-model](https://github.com/mlange-42/arche-model).

The packages [filter](https://pkg.go.dev/github.com/mlange-42/arche/filter)
and [generic](https://pkg.go.dev/github.com/mlange-42/arche/generic)
provide a layer around the core for richer resp. generic queries and manipulation.
They are built on top of the `ecs` package, so they could also be implemented by a user.
The type-safe generic API and advanced logic filters are provided in the packages
[generic](https://pkg.go.dev/github.com/mlange-42/arche/generic) and
[filter](https://pkg.go.dev/github.com/mlange-42/arche/filter), respectively.
Both packages are built on top of the core [ecs](https://pkg.go.dev/github.com/mlange-42/arche/ecs) package, so they could also be implemented by a user.

### Determinism

@@ -153,7 +147,7 @@ However, given the same operations on the `ecs.World`, iteration order will alwa
It panics on unexpected operations, like removing a dead entity,
adding a component that is already present, or attempting to change a locked world.
This may not seem idiomatic for Go.
However, explicit error handling in performance hotspots is not an option.
However, explicit error handling in performance hot spots is not an option.
Neither is silent failure, given the scientific background.

### Limitations
@@ -165,77 +159,8 @@ Neither is silent failure, given the scientific background.

A tabular overview of the runtime cost of typical *Arche* ECS operations is provided under [benchmarks](https://mlange-42.github.io/arche/background/benchmarks/) in the Arche's [User Guide](https://mlange-42.github.io/arche/).

See also the latest [Benchmarks CI run](https://github.com/mlange-42/arche/actions/workflows/benchmarks.yml).

### Arche vs. other Go ECS implementations

To the best of the author's knowledge, there are only a handful of ECS implementations in Go that are serious and somewhat maintained:

* [go-gameengine-ecs](https://github.com/marioolofo/go-gameengine-ecs)
* [Donburi](https://github.com/yohamta/donburi)
* [Ento](https://github.com/wwfranczyk/ento)
* [unitoftime/ecs](https://github.com/unitoftime/ecs)

Here, *Arche* is benchmarked against these implementations.
Feel free to open an issue if you have suggestions for improvements on the benchmarking code or other engines to include.

#### Position/Velocity

Build:
* Create 1000 entities with `Pos{float64, float64}` and `Vel{float64, float64}`.
* Create 9000 entities with only `Pos{float64, float64}`.

Iterate:
* Iterate all entities with `Pos` and `Vel`, and add `Vel` to `Pos`.

Benchmark code: [`benchmark/competition/pos_vel`](https://github.com/mlange-42/arche/tree/main/benchmark/competition/pos_vel).

<div align="center" width="100%">

![Benchmark vs. Go ECSs - Pos/Vel](https://github.com/mlange-42/arche/assets/44003176/7b73f9d8-238c-4d7a-98a1-267ad0b5e4a8)
*Position/Velocity benchmarks of Arche (left-most) vs. other Go ECS implementations.
Left panel: query iteration (log scale), right panel: world setup and entity creation.*
</div>

#### Add/remove component

Build:
* Create 1000 entities with `Pos{float64, float64}`.

Iterate:
* Get all entities with `Pos`, and add `Vel{float64, float64}` component.
* Get all entities with `Pos` and `Vel`, and remove `Vel` component.

> Note: The iteration is performed once before benchmarking,
> to avoid biasing slower implementations through one-time allocations.
Benchmark code: [`benchmark/competition/add_remove`](https://github.com/mlange-42/arche/tree/main/benchmark/competition/add_remove).

<div align="center" width="100%">

![Benchmark vs. Go ECSs - Add/remove](https://github.com/mlange-42/arche/assets/44003176/7a127568-e71a-441f-91b0-6e626b3fcf19)
*Add/remove component benchmarks of Arche (left-most) vs. other Go ECS implementations.
Left panel: iteration, right panel: world setup and entity creation.*
</div>

### Arche vs. Array of Structs

The plot below shows CPU time benchmarks of Arche (black) vs. Array of Structs (AoS, red) and Array of Pointers (AoP, blue) (with structs escaped to the heap).

Arche takes a constant time of just over 2ns per entity, regardless of the memory per entity (x-axis) and the number of entities (line styles).
For AoS and AoP, time per access increases with memory per entity as well as number of entities, due to cache misses.

In the given example with components of 16 bytes each, from 64 bytes per entity onwards (i.e. 4 components or 8 `float64` values),
Arche outperforms AoS and AoP, particularly with a large number of entities.
Note that the maximum shown here corresponds to only 25 MB of entity data!

Benchmark code: [`benchmark/competition/array_of_structs`](https://github.com/mlange-42/arche/tree/main/benchmark/competition/array_of_structs).

<div align="center" width="100%">

![Benchmark vs. AoS and AoP](https://user-images.githubusercontent.com/44003176/237245154-0070bba0-c8fe-447e-a710-e370af1dcdab.svg)
*CPU benchmarks of Arche (black) vs. Array of Structs (AoS, red) and Array of Pointers (AoP, blue).*
</div>
See also the latest [Benchmarks CI run](https://github.com/mlange-42/arche/actions/workflows/benchmarks.yml)
and the [go-ecs-benchmarks](https://github.com/mlange-42/go-ecs-benchmarks) repository.

## Cite as

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
7 changes: 0 additions & 7 deletions benchmark/competition/README.md

This file was deleted.

Loading

0 comments on commit b5e4acc

Please sign in to comment.