Skip to content

Commit 519eaf6

Browse files
comstudcomstud
authored
Release 0.99.0
 * one config to rule them all * Have fletchling load all active nests from DB and that's it. no filtering. Reserve mucking around with what you want to be active for external tools or API calls. * fix missing db rows.Close()s. Oops. * add refresh=1 query param to /api/config/reload to compute spawnpoint counts if any are unknown and re-run filters before reloading nests. this can run for a long time if there are a lot nests in the DB as it will also filter overlapping nests according to max overlap percent. * add spawnpoints=all query param to /api/config/reload to re-compute spawnpoints for ALL nests. This also re-runs all filters before reloading nests. * add concurrency=# query param to /api/config/reload to change amount of concurrency if refreshing. defaults to 4. * have the importer error if migrations need to be run. fletchling should be restarted to run them first before using the importer tool. * require go 1.22 * keep track of area parents and use that as part of area_name. This allows configuring the webhooks exactly like Golbat * Add publish workflow, update docker image, README, etc.
1 parent 758f99a commit 519eaf6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1840
-1038
lines changed

‎.dockerignore

+8
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,10 @@
1+
*~
2+
.dockerignore
3+
*.example
14
/logs
25
/.git
6+
/docker-compose.yml*
7+
/fletchling
8+
/fletchling-osm-importer
9+
/Makefile
10+
/configs

‎.github/workflows/publish.yml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
name: publish
2+
3+
on: [push]
4+
5+
env:
6+
REGISTRY: ghcr.io
7+
IMAGE_NAME: ${{ github.repository }}
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- name: Checkout repository
14+
uses: actions/checkout@v4
15+
- name: Setup Go
16+
uses: actions/setup-go@v5
17+
with:
18+
go-version: '1.22.x'
19+
- name: Install dependencies
20+
run: go get ./...
21+
- name: Test with the Go CLI
22+
run: go test -v ./...
23+
build-and-push-image:
24+
runs-on: ubuntu-latest
25+
needs: [test]
26+
permissions:
27+
contents: read
28+
packages: write
29+
steps:
30+
- name: Checkout repository
31+
uses: actions/checkout@v4
32+
- name: Set up QEMU
33+
uses: docker/setup-qemu-action@v2
34+
- name: Set up Docker Buildx
35+
uses: docker/setup-buildx-action@v2
36+
- name: Log in to the Container registry
37+
uses: docker/login-action@v3
38+
with:
39+
registry: ${{ env.REGISTRY }}
40+
username: ${{ github.actor }}
41+
password: ${{ secrets.GITHUB_TOKEN }}
42+
- name: Extract metadata (tags, labels) for Docker
43+
id: meta
44+
uses: docker/metadata-action@v5
45+
with:
46+
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
47+
- name: Build and push Docker image
48+
uses: docker/build-push-action@v5
49+
with:
50+
context: .
51+
labels: ${{ steps.meta.outputs.labels }}
52+
platforms: linux/amd64,linux/arm64
53+
push: true
54+
tags: ${{ steps.meta.outputs.tags }}

‎Dockerfile

+3-2
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,20 @@
11
# Build image
2-
FROM golang:1.21-alpine as build
2+
FROM golang:1.22-alpine as build
33

44
WORKDIR /go/src/app
55

66
COPY . .
77
RUN if [ ! -f vendor/modules.txt ]; then go mod download; fi
88
RUN CGO_ENABLED=0 go build -tags go_json -o /go/bin/fletchling ./bin/fletchling
99
RUN CGO_ENABLED=0 go build -o /go/bin/fletchling-osm-importer ./bin/fletchling-osm-importer
10+
RUN CGO_ENABLED=0 go build -o /go/bin/sleep ./bin/sleep
1011
RUN mkdir /empty-dir
1112

1213
# Now copy it into our base image.
1314
FROM gcr.io/distroless/static-debian11 as runner
14-
COPY --from=build /go/bin/fletchling /go/bin/fletchling-osm-importer /fletchling/
1515
COPY --from=build /empty-dir /fletchling/logs
1616
COPY --from=build /go/src/app/db_store/sql /fletchling/db_store/sql
17+
COPY --from=build /go/bin/fletchling /go/bin/fletchling-osm-importer /go/bin/sleep /fletchling/
1718

1819
WORKDIR /fletchling
1920
CMD ["./fletchling"]

‎README.md

+25-80
Original file line numberDiff line numberDiff line change
@@ -5,33 +5,37 @@ webhooks and computes nesting pokemon.
55

66
# Features
77

8-
* receives and processes pokemon on the fly via webhook from Golbat
9-
* fletchling-osm-importer (separate tool): Can copy nests from: overpass to db (or koji soon)
10-
* Koji can be used as an authortative source for nests (soon)
11-
* has an API to pull stats, purge stats, reload config, etc.
8+
* Receives and processes pokemon on the fly via webhook from Golbat
9+
* Uses global spawn data to not confuse event spawns (CD, Spotlight, etc) with nesting pokemon.
10+
* No reliance on external sites for event or current nesting mon data.
11+
* Tool for importing nests from overpass (fletchling-osm-importer)
12+
* API to pull stats, purge stats, reload config, etc.
1213

1314
# Configuration
1415

15-
1. rename and edit configs in configs/
16-
2. in golbat, add a new webhook. (i think restart is required.) it should look like this in the config:
16+
1. Copy configs/fletchling.toml.example to configs/fletchling.toml and edit it.
17+
2. In golbat, add a new webhook entry to its config. It should look like this:
1718

1819
```
1920
[[webhooks]]
20-
url = "http://your-fletchling-hostname:9042/webhook"
21+
url = "http://FLETCHLING-HOSTNAME:9042/webhook"
2122
types = ["pokemon_iv"]
2223
```
2324

24-
If you are using pm2, I would leave 'addr' in fletchling.toml as the default. Use '127.0.0.1' for your fletchling hostname in the above.
25+
Replace 'FLETCHLING-HOSTNAME' with your host that is running Fletchling. If you are using pm2 and not using
26+
docker, this should likely be '127.0.0.1'. If you are using docker, your fletchling hostname will likely be
27+
'fletchling'.
2528

26-
If you are using the included docker-compose.yml, golbat is also running under docker, and everything is attached to the same docker network, your fletchling hostname will be 'fletchling'.
29+
After adding the config to Golbat, restart Golbat to have it take effect.
30+
31+
3. If you plan to use docker-compose, copy docker-compose.yml.example to docker-compose.yml and edit.
2732

2833
# Running
2934

3035
## docker
3136

3237
1. There is an included docker-compose.yml.example file. Copy it to docker-compose.yml and edit it, if needed.
33-
2. `docker-compose build`
34-
3. `docker-compose up -d`
38+
2. `docker-compose up -d`
3539

3640
## pm2
3741

@@ -40,90 +44,31 @@ If you are using the included docker-compose.yml, golbat is also running under d
4044

4145
# Verifying it is working
4246

43-
1. curl http://localhost:9042/api/config -- this should give you the current processor configuration
44-
2. The logfile is configurable in configs/fletchling.toml but defaults to logs/fletchling.log. Check it for errors.
45-
3. Every minute, a log message will appear saying how many pokemon were processed. If this is 0, it means that Fletchling is not getting any webhooks. Check your Golbat webhooks configuration. Check the address Fletchling is listening on (http section in config).
47+
1. `curl http://localhost:9042/api/config` -- this should give you the current processor configuration
48+
2. Check the logs in logs/ (or the log_dir that you configured in fletchling.toml) for errors.
49+
* Every minute, a log message will appear saying how many pokemon were processed. If this is 0, it means that Fletchling is not getting any webhooks. Check your Golbat webhooks configuration. Check the address Fletchling is listening on (http section in config).
4650

4751
# Migrating from other nest processors
4852

49-
## nestcollector to Fletching with database as authortative source (SIMPLEST)
53+
## nestcollector to Fletching using existing Golbat DB for nests (SIMPLEST)
5054
1. Gather your Golbat DB info.
51-
2. Create `configs/fletchling.toml` from the existing example config.
52-
* configure your existing golbat db in configs/fletchling.toml in both 'nests_db' and 'golbat_db' sections.
55+
2. Create `configs/fletchling.toml` as per 'Configuration' instructions above.
56+
* configure your existing golbat db in configs/fletchling.toml in BOTH 'nests_db' and 'golbat_db' sections.
5357
* fix the listen address in 'http' section, if necessary.
54-
4. nuke your cronjob.
55-
5. start up fletchling
56-
57-
## nestcollector to Fletchling with Koji as authorative source
58-
59-
This will be available soon.
58+
3. nuke your cronjob.
59+
4. start up fletchling per 'Running' section above.
6060

6161
## Others
6262

6363
I would just start over, personally. :)
6464

6565
# Importing OSM data
6666

67-
Overpass API can be queried to find parks, etc, to import into your nests db (or Koji soon).
68-
69-
The fences/areas that are searched can come from either a poracle-style json file, or a geojson FeatureCollection file, or Koji(soon).
70-
71-
1. If you want to find nests for your areas that are in Koji, make sure you have a project which exports your areas.
72-
2. If you want to find nests for your areas that are in a file, make note of its location.
73-
3. Gather your nests DB info/credentials. If you are currently using the Golbat DB, use this. Otherwise, if you are starting from scratch, use golbat or make a new database.
74-
4. (Optional): Gather your golbat DB info/credentials (might be same as Step 3.). min_spawnpoints filtering will only work with this configured.
75-
5. Create `configs/fletchling-osm-importer.toml` from the existing example config. The comments in the file should explain.
76-
6. `./make`
77-
7. `./fletchling-osm-importer 'AreaName'` to import a single area first, if you wish.
78-
8. `./fletchling-osm-importer -all-areas` to import all areas.
67+
Importing docs can be found [here.](./docs/IMPORTING.md)
7968

8069
# API
8170

82-
## Get config
83-
`curl http://localhost:9042/api/config`
84-
85-
## Reload configuration
86-
`curl http://localhost:9042/api/config/reload`
87-
(Also supports PUT. You can also send a SIGHUP signal to the process)
88-
89-
## Get all nests
90-
`curl http://localhost:9042/api/nests`
91-
92-
## Get single nest
93-
`curl http://localhost:9042/api/nests/:nest_id`
94-
95-
## Get all nests and full stats history
96-
`curl http://localhost:9042/api/nests/_/stats`
97-
98-
## Get single nest and its stats history
99-
`curl http://localhost:9042/api/nests/_/:nest_id`
100-
101-
Untested:
102-
103-
## Purge all stats
104-
`curl -X PUT http://localhost:9042/api/stats/purge/all`
105-
106-
This ditches all stats history including the current time period. This starts the stats with a clean slate, but like startup.
107-
108-
## Purge some duration of oldest stats
109-
`curl -X PUT http://localhost:9042/api/stats/purge/oldest -d '{ "duration_minutes": xx }'`
110-
111-
Purges the specified duration of the stats starting from the oldest. This will never remove the current unfinished time period. This can be used to nuke everything but the current time period by specifying a very high number of minutes.
112-
113-
## Purge some duration of newest stats
114-
`curl -X PUT http://localhost:9042/api/stats/purge/newest -d '{ "duration_minutes": xx, "include_current": false }'`
115-
116-
Purges the specified amount of minutes of stats starting from the newest. 'include_current' specifies whether it should start with the current time period that is not done, or if it should start at the last period.
117-
118-
## Ensure only a certain duration of stats exists
119-
`curl -X PUT http://localhost:9042/api/stats/purge/keep -d '{ "duration_minutes": xx }'
120-
121-
This is another way to purge oldest stats. But with this one, you specify the duration to keep, not the duration to purge.
122-
123-
124-
# Known issues
125-
126-
The importer can import nests that are fully contained by other nests. For example, if a large park has a number of baseball fields, it is possible that nests for both the park and the fields will be imported. This will be fixed soon.
71+
API docs can be found [here.](./docs/API.md)
12772

12873
# Enjoy!
12974

‎app_config/config.go

+61-17
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,30 @@ import (
1313
"github.com/knadh/koanf/v2"
1414
"github.com/sirupsen/logrus"
1515

16+
"github.com/UnownHash/Fletchling/areas"
1617
"github.com/UnownHash/Fletchling/db_store"
18+
"github.com/UnownHash/Fletchling/filters"
1719
"github.com/UnownHash/Fletchling/httpserver"
20+
"github.com/UnownHash/Fletchling/importer"
1821
"github.com/UnownHash/Fletchling/logging"
22+
"github.com/UnownHash/Fletchling/overpass"
1923
"github.com/UnownHash/Fletchling/processor"
2024
"github.com/UnownHash/Fletchling/pyroscope"
2125
"github.com/UnownHash/Fletchling/stats_collector"
2226
"github.com/UnownHash/Fletchling/webhook_sender"
2327
)
2428

29+
const (
30+
DEFAULT_OVERPASS_URL = "https://overpass-api.de/api/interpreter"
31+
32+
DEFAULT_FILTER_CONCURRENCY = 4
33+
DEFAULT_MIN_SPAWNPOINTS = 10
34+
DEFAULT_MIN_AREA_M2 = float64(100)
35+
DEFAULT_MAX_AREA_M2 = float64(10000000)
36+
DEFAULT_MAX_OVERLAP_PERCENT = float64(60)
37+
DEFAULT_NEST_NAME = "Unknown Nest"
38+
)
39+
2540
type KojiConfig struct {
2641
Url string `koanf:"url"`
2742
Token string `koanf:"token"`
@@ -53,21 +68,19 @@ func (cfg *KojiConfig) Validate() error {
5368
}
5469

5570
type Config struct {
56-
Processor processor.Config `koanf:"processor"`
57-
58-
Webhooks webhook_sender.WebhooksConfig `koanf:"webhooks"`
59-
WebhookSettings webhook_sender.SettingsConfig `koanf:"webhook_settings"`
60-
61-
Logging logging.Config `koanf:"logging"`
62-
HTTP httpserver.Config `koanf:"http"`
63-
64-
Koji *KojiConfig `koanf:"koji"`
65-
66-
NestsDb db_store.DBConfig `koanf:"nests_db"`
67-
GolbatDb *db_store.DBConfig `koanf:"golbat_db"`
68-
69-
Pyroscope pyroscope.Config `koanf:"pyroscope"`
70-
Prometheus stats_collector.PrometheusConfig `koanf:"prometheus"`
71+
NestsDb db_store.DBConfig `koanf:"nests_db"`
72+
GolbatDb *db_store.DBConfig `koanf:"golbat_db"`
73+
Overpass overpass.Config `koanf:"overpass"`
74+
Filters filters.Config `koanf:"filters"`
75+
Importer importer.Config `koanf:"importer"`
76+
Areas areas.Config `koanf:"areas"`
77+
WebhookSettings webhook_sender.SettingsConfig `koanf:"webhook_settings"`
78+
Webhooks webhook_sender.WebhooksConfig `koanf:"webhooks"`
79+
HTTP httpserver.Config `koanf:"http"`
80+
Logging logging.Config `koanf:"logging"`
81+
Processor processor.Config `koanf:"processor"`
82+
Pyroscope pyroscope.Config `koanf:"pyroscope"`
83+
Prometheus stats_collector.PrometheusConfig `koanf:"prometheus"`
7184
}
7285

7386
func (cfg *Config) GetPrometheusConfig() stats_collector.PrometheusConfig {
@@ -79,7 +92,18 @@ func (cfg *Config) CreateLogger(rotate bool) *logrus.Logger {
7992
}
8093

8194
func (cfg *Config) Validate() error {
82-
if err := cfg.Koji.Validate(); err != nil {
95+
if err := cfg.Areas.Validate(); err != nil {
96+
return err
97+
}
98+
99+
if err := cfg.Filters.Validate(); err != nil {
100+
return err
101+
}
102+
103+
cfg.Importer.MinAreaM2 = cfg.Filters.MinAreaM2
104+
cfg.Importer.MaxAreaM2 = cfg.Filters.MaxAreaM2
105+
106+
if err := cfg.Importer.Validate(); err != nil {
83107
return err
84108
}
85109

@@ -124,14 +148,34 @@ func (cfg *Config) Validate() error {
124148

125149
func GetDefaultConfig() Config {
126150
return Config{
151+
Areas: areas.GetDefaultConfig(),
152+
127153
Processor: processor.GetDefaultConfig(),
128154

155+
Overpass: overpass.Config{
156+
Url: DEFAULT_OVERPASS_URL,
157+
},
158+
159+
Importer: importer.Config{
160+
DefaultName: DEFAULT_NEST_NAME,
161+
MinAreaM2: DEFAULT_MIN_AREA_M2,
162+
MaxAreaM2: DEFAULT_MAX_AREA_M2,
163+
},
164+
165+
Filters: filters.Config{
166+
Concurrency: DEFAULT_FILTER_CONCURRENCY,
167+
MinSpawnpoints: DEFAULT_MIN_SPAWNPOINTS,
168+
MinAreaM2: DEFAULT_MIN_AREA_M2,
169+
MaxAreaM2: DEFAULT_MAX_AREA_M2,
170+
MaxOverlapPercent: DEFAULT_MAX_OVERLAP_PERCENT,
171+
},
172+
129173
WebhookSettings: webhook_sender.SettingsConfig{
130174
FlushIntervalSeconds: 1,
131175
},
132176

133177
Logging: logging.Config{
134-
Filename: filepath.FromSlash("logs/fletchling.log"),
178+
LogDir: filepath.FromSlash("./logs"),
135179
MaxSizeMB: 500,
136180
MaxAgeDays: 7,
137181
MaxBackups: 20,

0 commit comments

Comments
 (0)