Panic on rule evaluation when config reloaded

[glightfoot]

We recently added recording rules with remote write to the remote write exporter and started getting the following panics on rule evaluation.

Version: 0.0.53
Remote Write Exporter running on port 9083
Config:

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 1m
  external_labels:
    source: promxy-prod-local
alerting:
  alert_relabel_configs:
  - source_labels: [alertroute]
    separator: ;
    regex: noalert
    replacement: $1
    action: drop
  - separator: ;
    regex: __replica__
    replacement: $1
    action: labeldrop
  alertmanagers:
  - dns_sd_configs:
    - names:
      - alertmanager.monitoring
      refresh_interval: 30s
      type: A
      port: 9093
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus-rules/*.yaml
remote_write:
- url: http://localhost:9083/receive
  remote_timeout: 30s
  queue_config:
    capacity: 10
    max_shards: 1000
    min_shards: 1
    max_samples_per_send: 100
    batch_send_deadline: 5s
    max_retries: 3
    min_backoff: 30ms
    max_backoff: 100ms

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1aa319e]

goroutine 1119 [running]:
github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Group).Eval.func1(0x27e22c0, 0xc0001a6500, 0xc0011ae240, 0x118f05e5, 0xed553842b, 0x3e110e0, 0x0, 0x2818e00, 0xc001090100)
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:534 +0x6ee
github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Group).Eval(0xc0011ae240, 0x27e22c0, 0xc0001a6500, 0x118f05e5, 0xed553842b, 0x3e110e0)
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:575 +0x109
github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Group).run.func1()
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:293 +0xc1
github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Group).run(0xc0011ae240, 0x27e22c0, 0xc0001a6500)
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:307 +0x257
github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Manager).Update.func1.1(0xc00013ebe0, 0xc0011ae240)
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:820 +0x60
created by github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules.(*Manager).Update.func1
	/home/tjackson/workspace/golang/src/github.com/jacksontj/promxy/vendor/github.com/prometheus/prometheus/rules/manager.go:815 +0x56

[glightfoot]

After some more investigation, this happens whenever we issue a reload to update rules

[jacksontj]

@glightfoot thanks for the report (and repro case!). I was able to reproduce and have a fix (#233).

I've also published a release with the fix -- https://github.com/jacksontj/promxy/releases/tag/v0.0.54

[glightfoot]

Thanks for the quick fix!