Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to exclude certain dates from chaos #69

Merged
merged 2 commits into from
Mar 8, 2018
Merged

Allow to exclude certain dates from chaos #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7fcd4d2
feat: specify days of a year without chaos
Mar 5, 2018
04f7317
docs: document days of year feature in readme
Mar 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ INFO[4804] Killing pod chaoskube/nginx-701339712-51nt8
...
```

`chaoskube` allows to filter target pods [by namespaces, labels and annotations](#filtering-targets) as well as [exclude certain weekdays or times of day](#limit-the-chaos) from chaos.
`chaoskube` allows to filter target pods [by namespaces, labels and annotations](#filtering-targets) as well as [exclude certain weekdays, times of day and days of a year](#limit-the-chaos) from chaos.

## How

Expand Down Expand Up @@ -140,9 +140,9 @@ spec:

## Limit the Chaos

You can limit the time when chaos is introduced by weekdays, time periods of a day or both.
You can limit the time when chaos is introduced by weekdays, time periods of a day, day of a year or all of them together.

Add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` options and/or a comma-separated list of time periods via the `--excluded-times-of-day` option and specify a `--timezone` by which to interpret them.
Add a comma-separated list of abbreviated weekdays via the `--excluded-weekdays` options, a comma-separated list of time periods via the `--excluded-times-of-day` option and/or a comma-separated list of days of a year via the `--excluded-days-of-year` option and specify a `--timezone` by which to interpret them.

Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). If you're testing `chaoskube` from your local machine then `Local` makes the most sense. Once you deploy `chaoskube` to your cluster you should deploy it with a specific timezone, e.g. where most of your team members are living, so that both your team and `chaoskube` have a common understanding when a particular weekday begins and ends, for instance. If your team is spread across multiple time zones it's probably best to pick `UTC` which is also the default. Picking the wrong timezone shifts the meaning of a particular weekday by a couple of hours between you and the server.

Expand All @@ -156,6 +156,7 @@ Use `UTC`, `Local` or pick a timezone name from the [(IANA) tz database](https:/
| `--namespaces` | namespace selector to filter pods by | (all namespaces) |
| `--excluded-weekdays` | weekdays when chaos is to be suspended, e.g. "Sat,Sun" | (no weekday excluded) |
| `--excluded-times-of-day` | times of day when chaos is to be suspended, e.g. "22:00-08:00" | (no times of day excluded) |
| `--excluded-days-of-year` | days of a year when chaos is to be suspended, e.g. "Apr1,Dec24" | (no days of year excluded) |
| `--timezone` | timezone from tz database, e.g. "America/New_York", "UTC" or "Local" | (UTC) |
| `--dry-run` | don't kill pods, only log what would have been done | true |

Expand Down
26 changes: 20 additions & 6 deletions chaoskube/chaoskube.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ type Chaoskube struct {
ExcludedWeekdays []time.Weekday
// a list of time periods of a day when termination is suspended
ExcludedTimesOfDay []util.TimePeriod
// a list of days of a year when termination is suspended
ExcludedDaysOfYear []time.Time
// the timezone to apply when detecting the current weekday
Timezone *time.Location
// an instance of logrus.StdLogger to write log messages to
Expand All @@ -50,21 +52,26 @@ var (
msgWeekdayExcluded = "weekday excluded"
// msgTimeOfDayExcluded is the log message when termination is suspended due to the time of day filter
msgTimeOfDayExcluded = "time of day excluded"
// msgDayOfYearExcluded is the log message when termination is suspended due to the day of year filter
msgDayOfYearExcluded = "day of year excluded"
)

// New returns a new instance of Chaoskube. It expects a kubernetes client, a
// label, annotation and/or namespace selector to reduce the amount of affected
// pods as well as whether to enable dryRun mode and a seed to seed the randomizer
// with. You can also provide a list of weekdays and corresponding time zone when
// chaoskube should be inactive.
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, logger log.FieldLogger, dryRun bool) *Chaoskube {
// New returns a new instance of Chaoskube. It expects:
// * a Kubernetes client to connect to a Kubernetes API
// * label, annotation and/or namespace selectors to reduce the amount of possible target pods
// * a list of weekdays, times of day and/or days of a year when chaos mode is disabled
// * a time zone to apply to the aforementioned time-based filters
// * a logger implementing logrus.FieldLogger to send log output to
// * whether to enable/disable dry-run mode
func New(client kubernetes.Interface, labels, annotations, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, logger log.FieldLogger, dryRun bool) *Chaoskube {
return &Chaoskube{
Client: client,
Labels: labels,
Annotations: annotations,
Namespaces: namespaces,
ExcludedWeekdays: excludedWeekdays,
ExcludedTimesOfDay: excludedTimesOfDay,
ExcludedDaysOfYear: excludedDaysOfYear,
Timezone: timezone,
Logger: logger,
DryRun: dryRun,
Expand Down Expand Up @@ -146,6 +153,13 @@ func (c *Chaoskube) TerminateVictim() error {
}
}

for _, d := range c.ExcludedDaysOfYear {
if d.Day() == now.Day() && d.Month() == now.Month() {
c.Logger.WithField("dayOfYear", now.Format(util.YearDay)).Debug(msgDayOfYearExcluded)
return nil
}
}

victim, err := c.Victim()
if err == errPodNotFound {
c.Logger.Debug(msgVictimNotFound)
Expand Down
86 changes: 84 additions & 2 deletions chaoskube/chaoskube_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ func (suite *Suite) TestNew() {
namespaces, _ = labels.Parse("qux")
excludedWeekdays = []time.Weekday{time.Friday}
excludedTimesOfDay = []util.TimePeriod{util.TimePeriod{}}
excludedDaysOfYear = []time.Time{time.Now()}
)

chaoskube := New(
Expand All @@ -48,6 +49,7 @@ func (suite *Suite) TestNew() {
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
time.UTC,
logger,
false,
Expand All @@ -60,6 +62,7 @@ func (suite *Suite) TestNew() {
suite.Equal("qux", chaoskube.Namespaces.String())
suite.Equal(excludedWeekdays, chaoskube.ExcludedWeekdays)
suite.Equal(excludedTimesOfDay, chaoskube.ExcludedTimesOfDay)
suite.Equal(excludedDaysOfYear, chaoskube.ExcludedDaysOfYear)
suite.Equal(time.UTC, chaoskube.Timezone)
suite.Equal(logger, chaoskube.Logger)
suite.Equal(false, chaoskube.DryRun)
Expand Down Expand Up @@ -102,6 +105,7 @@ func (suite *Suite) TestCandidates() {
namespaceSelector,
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand Down Expand Up @@ -134,6 +138,7 @@ func (suite *Suite) TestVictim() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand All @@ -150,6 +155,7 @@ func (suite *Suite) TestNoVictimReturnsError() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand All @@ -176,6 +182,7 @@ func (suite *Suite) TestDeletePod() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
tt.dryRun,
)
Expand Down Expand Up @@ -210,6 +217,7 @@ func (suite *Suite) TestTerminateVictim() {
for _, tt := range []struct {
excludedWeekdays []time.Weekday
excludedTimesOfDay []util.TimePeriod
excludedDaysOfYear []time.Time
now func() time.Time
timezone *time.Location
remainingPodCount int
Expand All @@ -218,6 +226,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
1,
Expand All @@ -226,6 +235,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -234,6 +244,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -242,6 +253,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(24 * time.Hour) },
time.UTC,
1,
Expand All @@ -250,6 +262,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(7 * 24 * time.Hour) },
time.UTC,
2,
Expand All @@ -258,6 +271,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(+2 * time.Hour) },
time.UTC,
1,
Expand All @@ -266,6 +280,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(+24 * time.Hour) },
time.UTC,
2,
Expand All @@ -274,6 +289,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
australia,
1,
Expand All @@ -282,6 +298,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
australia,
1,
Expand All @@ -290,6 +307,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{time.Monday, time.Friday},
[]util.TimePeriod{},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -298,6 +316,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{morning, afternoon},
[]time.Time{},
ThankGodItsFriday{}.Now,
time.UTC,
2,
Expand All @@ -306,6 +325,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-15 * time.Hour) },
time.UTC,
2,
Expand All @@ -314,6 +334,7 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-17 * time.Hour) },
time.UTC,
1,
Expand All @@ -322,17 +343,75 @@ func (suite *Suite) TestTerminateVictim() {
{
[]time.Weekday{},
[]util.TimePeriod{midnight},
[]time.Time{},
func() time.Time { return ThankGodItsFriday{}.Now().Add(-13 * time.Hour) },
time.UTC,
1,
},
// this day of year is excluded, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
ThankGodItsFriday{}.Now(), // today
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// this day of year in year 0 is excluded, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 24, 0, 00, 00, 00, time.UTC), // same year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// matching works fine even when multiple days-of-year are provided, no pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 25, 10, 00, 00, 00, time.UTC), // different year day
time.Date(0, 9, 24, 10, 00, 00, 00, time.UTC), // same year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
2,
},
// there is an excluded day of year but it's not today, one pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 9, 25, 10, 00, 00, 00, time.UTC), // different year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
1,
},
// there is an excluded day of year but the month is different, one pod should be killed
{
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{
time.Date(0, 10, 24, 10, 00, 00, 00, time.UTC), // different year day
},
func() time.Time { return ThankGodItsFriday{}.Now() },
time.UTC,
1,
},
} {
chaoskube := suite.setupWithPods(
labels.Everything(),
labels.Everything(),
labels.Everything(),
tt.excludedWeekdays,
tt.excludedTimesOfDay,
tt.excludedDaysOfYear,
tt.timezone,
false,
)
Expand All @@ -356,6 +435,7 @@ func (suite *Suite) TestTerminateNoVictimLogsInfo() {
labels.Everything(),
[]time.Weekday{},
[]util.TimePeriod{},
[]time.Time{},
time.UTC,
false,
)
Expand Down Expand Up @@ -406,13 +486,14 @@ func (suite *Suite) assertLog(level log.Level, msg string, fields log.Fields) {
}
}

func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, dryRun bool) *Chaoskube {
func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, dryRun bool) *Chaoskube {
chaoskube := suite.setup(
labelSelector,
annotations,
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
timezone,
dryRun,
)
Expand All @@ -430,7 +511,7 @@ func (suite *Suite) setupWithPods(labelSelector labels.Selector, annotations lab
return chaoskube
}

func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, timezone *time.Location, dryRun bool) *Chaoskube {
func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Selector, namespaces labels.Selector, excludedWeekdays []time.Weekday, excludedTimesOfDay []util.TimePeriod, excludedDaysOfYear []time.Time, timezone *time.Location, dryRun bool) *Chaoskube {
logOutput.Reset()

return New(
Expand All @@ -440,6 +521,7 @@ func (suite *Suite) setup(labelSelector labels.Selector, annotations labels.Sele
namespaces,
excludedWeekdays,
excludedTimesOfDay,
excludedDaysOfYear,
timezone,
logger,
dryRun,
Expand Down
Loading