Support for completely custom load pattern / shape

[max-rocket-internet]

Describe the solution you'd like

To be able to programmatically control the user and hatch rate over a period of time. This would allow us to scale up and down and generate spikes which allows more accurate representation of real world load.

We can generate this easily:

But not this:

And not this either:

Describe alternatives you've considered

I tried something quite hacky but it's not good enough:

def traffic_spike(spike_after, spike_length, normal_wait_time):
    def wait_time_func(self):
        if not hasattr(self,"_traffic_spike_start"):
            self._traffic_spike_start = time()
            return normal_wait_time
        else:
            run_time = time() - self._traffic_spike_start
            if run_time > (spike_after + spike_length) or run_time < spike_after:
                return normal_wait_time
            else:
                return 0

    return wait_time_func

class QuickstartUser(HttpUser):
    wait_time = traffic_spike(600, 300, 1)

    @task
    def index_page(self):
        self.client.get("/hello")
        self.client.get("/world")

Also I know we could implement this outside of locust and call the locust API but it's not ideal.

Additional context

k6 has something called stages that looks great:

export let options = {
  stages: [
    { duration: '30s', target: 20 },
    { duration: '1m30s', target: 10 },
    { duration: '20s', target: 0 },
  ],
};

It would be great to see something like this in locust. Perhaps it could look like this:

from locust import User, TaskSet, between, LoadShape


class LoadWithSpike(LoadShape):
    stages = [
        {'duration': 3600, 'users' 1000, 'hatch': 5},
        {'duration': 300, 'users' 5000, 'hatch': 50},
        {'duration': 1800, 'users' 1000, 'hatch': -50}
    ]


class MyUser(HttpUser):
    wait_time = between(1, 3)

    @task(2)
    def index(self):
        self.client.get("/")

[cyberw]

Sounds nice, and I definitely see the need for it. A PR would be welcome :)

[heyman]

I agree that this is a feature that would be really nice to have. I've been thinking about this quite a bit, but at the moment I don't have the time to start testing things out / implementing stuff, though here are some thoughts:

I think we should start with an even lower level API for a test plan. Perhaps a class that has one or more methods that gets called every second (or similar) with the current run time in seconds, and which can return the current user count and spawn rate, as well as a way for it to end the test.

class CustomPlan(TestPlan):
    def tick(run_time, current_user_count, current_spawn_rate):
        if run_time < 600:
            return UserCount(600, 100)
        elif run_time < 3600:
            return UserCount(2000, 100)
        elif run_time < 3900:
            return UserCount(5000, 50)
        elif run_time < 4500:
            return UserCount(1000, 100)
        else:
            return StopPlan()

This API could then support implementing an API on top of it, similar to the one you suggested @max-rocket-internet

Please note that the above code was written on the fly from the top of my head, and there might be issues with it that I haven't thought about. For example, we probably want to support specifying the exact number of users for specific User classes. Also, I'm imagining it as something that runs within the master node, and I'm not sure if there's any limitations that comes from that.

[max-rocket-internet]

A PR would be welcome :)

I would be keen but have holiday coming up. Happy to look at it afterwards.

I think we should start with an even lower level API for a test plan. Perhaps a class that has one or more methods that gets called every second (or similar) with the current run time in seconds

Sounds good.

This API could then support implementing an API on top of it, similar to the one you suggested

Also good.

Also, I'm imagining it as something that runs within the master node, and I'm not sure if there's any limitations that comes from that.

I think that's correct.

Any idea where this would go? Something similar to stepload_worker?

[heyman]

Any idea where this would go? Something similar to stepload_worker?

Unfortunately the stepload implementation isn't very good and I think it should be removed and re-implemented on top of the test plan API.

[max-rocket-internet]

@heyman or @cyberw could you just give me a rough pointer of where this would go in the code base? Something in spawn_users?

I will start working on it next week.

[max-rocket-internet]

Perhaps a class that has one or more methods that gets called every second (or similar)

@heyman where would this method get called? I can't work out a good place to put this part.

[cyberw]

In the end, it is Runner.spawn_users().hatch() that needs to know what to do, so maybe it can be called from there?

[max-rocket-internet]

Runner.spawn_users().hatch() is only called on each worker though, I was thinking to have it run on the master.

[cyberw]

Ok, then I dont have any bright ideas. Tbh, I think the code around spawning users is a little convoluted and could probably do with some refactoring (I havent spent a lot of time looking in to it though)

[max-rocket-internet]

The problem I see is that we need to run a loop to control the users and hatch_rate numbers, which I think would go in MasterRunner somewhere, maybe duplicate start() to a new one like start_custom() but these functions need to return, so not really the right place to run an infinite loop in 🤔

[max-rocket-internet]

Maybe I could create start_custom() that is similar to the existing start() but it:

1. It gets the initial user_count and hatch_rate from the custom CustomPlan(TestPlan) class
2. Runs self.server.send_to_client() using numbers from step 1
3. Starts a greenlet on the master that will get the user_count and hatch_rate from the custom CustomPlan(TestPlan) every second, this runs in the background until test is stopped
4. Returns like normal

[cyberw]

would it not be possible to use the same way for ”non-custom” runs? I’d prefer to only maintain one ramping feature :)

Also, I think everything should be done in the greenlet if possible (do 1 & 2 in the greenlet)

Other than that it sounds ok!

[max-rocket-internet]

@cyberw

would it not be possible to use the same way for ”non-custom” runs? I’d prefer to only maintain one ramping feature

Correct me if I'm wrong but the current process is like this:

1. User selects user_count/hatch_rate on master (UI or via its API)
2. Master calculates what user_count/hatch_rate to send to each worker based on how many workers there are
3. Master sends the request to workers and they start hatching and when reaching user_count, they are simply running
4. Master is reasonably idle at this point, just collecting stats, runs UI.

Currently we are editing a running test via the locust API to change the user_count/hatch_rate dynamically to get a custom load test shape. So this is what I was trying to automate, like constantly editing a running test, which must happen on the master in order to do steps 2/3 above. Does that make sense? It's also how it's done for stepload_worker, this runs on the master.

Anyway, this is my current very rough work: https://github.com/locustio/locust/compare/master...max-rocket-internet:loadtest_shaper?expand=1

Any feedback welcome 🙂

I think potentially we could replace the "step load" feature with this approach. This could tidy up a lot of the code.

[cyberw]

Looks nice. We'll need a couple really of good unit tests for this, including cases

• correctly applying stop_timeout when ramping down
• make sure there is no error for parameters (-u, -t etc) not being specified when there is a shape.
• where new slaves are added during the test (I think the appropriate behaviour is trying to keep the number of users balanced across slaves but not stopping users already running just to migrate them somewhere else)

A few opinions:

• I think the name should be Shape, not Shaper
• I think the shape should be possible to have in a separate file, specified using a new parameter (--shape?). Or maybe we should just allow multiple -f arguments (as the effect of --shape would be to load another file which happens to include a shape class)

[cyberw]

Tbh, I think we could skip the "high level"/k6-like api, at least as a first step.

Being able to specify the shape with a dict is only marginally more user friendly than having a function, and many users might not understand the the flexibility you can actually have if you can't see the code (if you want to do high/low load for X amount of time for instance)

[max-rocket-internet]

We'll need a couple really of good unit tests for this

For sure.

I think the name should be Shape, not Shaper

OK, sure thing.

I think the shape should be possible to have in a separate file

Can do but this is needless complexity, no? You can already import things from other files in python.

Tbh, I think we could skip the "high level"/k6-like api, at least as a first step.
Being able to specify the shape with a dict is only marginally more user friendly

Agreed 👍

[cyberw]

I think the shape should be possible to have in a separate file

Can do but this is needless complexity, no? You can already import things from other files in python.

I just like the idea of having the load profile "orthogonal" to the User definition.

Sure, you could specify the shape file with -f and have multiple shape files import the same "actual" locustfile (with User definitions), but it feels a bit upside down and ties the load profile tightly to a locustfile.

Maybe not super important though...

[max-rocket-internet]

Please view: #1505