Run Experiment on Openwhisk raise NotImplementedError

[DaY1zz]

When I tried to run the Experiment following the instruction (https://github.com/spcl/serverless-benchmarks/blob/master/docs/usage.md#experiment) at the deployment of Openwhisk.
./sebs.py experiment process perf-cost --config example.json --deployment openwhisk
It raised a NotImplementedError as below.

And I looked for this method in the source code. It directly raised the Error. So how can I run the Experiment correctly？

[mcopik]

@DaY1zz Thanks for raising the issue! Indeed, the enforcement of cold start is not implemented on OpenWhisk.

Let me check if this feature can be added easily. Meanwhile, you can run the perf-cost experiment for warm instances.

[DaY1zz]

@mcopik Thanks for your reply. So, can you tell me which experiments are implemented on OpenWhisk? And I have successfully tried the perf-cost experiment for warm instances.

[mcopik]

@DaY1zz You should be able to run the perf-cost experiment with experiments: warm and sequential. Other types require enforcing cold starts.

https://github.com/spcl/serverless-benchmarks/blob/master/config/example.json#L14

[harnaman-hk]

Hi @mcopik, can I work on this issue?

[mcopik]

@harnaman-hk Yes, please!

I'd recommend starting with the environment of OpenWhisk actions - see if updating any part of the configuration invalidates existing containers, and the consecutive invocations will be cold. On AWS and Azure we update environment variables, this might also work on OpenWhisk.