[RLlib] Evaluation logic needs clean-up

[simonsays1980]

What happened + What you expected to happen

What happened

I tested evaluation of algorithms and noted to things:

1. Algorithm.evaluate() does not count agent steps and environment steps, if a cusotm evaluation function is used.
2. The evaluaion worker set is not None even if the evaluation_num_workers=0.

What you expected to happen

That metrics are collected identically when a custom evaluation funciton is used and that the evaluation_num_workers=0 lets us run the evaluation on the local worker in Algorothm.workers.

Versions / Dependencies

Ray nightly
Python 3.9.12
Fedora Linux 39

Reproduction script

Run our custom evaluation example.

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[sven1977]

On the second point:

The evaluaion worker set is not None even if the evaluation_num_workers=0.

This is the expected behavior, as long as evaluation_interval is != 0.

from ray.rllib.algorithms.ppo import PPOConfig

config = (
    PPOConfig()
    .environment("CartPole-v1")
    .evaluation(
        evaluation_num_env_runners=0,
    ),
)

algo = config.build()

print(algo.evaluation_workers is None)

Should print True, however, if I change evaluation_interval to 1 (default is 0), then RLlib will create a eval WorkerSet with num_workers=0 (only 1 local worker, no remote workers).

[simonsays1980]

@sven1977 Are agent and env steps now counted in a custom evaluation function with the new Metrics logger?