Benchmark a real-time planning algorithm using Optaplanner

2
votes

I'm trying to benchmark a real-time planning algorithm but can't seem to find how to do it, is this supported in Optaplanner?

I've successfully run a benchmark using an offline version of my problem. I've implemented SolutionFileIO that reads my problem instances and converts them to a solution. I've read the docs and saw the video related to benchmarking but couldn't find what I'm looking for.

Alternatively, I can run the real-time algorithms using my own framework, but that would require me to manually define all Optaplanner heuristics that I want to run (which is quite cumbersome when using a matrix setup). Is there a way to instantiate the solvers (in Java) based on the benchmark xml definition? This would allow me to run my own real-time benchmark while still using the Optaplanner benchmark definition.

1
benchmarkingoptaplanner
With real-time planning, can we assume you mean that you're firing ProblemFactChanges at it? - Geoffrey De Smet
I actually didn't implement it using ProblemFactChange (but could've done so). Upon problem change, I'm stopping the algorithm and starting it with the new problem definition including latest best schedule. So essentially it is a warm restart. - rinde
Sounds like you should be using ProblemFactChange as that's potentially more efficient/faster. Any reason why you're not? - Geoffrey De Smet
Yes, I'm aware of that, thanks. I'm not (yet) doing that because the current setup requires less implementation, my simulator RinSim already gathers all necessary information. If this setup isn't fast enough I will definitely consider using ProblemFactChange. - rinde

1 Answers

2
votes

A benchmark config that also fires ProblemFactChange events (= real-time planning), is not yet supported, vote for this jira. How would you like the benchmark config to look like?

To HACK reusing the solvers from a benchmark configuration, cast PlannerBenchmark to PlannerBenchmarkRunner and use getPlannerBenchmarkResult().getSolverBenchmarkResultList(), but that will give up on a bunch of orchestration (including the report). Instead, if you can succeed in overriding SubSingleBenchmarkResult, you wouldn't loose that orchestration (but your hacks would be even deeper).

Whatever you end up doing, do share how you'd the benchmark config to look like, as this will give us inspiration when we implement it for a future OptaPlanner version.