International Planning Competition 2023 - Numeric Tracks

This is the website for the numeric tracks of the IPC 2023.

Calls

Please forward the following calls to all interested parties.

• Call for Domains

Preliminary Schedule

Tracks

Optimal Track

• single CPU core
• 8Gb memory limit
• 30min time limit
• Plans must be optimal
• The score of a planner is the number of solved tasks
• If a suboptimal or invalid plan is returned, all tasks in the domain are counted as unsolved.
• If that happens in more than one domain, the entry is disqualified.

Satisficing Track

• single CPU core
• 8Gb memory limit
• 30min time limit
• Multiple plans can be returned, the one with the lowest cost is counted.
• The score of a planner on a solved task is the ratio C/C where C is the cost of the cheapest discovered plan and C* is the cost of a reference plan. The score on an unsolved task is 0. The score of a planner is the sum of its scores for all tasks.
• If an invalid plan is returned, all tasks in the domain are counted as unsolved.
• If that happens in more than one domain, the entry is disqualified.

Agile Track

• single CPU core
• 8Gb memory limit
• 5min time limit
• The cost of the discovered plan is ignored, only the CPU time to discover a plan is counted.
• The score of a planner on a solved task is 1 if the task was solved within 1 second and 0 if the task was not solved within the resource limits. If the task was solved in T seconds (1 ≤ T ≤ 300) then its score is 1 - log(T)/log(300). The score of a planner is the sum of its scores for all tasks.
• If an invalid plan is returned, all tasks in the domain are counted as unsolved.
• If that happens in more than one domain, the entry is disqualified.

PDDL Fragment

We will use a subset of PDDL 2.1, considering two fragments:

• Simple Numeric Planning (SNP): Numeric variables can only be increased or decreased by a constant using the increase and decrease operations.
• Linear Numeric Planning (LNP): In addition to SNP, numeric variables can now be also assigned directly using the assign operator and the right-hand of an effect can now be a linear combination.

Preconditions and goals are arbitrary propositional formulas. That is, they include quantifiers, negative preconditions, disjunctive preconditions and any number of numeric predicates. Conditional effects are not considered. Terms can either be literals as in classical planning, or numeric terms of the form f(X){ >=,>, =} 0 where f(X) is a linear expressions. Each domain of the competition will be either a SNP or a LNP (LNP is more expressive than SNP). Planners supporting only SNP will be accepted and their performance will be reported when compared with respect to only SNP problems. A planner can also only support SNP. We are considering to have separate rankings.

In terms of the optimisation track, the optimisation functions will always be a minimisation. Problems will either be a minimisation of action costs, or a metric function that is a weighted sum with positive coefficients where each involved variable can only be increased by the actions.

Sample problems are here: Domain1, Problem1, Domain2, Problem2, Domain3, Problem3

Domains

The competition tasks can now be found in the following Github Repository.

Registration

We require that a planner is registered through a team. For this you need to send an e-mail with a subject containing “[Registration for Numeric Tracks]” to jea20@st-andrews.ac.uk. The email must contain:

names of participants, email contacts, track (agile/satisficing/optimal), at least one github account

Based on that, we will create a private repository under the ipc2023-numeric organization and add all participants as users with with write access and participants can both upload their planner there, and then commit to the repository as they wish until the “final planner submission” deadline.

All participants must subscribe to the Google Group. We will announce further details on the submission process there in due time.

To propose a domain for the competition, please contact the organizers (see below).

Planner submission

The competitors must submit the source code of their planners that will be run by the organizers on the actual competition domains/problems, unknown to the competitors until this time. This way no fine-tuning of the planners will be possible. We will host repositories of planners ourselves. The repositories will be hosted on Github under the ipc2023-numeric organisation, and they will be kept private until the end of the competition when we make them public, i.e., after the competition is concluded, we plan to make all planners, domains, and all related data accessible from one place. We require participants to provide self-contained code, with scripts for compilations and running, “compile” and “run”, respectively. The code should be compilable on a fresh installation of Ubuntu 20.04 LTS, and running has to require only three parameters as an input in the following order: the domain file, the problem file, and the name of the file where the last solution found will be saved. The solution format is the usual IPC solution format, which consists of a list of actions separated by a new line.

After Competition!

Participant Teams

• LNM-Plan (planner abstract)(frozen source)
  by Ryo Kuroiwa, Alexander Shleyfman and J. Christopher Beck
• OMTPlan (planner abstract)(frozen source)
  by Francesco Leofante

Results

The results were presented at the 33rd International Conference on Automated Planning and Scheduling on July 11 in Prague.

The presentation slides of this talk contain additional details. This is a summary:

• Optimal Track Winner SNP: LNM-Plan using NLM-CutPlan Orbit
• Optimal Track Winner LNP: LNM-Plan using NLM-CutPlan, NLM-CutPlan Orbit
• Optimal Track Winner SNP+LNP: LNM-Plan using NLM-CutPlan Orbit
• Satisficing Track Winner SNP: LNM-Plan using NLM-CutPlan SAT2
• Satisficing Track Winner LNP: LNM-Plan using NLM-CutPlan SAT
• Satisficing Track Winner SNP+LNP: LNM-Plan using NLM-CutPlan SAT
• Agile Track Winner SNP: LNM-Plan using NLM-CutPlan SAT2
• Agile Track Winner LNP: LNM-Plan using NLM-CutPlan SAT
• Agile Track Winner SNP+LNP: LNM-Plan using NLM-CutPlan SAT

Per-domain tables of solved instances are linked here for the Satisficing Track and for the Optimal Track. Note that when evaluating the tables, planners that were not able to process LNP problems have zeroes in the respective cells. For comparisons reasons, we used a well known greedy best first search algorithm with state of the arte heuristics from the literature (Scala et al. 2020) and $A^*$ with the numeric extension of the $h_{max}$ heuristic (Scala et al. 2017) for the satisficing and the optimal track respectively. These are implemented in the ENHSP planning system.

Organizers

• Joan Espasa Arxer (University of St Andrews)
• Enrico Scala (University of Brescia)

Contact us: ipc2023-numeric@googlegroups.com