agent-readings

Planning and Skill Acquisition

LLM+P

LLM+P gets reliable plans by making the LLM translate natural language into PDDL and letting a classical planner do the search.

7Claims
7Evidence

Why It Matters

The paper is a clean neuro-symbolic systems pattern for agents: separate semantic parsing from plan search, keep a domain model explicit, and route correctness through a verifier/planner instead of model plausibility.

Problem

  • LLMs produce plausible natural-language plans that often violate action preconditions or world constraints.
  • Classical planners can produce correct or optimal plans, but require structured domain and problem encodings.
  • Robot users naturally specify goals in language, not PDDL.
  • Long-horizon planning domains expose LLM weaknesses that are hidden by short or underspecified tasks.

Key gap: The paper asks whether an LLM can be used as a translation layer into PDDL so a classical planner can provide the actual planning competence.

Method

Use the LLM for translation into and out of the formal planning language, but use a classical planner for the search and correctness-critical plan generation.

  • Provide a domain PDDL file for the robot planning domain.
  • Prompt GPT-4 with the new natural-language planning problem plus one demonstration pair of natural language and PDDL.
  • Ask the model to produce only the problem PDDL.
  • Feed the domain and generated problem file to Fast Downward.
  • If a plan is found, translate the PDDL action sequence back to natural language or robot action execution.
  • Compare against direct LLM planning, in-context direct planning, and Tree-of-Thought-style direct planning.

Claims

C1 · paper-supported

LLM+P delegates planning search to a classical planner after an LLM translates a natural-language task into a PDDL problem file.

Scope: Abstract, Figure 1, and Section III-C.

  • E-pipeline: The pipeline asks the LLM to produce a problem PDDL file, feeds it with a human-provided domain PDDL file to a classical planner, then translates the planner output back to natural language.

C2 · paper-supported

A simple in-context example is crucial because GPT-4 often emits incorrect PDDL without a demonstration pair.

Scope: Sections III-A, III-B, and V-C.

  • E-context: Without context, GPT-4's PDDL example has made-up predicates and missing initial facts. With a simple natural-language/PDDL example, the sample problem becomes directly solvable. The results section says LLM+P without context fails to produce correct problem PDDL files.

C3 · paper-supported

The benchmark covers seven robot-planning domains with 20 generated tasks per domain.

Scope: Section V-A.

  • E-benchmark: The benchmark contains seven domains borrowed from past International Planning Competitions: Blocksworld, Barman, Floortile, Grippers, Storage, Termes, and Tyreworld, with 20 automatically generated tasks per domain.

C4 · paper-supported

Direct LLM planning mostly produces infeasible plans because it fails to track preconditions and spatial constraints.

Scope: Section V-C.

  • E-llm-as-p-fails: The paper reports that LLM-as-P returns natural-language plans for every problem but most are infeasible. It cites failures tracking Blocksworld on/clear predicates, Barman glass-cleaning requirements, spatial constraints in Floortile, Termes, and Storage, and ToT timeouts from many LLM calls.

C5 · paper-supported

LLM+P with context substantially improves success across most domains, including 90% Blocksworld, 95% optimal / 100% satisficing Grippers, and 85% Storage.

Scope: Section V-C, Table I.

  • E-table-results: LLM+P success rates are Barman 20% optimal / 100% satisficing, Blocksworld 90%, Floortile 0%, Grippers 95% optimal / 100% satisficing, Storage 85%, Termes 20%, and Tyreworld 10% optimal / 90% satisficing. Direct LLM and ToT baselines are much lower in the same table.

C6 · paper-supported

In the home tidy-up robot demonstration, LLM+P finds a lower-cost plan than direct LLM planning by exploiting shared path structure.

Scope: Section V-D.

  • E-robot-demo: In the tidy-up task, LLM+P finds an optimal plan with total cost 22 by taking the bottle partway, disposing of the soup can, re-grasping the bottle, and then going to the pantry. LLM-as-P takes the bottle to the pantry first and returns for the soup can, for cost 31.

C7 · paper-supported

The framework depends on external trigger detection, human-provided domain models, and accurate PDDL translation by the LLM.

Scope: Sections III-C and VI.

  • E-assumptions: The method assumes a robot knows when to trigger LLM+P, a domain PDDL file exists, and an example problem/PDDL pair is provided. The conclusion names auto-detecting when to apply LLM+P and reducing dependence on human-provided information as future work.

Agent Infrastructure Takeaways

  • Use LLMs as translators when a sound solver exists. Planning correctness should come from the planner, not from plausible prose.
  • The action model is infrastructure. Domain PDDL is a contract over capabilities, preconditions, effects, and costs.
  • Structured intermediate representations make agent failures inspectable: bad plan versus bad translation is a different bug.
  • One-shot examples can be enough to steer translation, but production systems need validation and repair loops for generated PDDL.
  • A planner-backed agent needs a router that decides when a task belongs to the planner rather than the general LLM loop.
  • The same pattern generalizes beyond planning: calculators, compilers, theorem provers, and simulators should own domains where they are sound.

Caveats

  • The domain model is manually supplied; LLM+P does not learn or verify the domain PDDL.
  • The planner can only be correct with respect to the given domain and problem encoding.
  • Generated PDDL can omit facts or use wrong predicates, causing planner failure or an invalid plan request.
  • Floortile remains unsolved in the reported table.
  • The real-robot demonstration is a single tidy-up scenario.
  • The paper does not solve when to invoke the planning pipeline automatically.