DevNews

NVIDIA NOOA: a typed answer can still be wrong

On this page
  1. What the class interface supplies
  2. Type validation and policy validation answer different questions
  3. A Python object is not an isolation boundary

NOOA makes an agent look more like ordinary Python. That can simplify review, while leaving two important questions: whether an answer is correct and what generated code can access.

Fictional policy example: boolean approved=true passes a shape check, but a 45-day-old order fails a policy allowing refunds only through day 30. Type validity does not prove business-rule correctness or confine generated code.
Fictional policy example: boolean approved=true passes a shape check, but a 45-day-old order fails a policy allowing refunds only through day 30. Type validity does not prove business-rule correctness or confine generated code. Chart : PeopleAreGeek. Data source.
View full-size image

What the class interface supplies

The NVIDIA project README describes typed fields as state, docstrings as prompts and methods with an ellipsis body as model-driven generation. Methods with ordinary bodies remain Python code. They are not automatically deterministic if their own implementation uses time, randomness or external services.

The framework supports typed output with retries and model selection through LiteLLM. Its current tracing instructions also matter: the viewer and dependencies must be running; otherwise tracing is silently disabled. The previous blanket claim that every execution leaves a trace was incomplete.

Type validation and policy validation answer different questions

Imagine an application asks for a result with a boolean field named approved. A response containing true is structurally plausible. If the actual rule forbids approval for an expired order, that same well-typed value can still be wrong.

Our cover separates the shape check from the rule check. A fictional order is 45 days old, while a fictional policy permits refunds only through day 30. The decision is false under that rule regardless of how confidently a model produces true. This example is about application correctness, not a refund policy recommended for a real business.

A deterministic policy function can centralize the rule and make it testable. It must also be the path the application actually uses before applying a decision. Merely exposing such a method to the model does not force generated code to call it or prevent another reachable action.

A Python object is not an isolation boundary

The project explicitly distinguishes its AST checks and module deny lists from operating-system containment. Generated code may execute with access to live objects and imports. A typed method signature does not stop unrelated filesystem or network access.

For an evaluation, use synthetic data in an isolated environment and confirm the trace viewer records the run. Compare the final business decision against explicit expected cases, including invalid inputs and unavailable dependencies. This tests something concrete: whether the class-based workflow is easier to understand while still enforcing the application’s rules. Successful installation alone cannot answer that question.

September 8: inspect current framework README; distinguish Python methods from guaranteed determinism, typed structure from semantic validity, and default tracing from an active trace viewer.