Lecture 10.1: Decision Theory and Utility

How agents make rational decisions under uncertainty using Utility Theory, Expected Utility, and Decision Networks.

Up until now, we've looked at logical agents that pursue absolute goals (like winning a game or proving a theorem). But in the real world, agents face uncertainty and conflicting goals. A goal-based agent only knows the binary difference between a "good" state and a "bad" state.

A Decision-Theoretic Agent, however, has a continuous measure of outcome quality. It combines probability theory with utility theory to make rational decisions based on what it believes (probabilities) and what it wants (utilities).

Utility Theory

An agent's preferences are captured by a Utility Function, which maps states of the world to a real number: $U: S \rightarrow \mathbb{R}$ The larger the utility value, the more the agent prefers that state.

Marginal Utility: The additional utility an agent gains from acquiring one more unit of a good.
- Diminishing Marginal Utility: Winning $1,000,000 has a massive utility change for a poor person, but is practically unnoticeable to a billionaire. The utility curve of money is typically logarithmic, not linear.
Framing Effect: In human decision-making, preferences change based on how options are framed (e.g., presenting a medical treatment as having a 90% survival rate vs. a 10% mortality rate). Even when mathematical expected utilities are identical, framing alters decisions.

Expected Utility

Since environments are uncertain and actions can fail (e.g., a robot slipping on ice), an agent must consider the Transition Function $T(s, a, s')$ , which denotes the probability of reaching state $s'$ given that the agent is in state $s$ and takes action $a$ .

The Expected Utility ( $EU$ ) of taking action $a$ in state $s$ is the weighted average of the utilities of all possible resulting states: $EU(a \mid s) = \sum_{s'} T(s, a, s') U(s')$

The Maximum Expected Utility (MEU) Principle

A rational agent's goal is to choose an action $a^*$ that maximizes its expected utility: $a^* = \arg\max_{a \in A} EU(a \mid s)$ This strategy is known as the Principle of Maximum Expected Utility.

The Value of Information (VPI)

In many scenarios, an agent must decide whether to make a decision immediately or first gather more information (which may cost time or money).

The Value of Perfect Information (VPI) measures the expected increase in utility that an agent obtains by learning a new piece of evidence $E$ before acting: $VPI(E) = EU(a^*_E \mid E) - EU(a^* \mid E)$ where:

$a^*_E$ is the optimal action chosen after learning evidence $E$ .
$a^*$ is the optimal action chosen without knowing evidence $E$ .

[!NOTE] VPI Example: The Doctor's Dilemma A doctor is deciding which drug to prescribe to a patient but is uncertain about the exact disease. She can run a diagnostic test.

If the test result—regardless of whether it comes back positive or negative—would still lead the doctor to prescribe the exact same drug, then the VPI of that test is 0.

If a positive result changes the prescription to Drug A, and a negative result changes it to Drug B, the test has a high VPI because it alters the doctor's action to maximize patient utility.

Decision Networks (Influence Diagrams)

Decision networks (also called influence diagrams) extend Bayesian networks by incorporating actions (decisions) and utilities. They contain three types of nodes:

Chance Nodes (Ovals): Represent random variables (uncertainty) in the environment.
Decision Nodes (Rectangles): Represent choices under the agent's direct control.
Utility Nodes (Diamonds/Octagons): Represent the agent's utility function mapping outcomes to payoffs.

Example: The Oil Wild-Catter Problem

An oil company must decide whether to drill for oil. Before making the decision, they can choose to run a seismic sound test to detect geological structures indicating oil presence.

Conditional Arcs: The arrow from Oil State to Sound Signal shows that the probability of hearing a certain sound depends on whether there is oil.
Informational Arcs: The arrow from Sound Signal to Drill Decision shows that the agent knows the test results before deciding whether to drill.
Functional Arcs: The arrows pointing to Payoff show that the utility depends on both the decision to drill and the actual oil state.