近端策略优化算法 (PPO) 详解与 PyTorch 实现

这里 shared_layer 将状态映射到隐层表示，actor 使用 Softmax 确保输出是合法的概率分布，critic 则直接回归状态值。

2. 经验记忆类

我们需要存储一个 episode 内的交互数据，包括状态、动作、对数概率、奖励等。

class Memory:
    def __init__(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

    def clear(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

3. PPO Agent 初始化

初始化策略网络、优化器以及超参数。注意这里需要维护一份 policy_old 用于计算概率比率。

class PPO:
    def __init__(self, state_dim, action_dim, lr=0.002, gamma=0.99, eps_clip=0.2, K_epochs=4):
        self.policy = ActorCritic(state_dim, action_dim)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=lr)
        self.policy_old = ActorCritic(state_dim, action_dim)
        self.policy_old.load_state_dict(self.policy.state_dict())
        self.MseLoss = nn.MSELoss()
        self.gamma = gamma
        self.eps_clip = eps_clip
        self.K_epochs = K_epochs

4. 动作选择与更新

在环境中采样时，使用旧策略选择动作并记录对数概率。更新阶段则利用采样的数据进行多轮迭代优化。

    def select_action(self, state, memory):
        state = torch.FloatTensor(state).to(device)
        with torch.no_grad():
            action_probs, _ = self.policy_old(state)
            dist = Categorical(action_probs)
            action = dist.sample()
            memory.states.append(state)
            memory.actions.append(action)
            memory.logprobs.append(dist.log_prob(action))
        return action.item()

    def update(self, memory):
        old_states = torch.stack(memory.states).to(device).detach()
        old_actions = torch.stack(memory.actions).to(device).detach()
        old_logprobs = torch.stack(memory.logprobs).to(device).detach()

        # 计算折扣回报
        rewards = []
        discounted_reward = 0
        for reward, is_terminal in zip(reversed(memory.rewards), reversed(memory.is_terminals)):
            if is_terminal:
                discounted_reward = 0
            discounted_reward = reward + (self.gamma * discounted_reward)
            rewards.insert(0, discounted_reward)
        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)

        # 多轮策略更新
        for _ in range(self.K_epochs):
            action_probs, state_values = self.policy(old_states)
            dist = Categorical(action_probs)
            new_logprobs = dist.log_prob(old_actions)
            entropy = dist.entropy()

            ratios = torch.exp(new_logprobs - old_logprobs.detach())
            advantages = rewards - state_values.detach().squeeze()

            surr1 = ratios * advantages
            surr2 = torch.clamp(ratios, 1 - self.eps_clip, 1 + self.eps_clip) * advantages
            loss_actor = -torch.min(surr1, surr2).mean()

            loss_critic = self.MseLoss(state_values.squeeze(), rewards)
            loss = loss_actor + 0.5 * loss_critic - 0.01 * entropy.mean()

            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

        self.policy_old.load_state_dict(self.policy.state_dict())

这里有一个细节需要注意：奖励归一化。在实际运行中，奖励的尺度可能差异很大，标准化处理有助于加快收敛。此外，advantages 的计算依赖于 Critic 提供的状态值，因此 Critic 的准确性至关重要。

5. 主程序

最后整合环境配置与训练循环。

import gym
import torch.optim as optim

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = gym.make("CartPole-v1")
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n

ppo = PPO(state_dim, action_dim)
memory = Memory()

for episode in range(1, 1001):
    state = env.reset()
    total_reward = 0
    for t in range(300):
        action = ppo.select_action(state, memory)
        state, reward, done, _ = env.step(action)
        memory.rewards.append(reward)
        memory.is_terminals.append(done)
        total_reward += reward
        if done:
            break
    ppo.update(memory)
    memory.clear()
    print(f"Episode {episode}, Total Reward: {total_reward}")

算法对比与总结

PPO 之所以成为当前的主流算法，是因为它在 TRPO 的理论保障和简单实现之间找到了最佳平衡点。相比 TRPO，它不需要复杂的二阶优化；相比 A3C，它的同步更新机制更容易调试且稳定性更好。

总的来说，PPO 通过限制策略变化范围，配合优势函数引导改进，充分利用了采样数据。如果你正在寻找一个既稳定又高效的强化学习算法，PPO 通常是首选方案。