近端策略优化算法 PPO 详解与 PyTorch 实现

这里 shared_layer 将状态映射到隐层表示，actor 使用 Softmax 确保输出为合法的概率分布，critic 则预测当前状态的预期回报。

2. 经验存储类 (Memory)

PPO 需要批量更新，因此需要一个缓冲区来存储 episode 的数据。

class Memory:
    def __init__(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

    def clear(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

3. PPO Agent 初始化

初始化策略网络、优化器以及超参数。

import torch.optim as optim

class PPO:
    def __init__(self, state_dim, action_dim, lr=0.002, gamma=0.99, eps_clip=0.2, K_epochs=4):
        self.policy = ActorCritic(state_dim, action_dim).to(device)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=lr)
        self.policy_old = ActorCritic(state_dim, action_dim).to(device)
        self.policy_old.load_state_dict(self.policy.state_dict())
        self.MseLoss = nn.MSELoss()
        self.gamma = gamma
        self.eps_clip = eps_clip
        self.K_epochs = K_epochs

注意 policy_old 用于计算概率比率，每次更新后需要同步最新参数。

4. 动作选择

根据当前策略采样动作，并记录对数概率。

def select_action(self, state, memory):
    state = torch.FloatTensor(state).to(device)
    with torch.no_grad():
        action_probs, _ = self.policy_old(state)
        dist = Categorical(action_probs)
        action = dist.sample()
        memory.states.append(state)
        memory.actions.append(action)
        memory.logprobs.append(dist.log_prob(action))
    return action.item()

5. 策略更新

这是 PPO 的核心部分，包括奖励归一化、优势计算和裁剪损失更新。

def update(self, memory):
    old_states = torch.stack(memory.states).to(device).detach()
    old_actions = torch.stack(memory.actions).to(device).detach()
    old_logprobs = torch.stack(memory.logprobs).to(device).detach()

    # 计算折扣回报
    rewards = []
    discounted_reward = 0
    for reward, is_terminal in zip(reversed(memory.rewards), reversed(memory.is_terminals)):
        if is_terminal:
            discounted_reward = 0
        discounted_reward = reward + (self.gamma * discounted_reward)
        rewards.insert(0, discounted_reward)
    
    rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
    # 奖励归一化，加速收敛
    rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)

    # 多轮 Epoch 更新
    for _ in range(self.K_epochs):
        action_probs, state_values = self.policy(old_states)
        dist = Categorical(action_probs)
        new_logprobs = dist.log_prob(old_actions)
        entropy = dist.entropy()

        # 计算概率比率
        ratios = torch.exp(new_logprobs - old_logprobs.detach())
        advantages = rewards - state_values.detach().squeeze()

        # 裁剪损失
        surr1 = ratios * advantages
        surr2 = torch.clamp(ratios, 1 - self.eps_clip, 1 + self.eps_clip) * advantages
        loss_actor = -torch.min(surr1, surr2).mean()

        # 值函数损失
        loss_critic = self.MseLoss(state_values.squeeze(), rewards)

        # 总损失
        loss = loss_actor + 0.5 * loss_critic - 0.01 * entropy.mean()

        # 反向传播
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    # 同步旧策略
    self.policy_old.load_state_dict(self.policy.state_dict())

这里有一个细节需要注意：advantages 的计算依赖于 state_values，如果 Critic 不准，Advantage 也会偏差较大，所以 Critic 的损失权重通常也需要调整。

6. 主程序流程

完整的训练循环如下：

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
import gym
env = gym.make("CartPole-v1")

state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n

ppo = PPO(state_dim, action_dim, lr=0.002, gamma=0.99, eps_clip=0.2, K_epochs=4)
memory = Memory()

for episode in range(1, 1001):
    state = env.reset()
    total_reward = 0
    for t in range(300):
        action = ppo.select_action(state, memory)
        state, reward, done, _ = env.step(action)
        memory.rewards.append(reward)
        memory.is_terminals.append(done)
        total_reward += reward
        if done:
            break
    ppo.update(memory)
    memory.clear()
    print(f"Episode {episode}, Total Reward: {total_reward}")

算法对比总结

PPO 作为 TRPO 的改进版，用简单的裁剪机制替代了复杂的二次优化，显著降低了实现难度，同时保持了良好的稳定性和效率。对于大多数强化学习任务，PPO 都是首选的默认算法。

注意事项

1. 环境适配：代码示例基于 Gym 的 CartPole 环境，实际项目中可能需要针对特定环境调整网络结构或超参数。
2. 超参数调优：学习率、折扣因子和裁剪阈值对训练效果影响较大，建议根据具体任务进行网格搜索。
3. 奖励归一化：在长序列任务中，奖励归一化能有效缓解梯度消失或爆炸问题。
4. 设备管理：确保 device 变量正确配置以利用 GPU 加速训练。