近端策略优化算法 (PPO) 详解与 PyTorch 实现

这里使用了共享层来提取状态特征，然后分叉为 Actor 和 Critic。Softmax 确保输出是合法的概率分布。

2. 经验回放管理

class Memory:
    def __init__(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

    def clear(self):
        self.states = []
        self.actions = []
        self.logprobs = []
        self.rewards = []
        self.is_terminals = []

Memory 类负责存储一个 episode 内的所有经验数据，包括状态、动作、对数概率、奖励及终止标志，供后续批量更新使用。

3. PPO Agent 初始化

import torch.optim as optim
from torch.distributions import Categorical

class PPO:
    def __init__(self, state_dim, action_dim, lr=0.002, gamma=0.99, eps_clip=0.2, K_epochs=4):
        self.policy = ActorCritic(state_dim, action_dim).to(device)
        self.optimizer = optim.Adam(self.policy.parameters(), lr=lr)
        self.policy_old = ActorCritic(state_dim, action_dim).to(device)
        self.policy_old.load_state_dict(self.policy.state_dict())
        self.MseLoss = nn.MSELoss()
        self.gamma = gamma
        self.eps_clip = eps_clip
        self.K_epochs = K_epochs

注意 policy_old 是旧策略的副本，用于计算概率比率。eps_clip 控制更新幅度的阈值。

4. 动作选择

def select_action(self, state, memory):
    state = torch.FloatTensor(state).to(device)
    action_probs, _ = self.policy_old(state)
    dist = Categorical(action_probs)
    action = dist.sample()
    
    memory.states.append(state)
    memory.actions.append(action)
    memory.logprobs.append(dist.log_prob(action))
    return action.item()

这里从旧策略中采样动作，并记录对数概率，这是计算优势函数的基础。

5. 策略更新

def update(self, memory):
    old_states = torch.stack(memory.states).to(device).detach()
    old_actions = torch.stack(memory.actions).to(device).detach()
    old_logprobs = torch.stack(memory.logprobs).to(device).detach()
    
    # 计算折扣回报
    rewards = []
    discounted_reward = 0
    for reward, is_terminal in zip(reversed(memory.rewards), reversed(memory.is_terminals)):
        if is_terminal:
            discounted_reward = 0
        discounted_reward = reward + (self.gamma * discounted_reward)
        rewards.insert(0, discounted_reward)
    
    rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
    # 奖励归一化，加速收敛
    rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)

    # 多轮迭代更新
    for _ in range(self.K_epochs):
        action_probs, state_values = self.policy(old_states)
        dist = Categorical(action_probs)
        new_logprobs = dist.log_prob(old_actions)
        entropy = dist.entropy()
        
        ratios = torch.exp(new_logprobs - old_logprobs.detach())
        advantages = rewards - state_values.detach().squeeze()
        
        surr1 = ratios * advantages
        surr2 = torch.clamp(ratios, 1 - self.eps_clip, 1 + self.eps_clip) * advantages
        loss_actor = -torch.min(surr1, surr2).mean()
        
        loss_critic = self.MseLoss(state_values.squeeze(), rewards)
        loss = loss_actor + 0.5 * loss_critic - 0.01 * entropy.mean()
        
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
    
    self.policy_old.load_state_dict(self.policy.state_dict())

关键点在于奖励归一化，这能显著加快收敛速度。此外，detach() 的使用确保了梯度的正确传播，避免影响旧策略的计算图。

6. 主程序

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = gym.make("CartPole-v1")
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n

ppo = PPO(state_dim, action_dim, lr=0.002, gamma=0.99, eps_clip=0.2, K_epochs=4)
memory = Memory()

for episode in range(1, 1001):
    state = env.reset()
    total_reward = 0
    for t in range(300):
        action = ppo.select_action(state, memory)
        state, reward, done, _ = env.step(action)
        memory.rewards.append(reward)
        memory.is_terminals.append(done)
        total_reward += reward
        if done:
            break
    ppo.update(memory)
    memory.clear()
    print(f"Episode {episode}, Total Reward: {total_reward}")

实际运行时，建议根据具体任务调整超参数，比如学习率和折扣因子。这段代码主要用于理解算法逻辑，若投入生产环境还需针对特定场景进行调优。

算法对比

PPO 因其简单、稳定、高效的特点，已成为强化学习领域的主流算法；TRPO 更适合对稳定性要求极高的任务；A3C 则在并行加速方面表现优异。

总结

PPO 的关键是通过限制策略的变化范围（剪辑），让优化更加稳定，同时通过优势函数引导策略改进，充分利用采样数据。这种平衡使得它在实际工程中非常受欢迎。如果你正在寻找一种易于实现且效果可靠的强化学习算法，PPO 值得优先考虑。