强化学习：演员评论家 Actor-Critic 算法详解与实现

算法主逻辑

这里实现了核心的 update 方法。注意我们在计算 Loss 时，Actor 的 Loss 依赖于 Critic 输出的 TD 误差，而 Critic 的 Loss 则是预测值与目标的均方误差。

class ActorCritic:
    def __init__(self, n_states, n_hiddens, n_actions, actor_lr, critic_lr, gamma):
        self.gamma = gamma
        self.actor = PolicyNet(n_states, n_hiddens, n_actions)
        self.critic = ValueNet(n_states, n_hiddens)
        
        self.actor_optimizer = torch.optim.Adam(self.actor.parameters(), lr=actor_lr)
        self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=critic_lr)

    def take_action(self, state):
        """根据当前策略采样动作"""
        state = torch.tensor(state[np.newaxis, :], dtype=torch.float)
        probs = self.actor(state)
        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample().item()
        return action

    def update(self, transition_dict):
        """批量更新模型参数"""
        states = torch.tensor(transition_dict['states'], dtype=torch.float)
        actions = torch.tensor(transition_dict['actions']).view(-1, 1)
        rewards = torch.tensor(transition_dict['rewards'], dtype=torch.float).view(-1, 1)
        next_states = torch.tensor(transition_dict['next_states'], dtype=torch.float)
        dones = torch.tensor(transition_dict['dones'], dtype=torch.float).view(-1, 1)

        # Critic 计算 TD 误差
        td_value = self.critic(states)
        td_target = rewards + self.gamma * self.critic(next_states) * (1 - dones)
        td_delta = td_target - td_value

        # Actor 损失：负对数似然 * TD 误差
        log_probs = torch.log(self.actor(states).gather(1, actions))
        actor_loss = torch.mean(-log_probs * td_delta.detach())

        # Critic 损失：均方误差
        critic_loss = torch.mean(F.mse_loss(self.critic(states), td_target.detach()))

        # 反向传播与更新
        self.actor_optimizer.zero_grad()
        self.critic_optimizer.zero_grad()
        actor_loss.backward()
        critic_loss.backward()
        self.actor_optimizer.step()
        self.critic_optimizer.step()

训练流程

最后，我们在 CartPole 环境中运行训练循环。这里需要注意数据收集的方式，我们通常在一个 episode 结束后再统一更新一次参数，这样能利用整个轨迹的信息。

import gym
import matplotlib.pyplot as plt

env_name = 'CartPole-v1'
num_episodes = 100
gamma = 0.9
actor_lr = 1e-3
critic_lr = 1e-2
n_hiddens = 16

env = gym.make(env_name)
n_states = env.observation_space.shape[0]
n_actions = env.action_space.n

agent = ActorCritic(
    n_states=n_states,
    n_hiddens=n_hiddens,
    n_actions=n_actions,
    actor_lr=actor_lr,
    critic_lr=critic_lr,
    gamma=gamma
)

return_list = []

for i in range(num_episodes):
    state = env.reset()[0]
    done = False
    episode_return = 0
    transition_dict = {
        'states': [], 'actions': [], 'next_states': [],
        'rewards': [], 'dones': []
    }

    while not done:
        action = agent.take_action(state)
        next_state, reward, done, _, _ = env.step(action)
        
        transition_dict['states'].append(state)
        transition_dict['actions'].append(action)
        transition_dict['next_states'].append(next_state)
        transition_dict['rewards'].append(reward)
        transition_dict['dones'].append(done)
        
        state = next_state
        episode_return += reward

    return_list.append(episode_return)
    agent.update(transition_dict)
    print(f'iter:{i}, return:{np.mean(return_list[-10:])}')

plt.plot(return_list)
plt.title('Training Return')
plt.show()

关键点总结

在实际应用中，有几个细节值得注意：

1. Critic 的稳定性：Critic 的误差直接决定了 Actor 的梯度方向，如果 Critic 学得太慢或太偏，Actor 可能会学歪。
2. 熵正则化：为了防止策略过早收敛到局部最优，可以在 Actor 的损失中加入熵项鼓励探索。
3. 超参数敏感：Actor 和 Critic 的学习率通常需要分开调节，有时 Critic 需要更快的学习速度。
4. 扩展性：基础的 Actor-Critic 可以进一步扩展为 A3C（异步多进程）或 PPO（近端策略优化），以解决更复杂的任务。

总结

Actor-Critic 算法通过结合策略梯度的灵活性和值函数的低方差特性，成为强化学习领域的重要基石。从数学原理到 PyTorch 落地，理解其核心在于把握 Actor 与 Critic 之间的梯度传递关系。希望这篇教程能帮助你更好地掌握这一算法，并在实际项目中灵活运用。