添加 A2C/QAC 算法实现及训练结果
- 新增 RL_Algothrithms 模块，包含 A2C、QAC 智能体 - 添加 SAC 章节笔记和 C10 笔记 - 上传训练结果图片 - 完善 README 与 .gitignore
This commit is contained in:
@@ -45,3 +45,9 @@ Thumbs.db
|
|||||||
# Byte-compiled / optimized / DLL files
|
# Byte-compiled / optimized / DLL files
|
||||||
*.pyc
|
*.pyc
|
||||||
*.pyd
|
*.pyd
|
||||||
|
|
||||||
|
# Images (uncomment/add specific images to track them)
|
||||||
|
# *.png
|
||||||
|
# *.jpg
|
||||||
|
# *.jpeg
|
||||||
|
# *.gif
|
||||||
|
|||||||
@@ -0,0 +1,610 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "ef80017f",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"# C10 演员评论家方法\n",
|
||||||
|
"\n",
|
||||||
|
"## 最基础的AC算法：QAC\n",
|
||||||
|
"\n",
|
||||||
|
"Actor网络：策略梯度上升\n",
|
||||||
|
"\n",
|
||||||
|
"直接更新策略参数，以采取更优的动作。\n",
|
||||||
|
"\n",
|
||||||
|
"我们希望最优化一个标量指标$J(\\theta)$，他的梯度包含一个期望值：\n",
|
||||||
|
"\n",
|
||||||
|
"$$\\nabla_{\\theta}J(\\theta)=\\mathbb{E}_{S\\sim\\eta,A\\sim\\pi}[\\nabla_{\\theta}\\ln \\pi(A|S,\\theta_{t})q_{\\pi}(S,A)]$$\n",
|
||||||
|
"\n",
|
||||||
|
"由于实际工程中很难求出这个数学期望的解析解，因此采用随机的采样方法进行梯度上升的近似更新：\n",
|
||||||
|
"\n",
|
||||||
|
"$$\\theta_{t+1}=\\theta_{t}+\\alpha_{\\theta}\\nabla_{\\theta}\\ln \\pi(a_{t}|s_{t},\\theta_{t})q(s_{t},a_{t},w_{t})$$\n",
|
||||||
|
"\n",
|
||||||
|
"如果某个动作在状态 $s_t$ 下得到了很高的 $q$ 值，梯度更新就会使得网络参数 $\\theta$ 发生改变，从而提高未来在相同状态下采取动作 $a_t$ 的概率 $\\pi$。这就是 Actor 学习的过程。\n",
|
||||||
|
"\n",
|
||||||
|
"Critic网络：价值的时序差分更新\n",
|
||||||
|
"\n",
|
||||||
|
"在 Actor 的更新公式中，我们需要知道动作价值 $q_t$ 。如果采用蒙特卡洛方法（走完一整个回合再算），那就是 REINFORCE 算法 。但为了实现更高效的单步在线学习，我们使用时序差分（TD）学习来估计这个值，这就是 Actor-Critic 的精髓。\n",
|
||||||
|
"\n",
|
||||||
|
"Critic 的任务是通过评估动作的价值来“批评” Actor 。在最基础的 QAC 算法中，它使用类似 Sarsa 的方式更新自己的参数 $w$：\n",
|
||||||
|
"\n",
|
||||||
|
"$$w_{t+1}=w_{t}+\\alpha_{w}[r_{t+1}+\\gamma q(s_{t+1},a_{t+1},w_{t})-q(s_{t},a_{t},w_{t})]\\nabla_{w}q(s_{t},a_{t},w_{t})$$。\n",
|
||||||
|
"\n",
|
||||||
|
"公式方括号内的部分 $[r_{t+1}+\\gamma q(s_{t+1},a_{t+1},w_{t})-q(s_{t},a_{t},w_{t})]$ 就是著名的 TD 误差。它衡量了“当前预测的 $q$ 值”与“实际得到的奖励加上下一步预测的 $q$ 值”之间的差距。\n",
|
||||||
|
"\n",
|
||||||
|
"算法流程：\n",
|
||||||
|
"\n",
|
||||||
|
"1. 策略执行（Actor 正向计算）： 系统当前处于状态 $s_t$，Actor 依据当前的概率分布 $\\pi(a|s_{t},\\theta_{t})$ 采样出一个动作$a_t$并执行，获得真实物理系统的反馈奖励 $r_{t+1}$ 和下一个状态 $s_{t+1}$ 。\n",
|
||||||
|
"2. 价值评估（Critic 反向更新）： 系统依据策略再预演一步动作 $a_{t+1}$，以此计算 TD 误差，并使用梯度下降法更新打分器参数 $w_{t+1}$。\n",
|
||||||
|
"3. 策略优化（Actor 反向更新）： 拿着刚刚打出的分数 $q(s_{t},a_{t},w_{t})$，沿着梯度上升的方向更新策略参数 $\\theta_{t+1}$。"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 8,
|
||||||
|
"id": "c2f49de7",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"PyTorch 版本: 2.5.1+cu121\n",
|
||||||
|
"GPU 是否可用: True\n",
|
||||||
|
"当前显卡: NVIDIA GeForce RTX 5060 Ti\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"import torch\n",
|
||||||
|
"\n",
|
||||||
|
"# 打印 PyTorch 版本\n",
|
||||||
|
"print(f\"PyTorch 版本: {torch.__version__}\")\n",
|
||||||
|
"# 检查 GPU 是否可用 (如果输出 True，说明大功告成！)\n",
|
||||||
|
"print(f\"GPU 是否可用: {torch.cuda.is_available()}\")\n",
|
||||||
|
"\n",
|
||||||
|
"if torch.cuda.is_available():\n",
|
||||||
|
" # 打印当前使用的显卡型号\n",
|
||||||
|
" print(f\"当前显卡: {torch.cuda.get_device_name(0)}\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 9,
|
||||||
|
"id": "79226b49",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import torch\n",
|
||||||
|
"import torch.nn as nn\n",
|
||||||
|
"import torch.optim as optim\n",
|
||||||
|
"import torch.nn.functional as F\n",
|
||||||
|
"import torch.distributions as distributions\n",
|
||||||
|
"\n",
|
||||||
|
"# 定义超参数\n",
|
||||||
|
"# 学习率\n",
|
||||||
|
"ACTOR_LR = 0.001\n",
|
||||||
|
"CRITIC_LR = 0.002\n",
|
||||||
|
"# 折扣因子\n",
|
||||||
|
"GAMMA = 0.99\n",
|
||||||
|
"# 训练迭代次数\n",
|
||||||
|
"NUM_EPISODES = 1000\n",
|
||||||
|
"\n",
|
||||||
|
"# 1. 定义 Actor 网络 (演员：输出策略)\n",
|
||||||
|
"class Actor(nn.Module):\n",
|
||||||
|
" def __init__(self, state_dim, action_dim):\n",
|
||||||
|
" super(Actor, self).__init__()\n",
|
||||||
|
" # 定义两层全连接网络提取状态特征\n",
|
||||||
|
" self.fc1 = nn.Linear(state_dim, 128)\n",
|
||||||
|
" self.fc2 = nn.Linear(128, action_dim)\n",
|
||||||
|
" \n",
|
||||||
|
" def forward(self, state):\n",
|
||||||
|
" # 使用 ReLU 激活函数\n",
|
||||||
|
" x = F.relu(self.fc1(state))\n",
|
||||||
|
" # 使用 Softmax 输出离散动作的概率分布\n",
|
||||||
|
" action_probs = F.softmax(self.fc2(x), dim=-1)\n",
|
||||||
|
" return action_probs\n",
|
||||||
|
"\n",
|
||||||
|
"# 2. 定义 Critic 网络 (评论家：输出动作价值 Q 值)\n",
|
||||||
|
"class Critic(nn.Module):\n",
|
||||||
|
" def __init__(self, state_dim, action_dim):\n",
|
||||||
|
" super(Critic, self).__init__()\n",
|
||||||
|
" # 定义全连接网络，输入是状态，输出是各个动作的 Q 值\n",
|
||||||
|
" self.fc1 = nn.Linear(state_dim, 128)\n",
|
||||||
|
" self.fc2 = nn.Linear(128, action_dim)\n",
|
||||||
|
" \n",
|
||||||
|
" def forward(self, state):\n",
|
||||||
|
" x = F.relu(self.fc1(state))\n",
|
||||||
|
" q_values = self.fc2(x)\n",
|
||||||
|
" return q_values\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"# 3. 核心训练逻辑 (对应 Algorithm 10.1)\n",
|
||||||
|
"def train_step(actor, critic, actor_optimizer, critic_optimizer, \n",
|
||||||
|
" state, action, reward, next_state, next_action, done):\n",
|
||||||
|
" \"\"\"\n",
|
||||||
|
" 执行一步 QAC 算法的参数更新\n",
|
||||||
|
" \"\"\"\n",
|
||||||
|
" # 转换数据格式为 Tensor\n",
|
||||||
|
" state = torch.FloatTensor(state)\n",
|
||||||
|
" next_state = torch.FloatTensor(next_state)\n",
|
||||||
|
" reward = torch.FloatTensor([reward])\n",
|
||||||
|
" \n",
|
||||||
|
" # -----------------------------------------\n",
|
||||||
|
" # Critic 更新 (价值更新)\n",
|
||||||
|
" # -----------------------------------------\n",
|
||||||
|
" # 计算当前状态动作的 Q(s_t, a_t, w_t)\n",
|
||||||
|
" q_values = critic(state) # 选取某个state对应的所有动作的Q值\n",
|
||||||
|
" current_q = q_values[action] # 选取当前动作对应的Q值\n",
|
||||||
|
" \n",
|
||||||
|
" # 计算下一状态动作的 Q(s_{t+1}, a_{t+1}, w_t)\n",
|
||||||
|
" # 使用 .detach() 截断梯度，因为目标值不需要传递梯度回网络\n",
|
||||||
|
" next_q_values = critic(next_state).detach()\n",
|
||||||
|
" next_q = next_q_values[next_action]\n",
|
||||||
|
" \n",
|
||||||
|
" # 计算 TD 目标：r_{t+1} + gamma * Q(s_{t+1}, a_{t+1}) (如果是终止状态则没有下一步的Q)\n",
|
||||||
|
" td_target = reward + GAMMA * next_q * (1 - int(done))\n",
|
||||||
|
" \n",
|
||||||
|
" # 计算 TD 误差并更新 Critic 参数\n",
|
||||||
|
" # 对应公式: w_{t+1} = w_t + alpha_w * TD_Error * grad(Q)\n",
|
||||||
|
" critic_loss = F.mse_loss(current_q, td_target)\n",
|
||||||
|
" \n",
|
||||||
|
" critic_optimizer.zero_grad()\n",
|
||||||
|
" critic_loss.backward()\n",
|
||||||
|
" critic_optimizer.step()\n",
|
||||||
|
" \n",
|
||||||
|
" # -----------------------------------------\n",
|
||||||
|
" # Actor 更新 (策略更新)\n",
|
||||||
|
" # -----------------------------------------\n",
|
||||||
|
" # 获取当前状态下所有动作的概率分布\n",
|
||||||
|
" action_probs = actor(state)\n",
|
||||||
|
" # 构建概率分布对象，方便计算对数概率\n",
|
||||||
|
" dist = distributions.Categorical(action_probs)\n",
|
||||||
|
" \n",
|
||||||
|
" # 计算 ln(pi(a_t | s_t, theta_t))\n",
|
||||||
|
" log_prob = dist.log_prob(torch.tensor(action))\n",
|
||||||
|
" \n",
|
||||||
|
" # Actor 梯度上升目标：ln(pi) * Q(s, a)\n",
|
||||||
|
" # 在 PyTorch 中优化器默认执行梯度下降，所以加个负号变成最小化损失\n",
|
||||||
|
" # 注意这里使用的是刚刚更新前算出的 current_q，为了阻断梯度传到 Critic，使用 .detach()\n",
|
||||||
|
" actor_loss = -log_prob * current_q.detach()\n",
|
||||||
|
" \n",
|
||||||
|
" actor_optimizer.zero_grad()\n",
|
||||||
|
" actor_loss.backward()\n",
|
||||||
|
" actor_optimizer.step()\n",
|
||||||
|
"\n",
|
||||||
|
" return actor_loss.item(), critic_loss.item()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "81a39722",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"\n",
|
||||||
|
"## 控制任务\n",
|
||||||
|
"\n",
|
||||||
|
"本处使用OpenAI的Gymnasium环境中的CartPole-v1作为测试环境。这个环境的目标是通过控制一个小车来保持杆子竖直。状态空间是连续的，包含了小车的位置、速度以及杆子的角度和角速度；动作空间是离散的，只有两个动作：向左或向右移动小车。\n",
|
||||||
|
"\n",
|
||||||
|
"状态空间: 一个四维的连续向量\n",
|
||||||
|
"- $s_0$: 小车的位置\n",
|
||||||
|
"- $s_1$: 小车的速度\n",
|
||||||
|
"- $s_2$: 杆子的角度\n",
|
||||||
|
"- $s_3$: 杆子的角速度\n",
|
||||||
|
"\n",
|
||||||
|
"动作空间: 两个离散动作\n",
|
||||||
|
"- $a_0$: 向左移动小车\n",
|
||||||
|
"- $a_1$: 向右移动小车\n",
|
||||||
|
"\n",
|
||||||
|
"奖励函数: 每个时间步获得的奖励为 +1，直到杆子倒下或小车移出边界。"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 10,
|
||||||
|
"id": "cec5bd6b",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"使用设备: cpu\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "stderr",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"/tmp/ipykernel_77052/2107052713.py:73: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
|
||||||
|
" critic_loss = F.mse_loss(current_q, td_target)\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"Episode 100/1000 completed. 近100回合平均reward: 15.25\n",
|
||||||
|
"Episode 200/1000 completed. 近100回合平均reward: 33.00\n",
|
||||||
|
"Episode 300/1000 completed. 近100回合平均reward: 53.27\n",
|
||||||
|
"Episode 400/1000 completed. 近100回合平均reward: 34.57\n",
|
||||||
|
"Episode 500/1000 completed. 近100回合平均reward: 59.50\n",
|
||||||
|
"Episode 600/1000 completed. 近100回合平均reward: 51.37\n",
|
||||||
|
"Episode 700/1000 completed. 近100回合平均reward: 77.24\n",
|
||||||
|
"Episode 800/1000 completed. 近100回合平均reward: 77.66\n",
|
||||||
|
"Episode 900/1000 completed. 近100回合平均reward: 78.07\n",
|
||||||
|
"Episode 1000/1000 completed. 近100回合平均reward: 51.60\n",
|
||||||
|
"\n",
|
||||||
|
"=== QAC 训练统计 ===\n",
|
||||||
|
"总回合数: 1000\n",
|
||||||
|
"平均Reward: 53.15\n",
|
||||||
|
"最高Reward: 336.00\n",
|
||||||
|
"最后100回合平均Reward: 51.60\n",
|
||||||
|
"图片已保存到: qac_training_results.png\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# 设置设备 - 使用 CPU\n",
|
||||||
|
"device = torch.device(\"cpu\")\n",
|
||||||
|
"print(f\"使用设备: {device}\")\n",
|
||||||
|
"\n",
|
||||||
|
"import matplotlib.pyplot as plt\n",
|
||||||
|
"import numpy as np\n",
|
||||||
|
"import gymnasium as gym\n",
|
||||||
|
"\n",
|
||||||
|
"# 训练的主循环\n",
|
||||||
|
"def main():\n",
|
||||||
|
" # 初始化环境\n",
|
||||||
|
" env = gym.make('CartPole-v1')\n",
|
||||||
|
" state_dim = env.observation_space.shape[0] # type: ignore\n",
|
||||||
|
" action_dim = int(env.action_space.n) # type: ignore\n",
|
||||||
|
"\n",
|
||||||
|
" # 实例化 Actor 和 Critic 网络\n",
|
||||||
|
" actor = Actor(state_dim, action_dim).to(device)\n",
|
||||||
|
" critic = Critic(state_dim, action_dim).to(device)\n",
|
||||||
|
" actor_optimizer = optim.Adam(actor.parameters(), lr=ACTOR_LR)\n",
|
||||||
|
" critic_optimizer = optim.Adam(critic.parameters(), lr=CRITIC_LR)\n",
|
||||||
|
" \n",
|
||||||
|
" # 记录训练数据\n",
|
||||||
|
" episode_rewards = []\n",
|
||||||
|
" actor_losses = []\n",
|
||||||
|
" critic_losses = []\n",
|
||||||
|
"\n",
|
||||||
|
" for episode in range(NUM_EPISODES):\n",
|
||||||
|
" state, _ = env.reset()\n",
|
||||||
|
" episode_reward = 0\n",
|
||||||
|
" episode_actor_loss = 0\n",
|
||||||
|
" episode_critic_loss = 0\n",
|
||||||
|
" steps = 0\n",
|
||||||
|
"\n",
|
||||||
|
" # 预先生成一个初始动作 a_0\n",
|
||||||
|
" state_tensor = torch.FloatTensor(state).to(device)\n",
|
||||||
|
" action_probs = actor(state_tensor)\n",
|
||||||
|
" action = distributions.Categorical(action_probs).sample().item()\n",
|
||||||
|
"\n",
|
||||||
|
" while True:\n",
|
||||||
|
" # 与环境交互，获取当前状态\n",
|
||||||
|
" next_state, reward, terminated, truncated, _ = env.step(action)\n",
|
||||||
|
" done = terminated or truncated\n",
|
||||||
|
"\n",
|
||||||
|
" # 预先生成下一个动作 a_{t+1}，供 Critic 更新使用\n",
|
||||||
|
" next_state_tensor = torch.FloatTensor(next_state).to(device)\n",
|
||||||
|
" next_action_probs = actor(next_state_tensor)\n",
|
||||||
|
" next_action = distributions.Categorical(next_action_probs).sample().item()\n",
|
||||||
|
"\n",
|
||||||
|
" # 执行单步的学习\n",
|
||||||
|
" actor_loss, critic_loss = train_step(actor, critic, actor_optimizer, critic_optimizer,\n",
|
||||||
|
" state, action, reward, next_state, next_action, done)\n",
|
||||||
|
" \n",
|
||||||
|
" # 记录loss\n",
|
||||||
|
" episode_actor_loss += actor_loss\n",
|
||||||
|
" episode_critic_loss += critic_loss\n",
|
||||||
|
" steps += 1\n",
|
||||||
|
" \n",
|
||||||
|
" # 更新状态和动作\n",
|
||||||
|
" state = next_state\n",
|
||||||
|
" action = next_action\n",
|
||||||
|
" episode_reward += reward # type: ignore\n",
|
||||||
|
"\n",
|
||||||
|
" if done:\n",
|
||||||
|
" break\n",
|
||||||
|
"\n",
|
||||||
|
" # 记录本回合数据\n",
|
||||||
|
" episode_rewards.append(episode_reward)\n",
|
||||||
|
" actor_losses.append(episode_actor_loss / steps)\n",
|
||||||
|
" critic_losses.append(episode_critic_loss / steps)\n",
|
||||||
|
"\n",
|
||||||
|
" # 打印训练进度\n",
|
||||||
|
" if (episode + 1) % 100 == 0:\n",
|
||||||
|
" avg_reward = np.mean(episode_rewards[-100:])\n",
|
||||||
|
" print(f\"Episode {episode + 1}/{NUM_EPISODES} completed. 近100回合平均reward: {avg_reward:.2f}\")\n",
|
||||||
|
" \n",
|
||||||
|
" env.close()\n",
|
||||||
|
" \n",
|
||||||
|
" # 绘制训练曲线\n",
|
||||||
|
" plot_qac_results(episode_rewards, actor_losses, critic_losses)\n",
|
||||||
|
"\n",
|
||||||
|
"def plot_qac_results(rewards, actor_losses, critic_losses):\n",
|
||||||
|
" \"\"\"绘制QAC训练结果（保存到文件）\"\"\"\n",
|
||||||
|
" fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
|
||||||
|
" \n",
|
||||||
|
" # 绘制 reward 曲线\n",
|
||||||
|
" axes[0].plot(rewards, alpha=0.6, label='Episode Reward')\n",
|
||||||
|
" # 添加移动平均线\n",
|
||||||
|
" window = min(50, len(rewards) // 10)\n",
|
||||||
|
" if window > 1:\n",
|
||||||
|
" moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')\n",
|
||||||
|
" axes[0].plot(np.arange(window-1, len(rewards)), moving_avg, 'r-', label=f'{window}-Episode Moving Avg')\n",
|
||||||
|
" axes[0].set_xlabel('Episode')\n",
|
||||||
|
" axes[0].set_ylabel('Reward')\n",
|
||||||
|
" axes[0].set_title('QAC Training Rewards')\n",
|
||||||
|
" axes[0].legend()\n",
|
||||||
|
" axes[0].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" # 绘制 Actor loss\n",
|
||||||
|
" axes[1].plot(actor_losses, alpha=0.6, color='orange')\n",
|
||||||
|
" axes[1].set_xlabel('Episode')\n",
|
||||||
|
" axes[1].set_ylabel('Loss')\n",
|
||||||
|
" axes[1].set_title('QAC Actor Loss')\n",
|
||||||
|
" axes[1].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" # 绘制 Critic loss\n",
|
||||||
|
" axes[2].plot(critic_losses, alpha=0.6, color='green')\n",
|
||||||
|
" axes[2].set_xlabel('Episode')\n",
|
||||||
|
" axes[2].set_ylabel('Loss')\n",
|
||||||
|
" axes[2].set_title('QAC Critic Loss')\n",
|
||||||
|
" axes[2].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" plt.tight_layout()\n",
|
||||||
|
" # 保存图片到文件\n",
|
||||||
|
" plt.savefig('qac_training_results.png', dpi=150)\n",
|
||||||
|
" plt.close()\n",
|
||||||
|
" \n",
|
||||||
|
" # 打印统计信息\n",
|
||||||
|
" print(f\"\\n=== QAC 训练统计 ===\")\n",
|
||||||
|
" print(f\"总回合数: {len(rewards)}\")\n",
|
||||||
|
" print(f\"平均Reward: {np.mean(rewards):.2f}\")\n",
|
||||||
|
" print(f\"最高Reward: {np.max(rewards):.2f}\")\n",
|
||||||
|
" print(f\"最后100回合平均Reward: {np.mean(rewards[-100:]):.2f}\")\n",
|
||||||
|
" print(f\"图片已保存到: qac_training_results.png\")\n",
|
||||||
|
"\n",
|
||||||
|
"if __name__ == \"__main__\":\n",
|
||||||
|
" main()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "4cc0f13e",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"## A2C算法\n",
|
||||||
|
"\n",
|
||||||
|
"通过上一个部分，我们看到其实最基础的QAC算法在这个控制算例中效果并不好，原因是他的Actor网络的更新时直接采用了Critic网络的输出作为动作价值的估计，这个估计可能非常不准确，导致Actor网络的更新方向错误，从而无法有效地学习到好的策略。另外，由于QAC是一个On-Policy的算法，一般产生一条轨迹，更新一次参数，而之后这些参数就不再被使用了，这样就浪费了很多数据。我们先解决第一个问题，即方差过大的问题，引入一个基线函数来降低方差，这就是 Advantage Actor-Critic (A2C) 算法。A2C 的核心思想是引入一个基线函数 $b(s)$ 来减去动作价值 $q(s,a)$ 中的平均水平，从而得到优势函数 $A(s,a) = q(s,a) - b(s)$。这个优势函数可以更准确地反映某个动作相对于平均水平的好坏，从而降低了梯度估计的方差。\n",
|
||||||
|
"\n",
|
||||||
|
"从控制系统设计的角度来看，最基础的 QAC 就像是一个只看绝对误差、没有稳态基准的开环打分器，系统噪声和方差极其容易被放大，导致策略输出剧烈震荡。\n",
|
||||||
|
"\n",
|
||||||
|
"A2C引入状态价值 $v_{\\pi}(s)$ 作为基线（Baseline），这就如同在闭环控制中引入了一个动态的参考基准。Critic 不再评估具体的动作有多好，而是评估当前状态的平均预期。Actor 更新的依据变成了优势函数（Advantage）：\n",
|
||||||
|
"$$\\delta_t = r_{t+1} + \\gamma v(s_{t+1}) - v(s_t)$$"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 11,
|
||||||
|
"id": "4809b58c",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"使用设备: cpu\n",
|
||||||
|
"开始 A2C 训练...\n",
|
||||||
|
"回合 100/1000 完成. 近 100 回合平均 Reward: 13.11\n",
|
||||||
|
"回合 200/1000 完成. 近 100 回合平均 Reward: 65.78\n",
|
||||||
|
"回合 300/1000 完成. 近 100 回合平均 Reward: 59.20\n",
|
||||||
|
"回合 400/1000 完成. 近 100 回合平均 Reward: 9.48\n",
|
||||||
|
"回合 500/1000 完成. 近 100 回合平均 Reward: 65.32\n",
|
||||||
|
"回合 600/1000 完成. 近 100 回合平均 Reward: 67.78\n",
|
||||||
|
"回合 700/1000 完成. 近 100 回合平均 Reward: 9.36\n",
|
||||||
|
"回合 800/1000 完成. 近 100 回合平均 Reward: 9.26\n",
|
||||||
|
"回合 900/1000 完成. 近 100 回合平均 Reward: 9.68\n",
|
||||||
|
"回合 1000/1000 完成. 近 100 回合平均 Reward: 51.43\n",
|
||||||
|
"\n",
|
||||||
|
"=== A2C 训练统计 ===\n",
|
||||||
|
"总回合数: 1000\n",
|
||||||
|
"平均Reward: 36.04\n",
|
||||||
|
"最高Reward: 500.00\n",
|
||||||
|
"最后100回合平均Reward: 51.43\n",
|
||||||
|
"图片已保存到: a2c_training_results.png\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# 设置设备 - 使用 CPU\n",
|
||||||
|
"device = torch.device(\"cpu\")\n",
|
||||||
|
"print(f\"使用设备: {device}\")\n",
|
||||||
|
"\n",
|
||||||
|
"# 定义超参数\n",
|
||||||
|
"ACTOR_LR = 0.001\n",
|
||||||
|
"CRITIC_LR = 0.002\n",
|
||||||
|
"GAMMA = 0.99\n",
|
||||||
|
"NUM_EPISODES = 1000\n",
|
||||||
|
"\n",
|
||||||
|
"# 1. 定义 Actor 网络 (演员：输出离散动作的概率分布)\n",
|
||||||
|
"class ActorA2C(nn.Module):\n",
|
||||||
|
" def __init__(self, state_dim, action_dim):\n",
|
||||||
|
" super(ActorA2C, self).__init__()\n",
|
||||||
|
" self.fc1 = nn.Linear(state_dim, 128)\n",
|
||||||
|
" self.fc2 = nn.Linear(128, action_dim)\n",
|
||||||
|
" \n",
|
||||||
|
" def forward(self, state):\n",
|
||||||
|
" x = F.relu(self.fc1(state))\n",
|
||||||
|
" action_probs = F.softmax(self.fc2(x), dim=-1)\n",
|
||||||
|
" return action_probs\n",
|
||||||
|
" \n",
|
||||||
|
"# 2. 定义 Critic 网络 (评论家：输出标量 V 值)\n",
|
||||||
|
"class CriticA2C(nn.Module):\n",
|
||||||
|
" def __init__(self, state_dim):\n",
|
||||||
|
" super(CriticA2C, self).__init__()\n",
|
||||||
|
" self.fc1 = nn.Linear(state_dim, 128)\n",
|
||||||
|
" self.fc2 = nn.Linear(128, 1)\n",
|
||||||
|
" \n",
|
||||||
|
" def forward(self, state):\n",
|
||||||
|
" x = F.relu(self.fc1(state))\n",
|
||||||
|
" v_value = self.fc2(x)\n",
|
||||||
|
" return v_value\n",
|
||||||
|
"\n",
|
||||||
|
"# 3. 核心训练逻辑：基于优势函数 (Advantage)\n",
|
||||||
|
"def train_step_a2c(actor, critic, actor_optimizer, critic_optimizer, \n",
|
||||||
|
" state, action, reward, next_state, done):\n",
|
||||||
|
" state = torch.FloatTensor(state).unsqueeze(0).to(device)\n",
|
||||||
|
" next_state = torch.FloatTensor(next_state).unsqueeze(0).to(device)\n",
|
||||||
|
" reward = torch.FloatTensor([reward]).unsqueeze(0).to(device)\n",
|
||||||
|
" \n",
|
||||||
|
" # Critic 更新\n",
|
||||||
|
" v_value = critic(state)\n",
|
||||||
|
" next_v_value = critic(next_state).detach()\n",
|
||||||
|
" td_target = reward + GAMMA * next_v_value * (1 - int(done))\n",
|
||||||
|
" advantage = td_target - v_value\n",
|
||||||
|
" critic_loss = F.mse_loss(v_value, td_target)\n",
|
||||||
|
" \n",
|
||||||
|
" critic_optimizer.zero_grad()\n",
|
||||||
|
" critic_loss.backward()\n",
|
||||||
|
" critic_optimizer.step()\n",
|
||||||
|
" \n",
|
||||||
|
" # Actor 更新\n",
|
||||||
|
" action_probs = actor(state)\n",
|
||||||
|
" dist = distributions.Categorical(action_probs)\n",
|
||||||
|
" log_prob = dist.log_prob(torch.tensor([action]).to(device))\n",
|
||||||
|
" actor_loss = -(log_prob * advantage.detach()).mean()\n",
|
||||||
|
" \n",
|
||||||
|
" actor_optimizer.zero_grad()\n",
|
||||||
|
" actor_loss.backward()\n",
|
||||||
|
" actor_optimizer.step()\n",
|
||||||
|
"\n",
|
||||||
|
" return actor_loss.item(), critic_loss.item()\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"def plot_a2c_results(rewards, actor_losses, critic_losses):\n",
|
||||||
|
" \"\"\"绘制A2C训练结果（保存到文件）\"\"\"\n",
|
||||||
|
" fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
|
||||||
|
" \n",
|
||||||
|
" axes[0].plot(rewards, alpha=0.6, label='Episode Reward')\n",
|
||||||
|
" window = min(50, len(rewards) // 10)\n",
|
||||||
|
" if window > 1:\n",
|
||||||
|
" moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')\n",
|
||||||
|
" axes[0].plot(np.arange(window-1, len(rewards)), moving_avg, 'r-', label=f'{window}-Episode Moving Avg')\n",
|
||||||
|
" axes[0].set_xlabel('Episode')\n",
|
||||||
|
" axes[0].set_ylabel('Reward')\n",
|
||||||
|
" axes[0].set_title('A2C Training Rewards')\n",
|
||||||
|
" axes[0].legend()\n",
|
||||||
|
" axes[0].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" axes[1].plot(actor_losses, alpha=0.6, color='orange')\n",
|
||||||
|
" axes[1].set_xlabel('Episode')\n",
|
||||||
|
" axes[1].set_ylabel('Loss')\n",
|
||||||
|
" axes[1].set_title('A2C Actor Loss')\n",
|
||||||
|
" axes[1].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" axes[2].plot(critic_losses, alpha=0.6, color='green')\n",
|
||||||
|
" axes[2].set_xlabel('Episode')\n",
|
||||||
|
" axes[2].set_ylabel('Loss')\n",
|
||||||
|
" axes[2].set_title('A2C Critic Loss')\n",
|
||||||
|
" axes[2].grid(True, alpha=0.3)\n",
|
||||||
|
" \n",
|
||||||
|
" plt.tight_layout()\n",
|
||||||
|
" plt.savefig('a2c_training_results.png', dpi=150)\n",
|
||||||
|
" plt.close()\n",
|
||||||
|
" \n",
|
||||||
|
" print(f\"\\n=== A2C 训练统计 ===\")\n",
|
||||||
|
" print(f\"总回合数: {len(rewards)}\")\n",
|
||||||
|
" print(f\"平均Reward: {np.mean(rewards):.2f}\")\n",
|
||||||
|
" print(f\"最高Reward: {np.max(rewards):.2f}\")\n",
|
||||||
|
" print(f\"最后100回合平均Reward: {np.mean(rewards[-100:]):.2f}\")\n",
|
||||||
|
" print(f\"图片已保存到: a2c_training_results.png\")\n",
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"# 4. 主循环函数\n",
|
||||||
|
"def main_a2c():\n",
|
||||||
|
" env = gym.make('CartPole-v1')\n",
|
||||||
|
" state_dim = env.observation_space.shape[0] # type: ignore\n",
|
||||||
|
" action_dim = int(env.action_space.n) # type: ignore\n",
|
||||||
|
"\n",
|
||||||
|
" actor = ActorA2C(state_dim, action_dim).to(device)\n",
|
||||||
|
" critic = CriticA2C(state_dim).to(device)\n",
|
||||||
|
" actor_optimizer = optim.Adam(actor.parameters(), lr=ACTOR_LR)\n",
|
||||||
|
" critic_optimizer = optim.Adam(critic.parameters(), lr=CRITIC_LR)\n",
|
||||||
|
" \n",
|
||||||
|
" episode_rewards = []\n",
|
||||||
|
" actor_losses = []\n",
|
||||||
|
" critic_losses = []\n",
|
||||||
|
" \n",
|
||||||
|
" print(\"开始 A2C 训练...\")\n",
|
||||||
|
" for episode in range(NUM_EPISODES):\n",
|
||||||
|
" state, _ = env.reset()\n",
|
||||||
|
" episode_reward = 0\n",
|
||||||
|
" episode_actor_loss = 0\n",
|
||||||
|
" episode_critic_loss = 0\n",
|
||||||
|
" steps = 0\n",
|
||||||
|
" \n",
|
||||||
|
" while True:\n",
|
||||||
|
" state_tensor = torch.FloatTensor(state).unsqueeze(0).to(device)\n",
|
||||||
|
" action_probs = actor(state_tensor)\n",
|
||||||
|
" action = distributions.Categorical(action_probs).sample().item()\n",
|
||||||
|
" \n",
|
||||||
|
" next_state, reward, terminated, truncated, _ = env.step(action)\n",
|
||||||
|
" done = terminated or truncated\n",
|
||||||
|
" \n",
|
||||||
|
" actor_loss, critic_loss = train_step_a2c(actor, critic, actor_optimizer, critic_optimizer, \n",
|
||||||
|
" state, action, reward, next_state, done)\n",
|
||||||
|
" \n",
|
||||||
|
" episode_actor_loss += actor_loss\n",
|
||||||
|
" episode_critic_loss += critic_loss\n",
|
||||||
|
" steps += 1\n",
|
||||||
|
" \n",
|
||||||
|
" state = next_state\n",
|
||||||
|
" episode_reward += reward\n",
|
||||||
|
" \n",
|
||||||
|
" if done:\n",
|
||||||
|
" break\n",
|
||||||
|
" \n",
|
||||||
|
" episode_rewards.append(episode_reward)\n",
|
||||||
|
" actor_losses.append(episode_actor_loss / steps)\n",
|
||||||
|
" critic_losses.append(episode_critic_loss / steps)\n",
|
||||||
|
" \n",
|
||||||
|
" if (episode + 1) % 100 == 0:\n",
|
||||||
|
" avg_reward = np.mean(episode_rewards[-100:])\n",
|
||||||
|
" print(f\"回合 {episode + 1}/{NUM_EPISODES} 完成. 近 100 回合平均 Reward: {avg_reward:.2f}\")\n",
|
||||||
|
"\n",
|
||||||
|
" env.close()\n",
|
||||||
|
" plot_a2c_results(episode_rewards, actor_losses, critic_losses)\n",
|
||||||
|
"\n",
|
||||||
|
"if __name__ == \"__main__\":\n",
|
||||||
|
" main_a2c()"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "RL_Env",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python3"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.11.15"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 5
|
||||||
|
}
|
||||||
File diff suppressed because one or more lines are too long
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 151 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 135 KiB |
@@ -1,56 +1,57 @@
|
|||||||
# 强化学习的数学基础
|
# RL-Study
|
||||||
|
|
||||||
|
强化学习算法实现与学习笔记，基于赵世钰老师《Mathematical Foundations of Reinforcement Learning》。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
本项目是我在学习赵世钰老师书籍 **《Mathematical Foundations of Reinforcement Learning》** 时建立的个人仓库。主要用于记录学习笔记、公式推导，以及使用 Python/Jupyter Notebook 复现书中的算法和 Grid World 示例。
|
## 项目结构
|
||||||
|
|
||||||
## 📚 关于原书 (Original Book)
|
```
|
||||||
|
RL-Study/
|
||||||
|
├── Lecture slides/ # 课程幻灯片
|
||||||
|
│ ├── slidesForMyLectureVideos/ # 配套视频课件
|
||||||
|
│ └── slidesContinuouslyUpdated/ # 持续更新的课件
|
||||||
|
├── Notebooks/ # Jupyter 学习笔记
|
||||||
|
│ ├── C1.ipynb ~ C10.ipynb # 各章节推导与实验
|
||||||
|
│ ├── SAC.ipynb # SAC (Soft Actor-Critic) 算法
|
||||||
|
│ └── *_training_results.png # 训练结果可视化
|
||||||
|
├── RawBook/ # 原书资源
|
||||||
|
└── RL_Algothrithms/ # 核心算法实现
|
||||||
|
├── agents/ # 智能体实现
|
||||||
|
│ ├── a2c.py # A2C (Advantage Actor-Critic)
|
||||||
|
│ └── qac.py # QAC (Soft Actor-Critic / Q-Value Actor-Critic)
|
||||||
|
├── networks.py # 神经网络定义
|
||||||
|
├── utils.py # 工具函数
|
||||||
|
└── main.py # 训练入口
|
||||||
|
```
|
||||||
|
|
||||||
本项目的核心内容基于赵世钰老师的开源书籍，以下是原书的相关信息：
|
## 已实现算法
|
||||||
|
|
||||||
|
| 算法 | 文件 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| A2C | [a2c.py](RL_Algothrithms/agents/a2c.py) | Advantage Actor-Critic，同步版本 |
|
||||||
|
| QAC | [qac.py](RL_Algothrithms/agents/qac.py) | Q-Value Actor-Critic，支持 GPU |
|
||||||
|
|
||||||
|
## 环境配置
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install torch numpy matplotlib gymnasium
|
||||||
|
```
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd RL_Algothrithms
|
||||||
|
python main.py --agent a2c # 训练 A2C
|
||||||
|
python main.py --agent qac # 训练 QAC
|
||||||
|
```
|
||||||
|
|
||||||
|
## 关于原书
|
||||||
|
|
||||||
- **书名**: Mathematical Foundations of Reinforcement Learning
|
- **书名**: Mathematical Foundations of Reinforcement Learning
|
||||||
- **作者**: Shiyu Zhao (Westlake University)
|
- **作者**: Shiyu Zhao (Westlake University)
|
||||||
- **GitHub 仓库**: [Book-Mathematical-Foundation-of-Reinforcement-Learning](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning)
|
- **GitHub**: [MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning)
|
||||||
- **配套视频**:
|
- **B站**: [赵世钰老师频道](https://space.bilibili.com/2044042934)
|
||||||
- [Bilibili (中文)](https://space.bilibili.com/2044042934)
|
- **YouTube**: [课程列表](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8)
|
||||||
- [YouTube (English)](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8)
|
|
||||||
|
|
||||||
---
|
## License
|
||||||
|
|
||||||
## 📂 项目结构
|
MIT License（代码部分）
|
||||||
|
|
||||||
本项目主要包含以下内容：
|
|
||||||
|
|
||||||
- `Lecture slides/`: 课程相关的幻灯片。
|
|
||||||
- `Notebooks/`: 包含详细推导和实验过程的 Jupyter Notebooks。
|
|
||||||
- `RawBook/`: 原书中的相关资源和代码。
|
|
||||||
- `.gitignore`: Git 忽略文件，指定不需要提交到版本控制的文件或目录。
|
|
||||||
- `README.md`: 项目说明文档。
|
|
||||||
- `LICENSE`: 代码的开源协议说明。
|
|
||||||
|
|
||||||
## 🛠️ 环境配置
|
|
||||||
|
|
||||||
本项目使用 Python 进行开发。推荐使用 `uv` 或 `conda` 管理环境。
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# 安装依赖 (示例)
|
|
||||||
pip install numpy matplotlib jupyter
|
|
||||||
```
|
|
||||||
|
|
||||||
## 📝 开源协议 (License)
|
|
||||||
|
|
||||||
### 关于代码
|
|
||||||
|
|
||||||
本项目中由本人编写的复现代码遵循 **MIT License** 开源协议。这意味着你可以自由地使用、修改和分发这些代码，但请保留原作者的版权声明。
|
|
||||||
|
|
||||||
### 关于笔记
|
|
||||||
|
|
||||||
项目中的学习笔记内容仅供个人学习交流使用。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🤝 致谢
|
|
||||||
|
|
||||||
特别感谢赵世钰老师提供的精彩教材和开源资源，帮助我们深入理解强化学习的数学原理。
|
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
import torch
|
||||||
|
import torch.optim as optim
|
||||||
|
import torch.nn.functional as F
|
||||||
|
import torch.distributions as distributions
|
||||||
|
from networks import Actor, VCritic
|
||||||
|
|
||||||
|
class A2CAgent:
|
||||||
|
def __init__(self, state_dim, action_dim, device, actor_lr=0.001, critic_lr=0.002, gamma=0.99):
|
||||||
|
self.device = device
|
||||||
|
self.gamma = gamma
|
||||||
|
|
||||||
|
# A2C 使用 VCritic
|
||||||
|
self.actor = Actor(state_dim, action_dim).to(self.device)
|
||||||
|
self.critic = VCritic(state_dim).to(self.device)
|
||||||
|
|
||||||
|
self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
|
||||||
|
self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=critic_lr)
|
||||||
|
|
||||||
|
def select_action(self, state):
|
||||||
|
with torch.no_grad():
|
||||||
|
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
|
||||||
|
action_probs = self.actor(state_tensor)
|
||||||
|
action = distributions.Categorical(action_probs).sample().item()
|
||||||
|
return action
|
||||||
|
|
||||||
|
# 注意：A2C 的更新不需要 next_action，但为了与 QAC 的接口统一，此处用 *args 吸收多余参数
|
||||||
|
def update(self, state, action, reward, next_state, next_action, done):
|
||||||
|
state = torch.FloatTensor(state).unsqueeze(0).to(self.device)
|
||||||
|
next_state = torch.FloatTensor(next_state).unsqueeze(0).to(self.device)
|
||||||
|
reward = torch.FloatTensor([reward]).unsqueeze(0).to(self.device)
|
||||||
|
|
||||||
|
# --- Critic 更新 ---
|
||||||
|
v_value = self.critic(state)
|
||||||
|
next_v_value = self.critic(next_state).detach()
|
||||||
|
|
||||||
|
td_target = reward + self.gamma * next_v_value * (1 - int(done))
|
||||||
|
# 计算优势函数 (Advantage)
|
||||||
|
advantage = td_target - v_value
|
||||||
|
|
||||||
|
critic_loss = F.mse_loss(v_value, td_target)
|
||||||
|
|
||||||
|
self.critic_optimizer.zero_grad()
|
||||||
|
critic_loss.backward()
|
||||||
|
self.critic_optimizer.step()
|
||||||
|
|
||||||
|
# --- Actor 更新 ---
|
||||||
|
action_probs = self.actor(state)
|
||||||
|
dist = distributions.Categorical(action_probs)
|
||||||
|
log_prob = dist.log_prob(torch.tensor([action]).to(self.device))
|
||||||
|
|
||||||
|
# 计算策略的熵，鼓励探索
|
||||||
|
entropy = dist.entropy()
|
||||||
|
|
||||||
|
# Actor 梯度上升目标：ln(pi) * Advantage，方差更小
|
||||||
|
actor_loss = -(log_prob * advantage.detach()).mean()- 0.01 * entropy.mean()
|
||||||
|
|
||||||
|
self.actor_optimizer.zero_grad()
|
||||||
|
actor_loss.backward()
|
||||||
|
self.actor_optimizer.step()
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
import torch
|
||||||
|
import torch.optim as optim
|
||||||
|
import torch.nn.functional as F
|
||||||
|
import torch.distributions as distributions
|
||||||
|
from networks import Actor, QCritic
|
||||||
|
|
||||||
|
class QACAgent:
|
||||||
|
def __init__(self, state_dim, action_dim, device, actor_lr=0.001, critic_lr=0.002, gamma=0.99):
|
||||||
|
# 接收设备参数，确保网络挂载在 GPU 或 CPU 上
|
||||||
|
self.device = device
|
||||||
|
self.gamma = gamma
|
||||||
|
|
||||||
|
# 实例化网络并移动到指定设备
|
||||||
|
self.actor = Actor(state_dim, action_dim).to(self.device)
|
||||||
|
self.critic = QCritic(state_dim, action_dim).to(self.device)
|
||||||
|
|
||||||
|
self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
|
||||||
|
self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=critic_lr)
|
||||||
|
|
||||||
|
def select_action(self, state):
|
||||||
|
# 推理时禁用梯度图计算，加快速度
|
||||||
|
with torch.no_grad():
|
||||||
|
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
|
||||||
|
action_probs = self.actor(state_tensor)
|
||||||
|
action = distributions.Categorical(action_probs).sample().item()
|
||||||
|
return action
|
||||||
|
|
||||||
|
def update(self, state, action, reward, next_state, next_action, done):
|
||||||
|
# 将数据转换为张量并送入 GPU
|
||||||
|
state = torch.FloatTensor(state).unsqueeze(0).to(self.device)
|
||||||
|
next_state = torch.FloatTensor(next_state).unsqueeze(0).to(self.device)
|
||||||
|
reward = torch.FloatTensor([reward]).unsqueeze(0).to(self.device)
|
||||||
|
|
||||||
|
# --- Critic 更新 ---
|
||||||
|
q_values = self.critic(state)
|
||||||
|
current_q = q_values[0, action]
|
||||||
|
|
||||||
|
next_q_values = self.critic(next_state).detach()
|
||||||
|
next_q = next_q_values[0, next_action]
|
||||||
|
|
||||||
|
# 计算 TD 目标
|
||||||
|
td_target = reward + self.gamma * next_q * (1 - int(done))
|
||||||
|
critic_loss = F.mse_loss(current_q, td_target.squeeze())
|
||||||
|
|
||||||
|
self.critic_optimizer.zero_grad()
|
||||||
|
critic_loss.backward()
|
||||||
|
self.critic_optimizer.step()
|
||||||
|
|
||||||
|
# --- Actor 更新 ---
|
||||||
|
action_probs = self.actor(state)
|
||||||
|
dist = distributions.Categorical(action_probs)
|
||||||
|
log_prob = dist.log_prob(torch.tensor([action]).to(self.device))
|
||||||
|
|
||||||
|
# Actor 梯度上升目标：ln(pi) * Q(s, a)
|
||||||
|
actor_loss = -(log_prob * current_q.detach())
|
||||||
|
|
||||||
|
self.actor_optimizer.zero_grad()
|
||||||
|
actor_loss.backward()
|
||||||
|
self.actor_optimizer.step()
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
import gymnasium as gym
|
||||||
|
import torch
|
||||||
|
from agents.qac import QACAgent
|
||||||
|
from agents.a2c import A2CAgent
|
||||||
|
from utils import plot_comparison
|
||||||
|
|
||||||
|
def train_agent(env_name, agent_class, device, num_episodes=500):
|
||||||
|
"""
|
||||||
|
通用的训练循环函数
|
||||||
|
"""
|
||||||
|
env = gym.make(env_name)
|
||||||
|
state_dim = env.observation_space.shape[0] # type: ignore
|
||||||
|
action_dim = int(env.action_space.n) # type: ignore
|
||||||
|
|
||||||
|
# 实例化传入的算法代理
|
||||||
|
agent = agent_class(state_dim, action_dim, device)
|
||||||
|
|
||||||
|
rewards_history = []
|
||||||
|
|
||||||
|
for episode in range(num_episodes):
|
||||||
|
state, _ = env.reset()
|
||||||
|
episode_reward = 0
|
||||||
|
|
||||||
|
# QAC 属于 Sarsa 类，需要提前采样第一个动作
|
||||||
|
action = agent.select_action(state)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
next_state, reward, terminated, truncated, _ = env.step(action)
|
||||||
|
done = terminated or truncated
|
||||||
|
|
||||||
|
# 预采样下一个动作
|
||||||
|
next_action = agent.select_action(next_state)
|
||||||
|
|
||||||
|
# 统一的接口调用更新
|
||||||
|
agent.update(state, action, reward, next_state, next_action, done)
|
||||||
|
|
||||||
|
state = next_state
|
||||||
|
action = next_action
|
||||||
|
episode_reward += reward # type: ignore
|
||||||
|
|
||||||
|
if done:
|
||||||
|
break
|
||||||
|
|
||||||
|
rewards_history.append(episode_reward)
|
||||||
|
if (episode + 1) % 100 == 0:
|
||||||
|
print(f"[{agent_class.__name__}] 回合 {episode+1}/{num_episodes}, 近100回合均分: {sum(rewards_history[-100:])/100:.2f}")
|
||||||
|
|
||||||
|
env.close()
|
||||||
|
return rewards_history
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
# 检测 GPU
|
||||||
|
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||||
|
print(f"正在使用的计算设备: {device}")
|
||||||
|
|
||||||
|
ENV_NAME = 'CartPole-v1'
|
||||||
|
EPISODES = 600
|
||||||
|
|
||||||
|
print("\n--- 开始训练 QAC ---")
|
||||||
|
qac_rewards = train_agent(ENV_NAME, QACAgent, device, EPISODES)
|
||||||
|
|
||||||
|
print("\n--- 开始训练 A2C ---")
|
||||||
|
a2c_rewards = train_agent(ENV_NAME, A2CAgent, device, EPISODES)
|
||||||
|
|
||||||
|
# 收集结果并绘图对比
|
||||||
|
results = {
|
||||||
|
'QAC (High Variance)': qac_rewards,
|
||||||
|
'A2C (Low Variance)': a2c_rewards
|
||||||
|
}
|
||||||
|
|
||||||
|
plot_comparison(results, window=50, save_path='qac_vs_a2c_gpu.png')
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import torch
|
||||||
|
import torch.nn as nn
|
||||||
|
import torch.nn.functional as F
|
||||||
|
|
||||||
|
# 策略网络 (离散动作 Actor)
|
||||||
|
class Actor(nn.Module):
|
||||||
|
def __init__(self, state_dim, action_dim):
|
||||||
|
super(Actor, self).__init__()
|
||||||
|
# 定义两层隐藏层
|
||||||
|
self.fc1 = nn.Linear(state_dim, 128)
|
||||||
|
self.fc2 = nn.Linear(128, action_dim)
|
||||||
|
|
||||||
|
def forward(self, state):
|
||||||
|
x = F.relu(self.fc1(state))
|
||||||
|
# 输出动作的概率分布
|
||||||
|
action_probs = F.softmax(self.fc2(x), dim=-1)
|
||||||
|
return action_probs
|
||||||
|
|
||||||
|
# Q值网络 (用于 QAC)
|
||||||
|
class QCritic(nn.Module):
|
||||||
|
def __init__(self, state_dim, action_dim):
|
||||||
|
super(QCritic, self).__init__()
|
||||||
|
self.fc1 = nn.Linear(state_dim, 128)
|
||||||
|
self.fc2 = nn.Linear(128, action_dim)
|
||||||
|
|
||||||
|
def forward(self, state):
|
||||||
|
x = F.relu(self.fc1(state))
|
||||||
|
# 输出各个动作的具体价值
|
||||||
|
q_values = self.fc2(x)
|
||||||
|
return q_values
|
||||||
|
|
||||||
|
# V值网络 (用于 A2C，由于加入了基线，只需输出状态价值标量)
|
||||||
|
class VCritic(nn.Module):
|
||||||
|
def __init__(self, state_dim):
|
||||||
|
super(VCritic, self).__init__()
|
||||||
|
self.fc1 = nn.Linear(state_dim, 128)
|
||||||
|
self.fc2 = nn.Linear(128, 1)
|
||||||
|
|
||||||
|
def forward(self, state):
|
||||||
|
x = F.relu(self.fc1(state))
|
||||||
|
# 输出当前状态的价值评估
|
||||||
|
v_value = self.fc2(x)
|
||||||
|
return v_value
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
import matplotlib
|
||||||
|
# 关键设置：针对 Linux 服务器无 GUI 环境，强制使用 Agg 后端进行纯文件渲染
|
||||||
|
matplotlib.use('Agg')
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
def plot_comparison(results_dict, window=50, save_path='comparison_result.png'):
|
||||||
|
"""
|
||||||
|
绘制并保存算法对比曲线
|
||||||
|
:param results_dict: 字典格式 {'QAC': [奖励列表], 'A2C': [奖励列表]}
|
||||||
|
:param window: 移动平均的窗口大小
|
||||||
|
:param save_path: 图片保存路径
|
||||||
|
"""
|
||||||
|
plt.figure(figsize=(10, 6))
|
||||||
|
|
||||||
|
for algo_name, rewards in results_dict.items():
|
||||||
|
# 绘制原始透明度较低的曲线
|
||||||
|
plt.plot(rewards, alpha=0.3, label=f'{algo_name} (Raw)')
|
||||||
|
|
||||||
|
# 计算并绘制移动平均曲线，使趋势更平滑
|
||||||
|
if len(rewards) >= window:
|
||||||
|
moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')
|
||||||
|
plt.plot(np.arange(window-1, len(rewards)), moving_avg, linewidth=2, label=f'{algo_name} (Avg {window})')
|
||||||
|
|
||||||
|
plt.xlabel('Episode')
|
||||||
|
plt.ylabel('Total Reward')
|
||||||
|
plt.title('Algorithm Comparison: QAC vs A2C')
|
||||||
|
plt.legend()
|
||||||
|
plt.grid(True, alpha=0.3)
|
||||||
|
|
||||||
|
# 将图像保存到服务器硬盘
|
||||||
|
plt.savefig(save_path, dpi=300)
|
||||||
|
plt.close()
|
||||||
|
print(f"对比图像已成功保存至: {save_path}")
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 390 KiB |
Reference in New Issue
Block a user