添加 A2C/QAC 算法实现及训练结果

- 新增 RL_Algothrithms 模块，包含 A2C、QAC 智能体
- 添加 SAC 章节笔记和 C10 笔记
- 上传训练结果图片
- 完善 README 与 .gitignore

.gitignore

@@ -45,3 +45,9 @@ Thumbs.db
 # Byte-compiled / optimized / DLL files
 *.pyc
 *.pyd
+
+# Images (uncomment/add specific images to track them)
+# *.png
+# *.jpg
+# *.jpeg
+# *.gif

Notebooks/C10.ipynb

@@ -0,0 +1,610 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "ef80017f",
+   "metadata": {},
+   "source": [
+    "# C10 演员评论家方法\n",
+    "\n",
+    "## 最基础的AC算法：QAC\n",
+    "\n",
+    "Actor网络：策略梯度上升\n",
+    "\n",
+    "直接更新策略参数，以采取更优的动作。\n",
+    "\n",
+    "我们希望最优化一个标量指标$J(\\theta)$，他的梯度包含一个期望值：\n",
+    "\n",
+    "$$\\nabla_{\\theta}J(\\theta)=\\mathbb{E}_{S\\sim\\eta,A\\sim\\pi}[\\nabla_{\\theta}\\ln \\pi(A|S,\\theta_{t})q_{\\pi}(S,A)]$$\n",
+    "\n",
+    "由于实际工程中很难求出这个数学期望的解析解，因此采用随机的采样方法进行梯度上升的近似更新：\n",
+    "\n",
+    "$$\\theta_{t+1}=\\theta_{t}+\\alpha_{\\theta}\\nabla_{\\theta}\\ln \\pi(a_{t}|s_{t},\\theta_{t})q(s_{t},a_{t},w_{t})$$\n",
+    "\n",
+    "如果某个动作在状态 $s_t$ 下得到了很高的 $q$ 值，梯度更新就会使得网络参数 $\\theta$ 发生改变，从而提高未来在相同状态下采取动作 $a_t$ 的概率 $\\pi$。这就是 Actor 学习的过程。\n",
+    "\n",
+    "Critic网络：价值的时序差分更新\n",
+    "\n",
+    "在 Actor 的更新公式中，我们需要知道动作价值 $q_t$ 。如果采用蒙特卡洛方法（走完一整个回合再算），那就是 REINFORCE 算法 。但为了实现更高效的单步在线学习，我们使用时序差分（TD）学习来估计这个值，这就是 Actor-Critic 的精髓。\n",
+    "\n",
+    "Critic 的任务是通过评估动作的价值来“批评” Actor 。在最基础的 QAC 算法中，它使用类似 Sarsa 的方式更新自己的参数 $w$：\n",
+    "\n",
+    "$$w_{t+1}=w_{t}+\\alpha_{w}[r_{t+1}+\\gamma q(s_{t+1},a_{t+1},w_{t})-q(s_{t},a_{t},w_{t})]\\nabla_{w}q(s_{t},a_{t},w_{t})$$。\n",
+    "\n",
+    "公式方括号内的部分 $[r_{t+1}+\\gamma q(s_{t+1},a_{t+1},w_{t})-q(s_{t},a_{t},w_{t})]$ 就是著名的 TD 误差。它衡量了“当前预测的 $q$ 值”与“实际得到的奖励加上下一步预测的 $q$ 值”之间的差距。\n",
+    "\n",
+    "算法流程：\n",
+    "\n",
+    "1. 策略执行（Actor 正向计算）： 系统当前处于状态 $s_t$，Actor 依据当前的概率分布 $\\pi(a|s_{t},\\theta_{t})$ 采样出一个动作$a_t$并执行，获得真实物理系统的反馈奖励 $r_{t+1}$ 和下一个状态 $s_{t+1}$ 。\n",
+    "2. 价值评估（Critic 反向更新）： 系统依据策略再预演一步动作 $a_{t+1}$，以此计算 TD 误差，并使用梯度下降法更新打分器参数 $w_{t+1}$。\n",
+    "3. 策略优化（Actor 反向更新）： 拿着刚刚打出的分数 $q(s_{t},a_{t},w_{t})$，沿着梯度上升的方向更新策略参数 $\\theta_{t+1}$。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "c2f49de7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "PyTorch 版本: 2.5.1+cu121\n",
+      "GPU 是否可用: True\n",
+      "当前显卡: NVIDIA GeForce RTX 5060 Ti\n"
+     ]
+    }
+   ],
+   "source": [
+    "import torch\n",
+    "\n",
+    "# 打印 PyTorch 版本\n",
+    "print(f\"PyTorch 版本: {torch.__version__}\")\n",
+    "# 检查 GPU 是否可用 (如果输出 True，说明大功告成！)\n",
+    "print(f\"GPU 是否可用: {torch.cuda.is_available()}\")\n",
+    "\n",
+    "if torch.cuda.is_available():\n",
+    "    # 打印当前使用的显卡型号\n",
+    "    print(f\"当前显卡: {torch.cuda.get_device_name(0)}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "id": "79226b49",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "import torch.optim as optim\n",
+    "import torch.nn.functional as F\n",
+    "import torch.distributions as distributions\n",
+    "\n",
+    "# 定义超参数\n",
+    "# 学习率\n",
+    "ACTOR_LR = 0.001\n",
+    "CRITIC_LR = 0.002\n",
+    "# 折扣因子\n",
+    "GAMMA = 0.99\n",
+    "# 训练迭代次数\n",
+    "NUM_EPISODES = 1000\n",
+    "\n",
+    "# 1. 定义 Actor 网络 (演员：输出策略)\n",
+    "class Actor(nn.Module):\n",
+    "    def __init__(self, state_dim, action_dim):\n",
+    "        super(Actor, self).__init__()\n",
+    "        # 定义两层全连接网络提取状态特征\n",
+    "        self.fc1 = nn.Linear(state_dim, 128)\n",
+    "        self.fc2 = nn.Linear(128, action_dim)\n",
+    "        \n",
+    "    def forward(self, state):\n",
+    "        # 使用 ReLU 激活函数\n",
+    "        x = F.relu(self.fc1(state))\n",
+    "        # 使用 Softmax 输出离散动作的概率分布\n",
+    "        action_probs = F.softmax(self.fc2(x), dim=-1)\n",
+    "        return action_probs\n",
+    "\n",
+    "# 2. 定义 Critic 网络 (评论家：输出动作价值 Q 值)\n",
+    "class Critic(nn.Module):\n",
+    "    def __init__(self, state_dim, action_dim):\n",
+    "        super(Critic, self).__init__()\n",
+    "        # 定义全连接网络，输入是状态，输出是各个动作的 Q 值\n",
+    "        self.fc1 = nn.Linear(state_dim, 128)\n",
+    "        self.fc2 = nn.Linear(128, action_dim)\n",
+    "        \n",
+    "    def forward(self, state):\n",
+    "        x = F.relu(self.fc1(state))\n",
+    "        q_values = self.fc2(x)\n",
+    "        return q_values\n",
+    "\n",
+    "\n",
+    "# 3. 核心训练逻辑 (对应 Algorithm 10.1)\n",
+    "def train_step(actor, critic, actor_optimizer, critic_optimizer, \n",
+    "               state, action, reward, next_state, next_action, done):\n",
+    "    \"\"\"\n",
+    "    执行一步 QAC 算法的参数更新\n",
+    "    \"\"\"\n",
+    "    # 转换数据格式为 Tensor\n",
+    "    state = torch.FloatTensor(state)\n",
+    "    next_state = torch.FloatTensor(next_state)\n",
+    "    reward = torch.FloatTensor([reward])\n",
+    "    \n",
+    "    # -----------------------------------------\n",
+    "    # Critic 更新 (价值更新)\n",
+    "    # -----------------------------------------\n",
+    "    # 计算当前状态动作的 Q(s_t, a_t, w_t)\n",
+    "    q_values = critic(state)        # 选取某个state对应的所有动作的Q值\n",
+    "    current_q = q_values[action]    # 选取当前动作对应的Q值\n",
+    "    \n",
+    "    # 计算下一状态动作的 Q(s_{t+1}, a_{t+1}, w_t)\n",
+    "    # 使用 .detach() 截断梯度，因为目标值不需要传递梯度回网络\n",
+    "    next_q_values = critic(next_state).detach()\n",
+    "    next_q = next_q_values[next_action]\n",
+    "    \n",
+    "    # 计算 TD 目标：r_{t+1} + gamma * Q(s_{t+1}, a_{t+1}) (如果是终止状态则没有下一步的Q)\n",
+    "    td_target = reward + GAMMA * next_q * (1 - int(done))\n",
+    "    \n",
+    "    # 计算 TD 误差并更新 Critic 参数\n",
+    "    # 对应公式: w_{t+1} = w_t + alpha_w * TD_Error * grad(Q)\n",
+    "    critic_loss = F.mse_loss(current_q, td_target)\n",
+    "    \n",
+    "    critic_optimizer.zero_grad()\n",
+    "    critic_loss.backward()\n",
+    "    critic_optimizer.step()\n",
+    "    \n",
+    "    # -----------------------------------------\n",
+    "    # Actor 更新 (策略更新)\n",
+    "    # -----------------------------------------\n",
+    "    # 获取当前状态下所有动作的概率分布\n",
+    "    action_probs = actor(state)\n",
+    "    # 构建概率分布对象，方便计算对数概率\n",
+    "    dist = distributions.Categorical(action_probs)\n",
+    "    \n",
+    "    # 计算 ln(pi(a_t | s_t, theta_t))\n",
+    "    log_prob = dist.log_prob(torch.tensor(action))\n",
+    "    \n",
+    "    # Actor 梯度上升目标：ln(pi) * Q(s, a)\n",
+    "    # 在 PyTorch 中优化器默认执行梯度下降，所以加个负号变成最小化损失\n",
+    "    # 注意这里使用的是刚刚更新前算出的 current_q，为了阻断梯度传到 Critic，使用 .detach()\n",
+    "    actor_loss = -log_prob * current_q.detach()\n",
+    "    \n",
+    "    actor_optimizer.zero_grad()\n",
+    "    actor_loss.backward()\n",
+    "    actor_optimizer.step()\n",
+    "\n",
+    "    return actor_loss.item(), critic_loss.item()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "81a39722",
+   "metadata": {},
+   "source": [
+    "\n",
+    "## 控制任务\n",
+    "\n",
+    "本处使用OpenAI的Gymnasium环境中的CartPole-v1作为测试环境。这个环境的目标是通过控制一个小车来保持杆子竖直。状态空间是连续的，包含了小车的位置、速度以及杆子的角度和角速度；动作空间是离散的，只有两个动作：向左或向右移动小车。\n",
+    "\n",
+    "状态空间: 一个四维的连续向量\n",
+    "- $s_0$: 小车的位置\n",
+    "- $s_1$: 小车的速度\n",
+    "- $s_2$: 杆子的角度\n",
+    "- $s_3$: 杆子的角速度\n",
+    "\n",
+    "动作空间: 两个离散动作\n",
+    "- $a_0$: 向左移动小车\n",
+    "- $a_1$: 向右移动小车\n",
+    "\n",
+    "奖励函数: 每个时间步获得的奖励为 +1，直到杆子倒下或小车移出边界。"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "cec5bd6b",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "使用设备: cpu\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/tmp/ipykernel_77052/2107052713.py:73: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.\n",
+      "  critic_loss = F.mse_loss(current_q, td_target)\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Episode 100/1000 completed. 近100回合平均reward: 15.25\n",
+      "Episode 200/1000 completed. 近100回合平均reward: 33.00\n",
+      "Episode 300/1000 completed. 近100回合平均reward: 53.27\n",
+      "Episode 400/1000 completed. 近100回合平均reward: 34.57\n",
+      "Episode 500/1000 completed. 近100回合平均reward: 59.50\n",
+      "Episode 600/1000 completed. 近100回合平均reward: 51.37\n",
+      "Episode 700/1000 completed. 近100回合平均reward: 77.24\n",
+      "Episode 800/1000 completed. 近100回合平均reward: 77.66\n",
+      "Episode 900/1000 completed. 近100回合平均reward: 78.07\n",
+      "Episode 1000/1000 completed. 近100回合平均reward: 51.60\n",
+      "\n",
+      "=== QAC 训练统计 ===\n",
+      "总回合数: 1000\n",
+      "平均Reward: 53.15\n",
+      "最高Reward: 336.00\n",
+      "最后100回合平均Reward: 51.60\n",
+      "图片已保存到: qac_training_results.png\n"
+     ]
+    }
+   ],
+   "source": [
+    "# 设置设备 - 使用 CPU\n",
+    "device = torch.device(\"cpu\")\n",
+    "print(f\"使用设备: {device}\")\n",
+    "\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "import gymnasium as gym\n",
+    "\n",
+    "# 训练的主循环\n",
+    "def main():\n",
+    "    # 初始化环境\n",
+    "    env = gym.make('CartPole-v1')\n",
+    "    state_dim = env.observation_space.shape[0]  # type: ignore\n",
+    "    action_dim = int(env.action_space.n) # type: ignore\n",
+    "\n",
+    "    # 实例化 Actor 和 Critic 网络\n",
+    "    actor = Actor(state_dim, action_dim).to(device)\n",
+    "    critic = Critic(state_dim, action_dim).to(device)\n",
+    "    actor_optimizer = optim.Adam(actor.parameters(), lr=ACTOR_LR)\n",
+    "    critic_optimizer = optim.Adam(critic.parameters(), lr=CRITIC_LR)\n",
+    "    \n",
+    "    # 记录训练数据\n",
+    "    episode_rewards = []\n",
+    "    actor_losses = []\n",
+    "    critic_losses = []\n",
+    "\n",
+    "    for episode in range(NUM_EPISODES):\n",
+    "        state, _ = env.reset()\n",
+    "        episode_reward = 0\n",
+    "        episode_actor_loss = 0\n",
+    "        episode_critic_loss = 0\n",
+    "        steps = 0\n",
+    "\n",
+    "        # 预先生成一个初始动作 a_0\n",
+    "        state_tensor = torch.FloatTensor(state).to(device)\n",
+    "        action_probs = actor(state_tensor)\n",
+    "        action = distributions.Categorical(action_probs).sample().item()\n",
+    "\n",
+    "        while True:\n",
+    "            # 与环境交互，获取当前状态\n",
+    "            next_state, reward, terminated, truncated, _ = env.step(action)\n",
+    "            done = terminated or truncated\n",
+    "\n",
+    "            # 预先生成下一个动作 a_{t+1}，供 Critic 更新使用\n",
+    "            next_state_tensor = torch.FloatTensor(next_state).to(device)\n",
+    "            next_action_probs = actor(next_state_tensor)\n",
+    "            next_action = distributions.Categorical(next_action_probs).sample().item()\n",
+    "\n",
+    "            # 执行单步的学习\n",
+    "            actor_loss, critic_loss = train_step(actor, critic, actor_optimizer, critic_optimizer,\n",
+    "                          state, action, reward, next_state, next_action, done)\n",
+    "            \n",
+    "            # 记录loss\n",
+    "            episode_actor_loss += actor_loss\n",
+    "            episode_critic_loss += critic_loss\n",
+    "            steps += 1\n",
+    "            \n",
+    "            # 更新状态和动作\n",
+    "            state = next_state\n",
+    "            action = next_action\n",
+    "            episode_reward += reward  # type: ignore\n",
+    "\n",
+    "            if done:\n",
+    "                break\n",
+    "\n",
+    "        # 记录本回合数据\n",
+    "        episode_rewards.append(episode_reward)\n",
+    "        actor_losses.append(episode_actor_loss / steps)\n",
+    "        critic_losses.append(episode_critic_loss / steps)\n",
+    "\n",
+    "        # 打印训练进度\n",
+    "        if (episode + 1) % 100 == 0:\n",
+    "            avg_reward = np.mean(episode_rewards[-100:])\n",
+    "            print(f\"Episode {episode + 1}/{NUM_EPISODES} completed. 近100回合平均reward: {avg_reward:.2f}\")\n",
+    "    \n",
+    "    env.close()\n",
+    "    \n",
+    "    # 绘制训练曲线\n",
+    "    plot_qac_results(episode_rewards, actor_losses, critic_losses)\n",
+    "\n",
+    "def plot_qac_results(rewards, actor_losses, critic_losses):\n",
+    "    \"\"\"绘制QAC训练结果（保存到文件）\"\"\"\n",
+    "    fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
+    "    \n",
+    "    # 绘制 reward 曲线\n",
+    "    axes[0].plot(rewards, alpha=0.6, label='Episode Reward')\n",
+    "    # 添加移动平均线\n",
+    "    window = min(50, len(rewards) // 10)\n",
+    "    if window > 1:\n",
+    "        moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')\n",
+    "        axes[0].plot(np.arange(window-1, len(rewards)), moving_avg, 'r-', label=f'{window}-Episode Moving Avg')\n",
+    "    axes[0].set_xlabel('Episode')\n",
+    "    axes[0].set_ylabel('Reward')\n",
+    "    axes[0].set_title('QAC Training Rewards')\n",
+    "    axes[0].legend()\n",
+    "    axes[0].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    # 绘制 Actor loss\n",
+    "    axes[1].plot(actor_losses, alpha=0.6, color='orange')\n",
+    "    axes[1].set_xlabel('Episode')\n",
+    "    axes[1].set_ylabel('Loss')\n",
+    "    axes[1].set_title('QAC Actor Loss')\n",
+    "    axes[1].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    # 绘制 Critic loss\n",
+    "    axes[2].plot(critic_losses, alpha=0.6, color='green')\n",
+    "    axes[2].set_xlabel('Episode')\n",
+    "    axes[2].set_ylabel('Loss')\n",
+    "    axes[2].set_title('QAC Critic Loss')\n",
+    "    axes[2].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    plt.tight_layout()\n",
+    "    # 保存图片到文件\n",
+    "    plt.savefig('qac_training_results.png', dpi=150)\n",
+    "    plt.close()\n",
+    "    \n",
+    "    # 打印统计信息\n",
+    "    print(f\"\\n=== QAC 训练统计 ===\")\n",
+    "    print(f\"总回合数: {len(rewards)}\")\n",
+    "    print(f\"平均Reward: {np.mean(rewards):.2f}\")\n",
+    "    print(f\"最高Reward: {np.max(rewards):.2f}\")\n",
+    "    print(f\"最后100回合平均Reward: {np.mean(rewards[-100:]):.2f}\")\n",
+    "    print(f\"图片已保存到: qac_training_results.png\")\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4cc0f13e",
+   "metadata": {},
+   "source": [
+    "## A2C算法\n",
+    "\n",
+    "通过上一个部分，我们看到其实最基础的QAC算法在这个控制算例中效果并不好，原因是他的Actor网络的更新时直接采用了Critic网络的输出作为动作价值的估计，这个估计可能非常不准确，导致Actor网络的更新方向错误，从而无法有效地学习到好的策略。另外，由于QAC是一个On-Policy的算法，一般产生一条轨迹，更新一次参数，而之后这些参数就不再被使用了，这样就浪费了很多数据。我们先解决第一个问题，即方差过大的问题，引入一个基线函数来降低方差，这就是 Advantage Actor-Critic (A2C) 算法。A2C 的核心思想是引入一个基线函数 $b(s)$ 来减去动作价值 $q(s,a)$ 中的平均水平，从而得到优势函数 $A(s,a) = q(s,a) - b(s)$。这个优势函数可以更准确地反映某个动作相对于平均水平的好坏，从而降低了梯度估计的方差。\n",
+    "\n",
+    "从控制系统设计的角度来看，最基础的 QAC 就像是一个只看绝对误差、没有稳态基准的开环打分器，系统噪声和方差极其容易被放大，导致策略输出剧烈震荡。\n",
+    "\n",
+    "A2C引入状态价值 $v_{\\pi}(s)$ 作为基线（Baseline），这就如同在闭环控制中引入了一个动态的参考基准。Critic 不再评估具体的动作有多好，而是评估当前状态的平均预期。Actor 更新的依据变成了优势函数（Advantage）：\n",
+    "$$\\delta_t = r_{t+1} + \\gamma v(s_{t+1}) - v(s_t)$$"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "4809b58c",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "使用设备: cpu\n",
+      "开始 A2C 训练...\n",
+      "回合 100/1000 完成. 近 100 回合平均 Reward: 13.11\n",
+      "回合 200/1000 完成. 近 100 回合平均 Reward: 65.78\n",
+      "回合 300/1000 完成. 近 100 回合平均 Reward: 59.20\n",
+      "回合 400/1000 完成. 近 100 回合平均 Reward: 9.48\n",
+      "回合 500/1000 完成. 近 100 回合平均 Reward: 65.32\n",
+      "回合 600/1000 完成. 近 100 回合平均 Reward: 67.78\n",
+      "回合 700/1000 完成. 近 100 回合平均 Reward: 9.36\n",
+      "回合 800/1000 完成. 近 100 回合平均 Reward: 9.26\n",
+      "回合 900/1000 完成. 近 100 回合平均 Reward: 9.68\n",
+      "回合 1000/1000 完成. 近 100 回合平均 Reward: 51.43\n",
+      "\n",
+      "=== A2C 训练统计 ===\n",
+      "总回合数: 1000\n",
+      "平均Reward: 36.04\n",
+      "最高Reward: 500.00\n",
+      "最后100回合平均Reward: 51.43\n",
+      "图片已保存到: a2c_training_results.png\n"
+     ]
+    }
+   ],
+   "source": [
+    "# 设置设备 - 使用 CPU\n",
+    "device = torch.device(\"cpu\")\n",
+    "print(f\"使用设备: {device}\")\n",
+    "\n",
+    "# 定义超参数\n",
+    "ACTOR_LR = 0.001\n",
+    "CRITIC_LR = 0.002\n",
+    "GAMMA = 0.99\n",
+    "NUM_EPISODES = 1000\n",
+    "\n",
+    "# 1. 定义 Actor 网络 (演员：输出离散动作的概率分布)\n",
+    "class ActorA2C(nn.Module):\n",
+    "    def __init__(self, state_dim, action_dim):\n",
+    "        super(ActorA2C, self).__init__()\n",
+    "        self.fc1 = nn.Linear(state_dim, 128)\n",
+    "        self.fc2 = nn.Linear(128, action_dim)\n",
+    "        \n",
+    "    def forward(self, state):\n",
+    "        x = F.relu(self.fc1(state))\n",
+    "        action_probs = F.softmax(self.fc2(x), dim=-1)\n",
+    "        return action_probs\n",
+    "    \n",
+    "# 2. 定义 Critic 网络 (评论家：输出标量 V 值)\n",
+    "class CriticA2C(nn.Module):\n",
+    "    def __init__(self, state_dim):\n",
+    "        super(CriticA2C, self).__init__()\n",
+    "        self.fc1 = nn.Linear(state_dim, 128)\n",
+    "        self.fc2 = nn.Linear(128, 1)\n",
+    "        \n",
+    "    def forward(self, state):\n",
+    "        x = F.relu(self.fc1(state))\n",
+    "        v_value = self.fc2(x)\n",
+    "        return v_value\n",
+    "\n",
+    "# 3. 核心训练逻辑：基于优势函数 (Advantage)\n",
+    "def train_step_a2c(actor, critic, actor_optimizer, critic_optimizer, \n",
+    "                   state, action, reward, next_state, done):\n",
+    "    state = torch.FloatTensor(state).unsqueeze(0).to(device)\n",
+    "    next_state = torch.FloatTensor(next_state).unsqueeze(0).to(device)\n",
+    "    reward = torch.FloatTensor([reward]).unsqueeze(0).to(device)\n",
+    "    \n",
+    "    # Critic 更新\n",
+    "    v_value = critic(state)\n",
+    "    next_v_value = critic(next_state).detach()\n",
+    "    td_target = reward + GAMMA * next_v_value * (1 - int(done))\n",
+    "    advantage = td_target - v_value\n",
+    "    critic_loss = F.mse_loss(v_value, td_target)\n",
+    "    \n",
+    "    critic_optimizer.zero_grad()\n",
+    "    critic_loss.backward()\n",
+    "    critic_optimizer.step()\n",
+    "    \n",
+    "    # Actor 更新\n",
+    "    action_probs = actor(state)\n",
+    "    dist = distributions.Categorical(action_probs)\n",
+    "    log_prob = dist.log_prob(torch.tensor([action]).to(device))\n",
+    "    actor_loss = -(log_prob * advantage.detach()).mean()\n",
+    "    \n",
+    "    actor_optimizer.zero_grad()\n",
+    "    actor_loss.backward()\n",
+    "    actor_optimizer.step()\n",
+    "\n",
+    "    return actor_loss.item(), critic_loss.item()\n",
+    "\n",
+    "\n",
+    "def plot_a2c_results(rewards, actor_losses, critic_losses):\n",
+    "    \"\"\"绘制A2C训练结果（保存到文件）\"\"\"\n",
+    "    fig, axes = plt.subplots(1, 3, figsize=(15, 4))\n",
+    "    \n",
+    "    axes[0].plot(rewards, alpha=0.6, label='Episode Reward')\n",
+    "    window = min(50, len(rewards) // 10)\n",
+    "    if window > 1:\n",
+    "        moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')\n",
+    "        axes[0].plot(np.arange(window-1, len(rewards)), moving_avg, 'r-', label=f'{window}-Episode Moving Avg')\n",
+    "    axes[0].set_xlabel('Episode')\n",
+    "    axes[0].set_ylabel('Reward')\n",
+    "    axes[0].set_title('A2C Training Rewards')\n",
+    "    axes[0].legend()\n",
+    "    axes[0].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    axes[1].plot(actor_losses, alpha=0.6, color='orange')\n",
+    "    axes[1].set_xlabel('Episode')\n",
+    "    axes[1].set_ylabel('Loss')\n",
+    "    axes[1].set_title('A2C Actor Loss')\n",
+    "    axes[1].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    axes[2].plot(critic_losses, alpha=0.6, color='green')\n",
+    "    axes[2].set_xlabel('Episode')\n",
+    "    axes[2].set_ylabel('Loss')\n",
+    "    axes[2].set_title('A2C Critic Loss')\n",
+    "    axes[2].grid(True, alpha=0.3)\n",
+    "    \n",
+    "    plt.tight_layout()\n",
+    "    plt.savefig('a2c_training_results.png', dpi=150)\n",
+    "    plt.close()\n",
+    "    \n",
+    "    print(f\"\\n=== A2C 训练统计 ===\")\n",
+    "    print(f\"总回合数: {len(rewards)}\")\n",
+    "    print(f\"平均Reward: {np.mean(rewards):.2f}\")\n",
+    "    print(f\"最高Reward: {np.max(rewards):.2f}\")\n",
+    "    print(f\"最后100回合平均Reward: {np.mean(rewards[-100:]):.2f}\")\n",
+    "    print(f\"图片已保存到: a2c_training_results.png\")\n",
+    "\n",
+    "\n",
+    "# 4. 主循环函数\n",
+    "def main_a2c():\n",
+    "    env = gym.make('CartPole-v1')\n",
+    "    state_dim = env.observation_space.shape[0]   # type: ignore\n",
+    "    action_dim = int(env.action_space.n)         # type: ignore\n",
+    "\n",
+    "    actor = ActorA2C(state_dim, action_dim).to(device)\n",
+    "    critic = CriticA2C(state_dim).to(device)\n",
+    "    actor_optimizer = optim.Adam(actor.parameters(), lr=ACTOR_LR)\n",
+    "    critic_optimizer = optim.Adam(critic.parameters(), lr=CRITIC_LR)\n",
+    "    \n",
+    "    episode_rewards = []\n",
+    "    actor_losses = []\n",
+    "    critic_losses = []\n",
+    "    \n",
+    "    print(\"开始 A2C 训练...\")\n",
+    "    for episode in range(NUM_EPISODES):\n",
+    "        state, _ = env.reset()\n",
+    "        episode_reward = 0\n",
+    "        episode_actor_loss = 0\n",
+    "        episode_critic_loss = 0\n",
+    "        steps = 0\n",
+    "        \n",
+    "        while True:\n",
+    "            state_tensor = torch.FloatTensor(state).unsqueeze(0).to(device)\n",
+    "            action_probs = actor(state_tensor)\n",
+    "            action = distributions.Categorical(action_probs).sample().item()\n",
+    "            \n",
+    "            next_state, reward, terminated, truncated, _ = env.step(action)\n",
+    "            done = terminated or truncated\n",
+    "            \n",
+    "            actor_loss, critic_loss = train_step_a2c(actor, critic, actor_optimizer, critic_optimizer, \n",
+    "                           state, action, reward, next_state, done)\n",
+    "            \n",
+    "            episode_actor_loss += actor_loss\n",
+    "            episode_critic_loss += critic_loss\n",
+    "            steps += 1\n",
+    "            \n",
+    "            state = next_state\n",
+    "            episode_reward += reward\n",
+    "            \n",
+    "            if done:\n",
+    "                break\n",
+    "        \n",
+    "        episode_rewards.append(episode_reward)\n",
+    "        actor_losses.append(episode_actor_loss / steps)\n",
+    "        critic_losses.append(episode_critic_loss / steps)\n",
+    "        \n",
+    "        if (episode + 1) % 100 == 0:\n",
+    "            avg_reward = np.mean(episode_rewards[-100:])\n",
+    "            print(f\"回合 {episode + 1}/{NUM_EPISODES} 完成. 近 100 回合平均 Reward: {avg_reward:.2f}\")\n",
+    "\n",
+    "    env.close()\n",
+    "    plot_a2c_results(episode_rewards, actor_losses, critic_losses)\n",
+    "\n",
+    "if __name__ == \"__main__\":\n",
+    "    main_a2c()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "RL_Env",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.15"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

README.md

@@ -1,56 +1,57 @@
-# 强化学习的数学基础
+# RL-Study
 
-![Python](https://img.shields.io/badge/Python-3.10%2B-blue)
-![Status](https://img.shields.io/badge/Status-Learning-orange)
-![License](https://img.shields.io/badge/License-MIT-green)
+强化学习算法实现与学习笔记，基于赵世钰老师《Mathematical Foundations of Reinforcement Learning》。
 
-本项目是我在学习赵世钰老师书籍 **《Mathematical Foundations of Reinforcement Learning》** 时建立的个人仓库。主要用于记录学习笔记、公式推导，以及使用 Python/Jupyter Notebook 复现书中的算法和 Grid World 示例。
+## 项目结构
 
-## 📚 关于原书 (Original Book)
+```
+RL-Study/
+├── Lecture slides/           # 课程幻灯片
+│   ├── slidesForMyLectureVideos/  # 配套视频课件
+│   └── slidesContinuouslyUpdated/ # 持续更新的课件
+├── Notebooks/               # Jupyter 学习笔记
+│   ├── C1.ipynb ~ C10.ipynb  # 各章节推导与实验
+│   ├── SAC.ipynb            # SAC (Soft Actor-Critic) 算法
+│   └── *_training_results.png  # 训练结果可视化
+├── RawBook/                 # 原书资源
+└── RL_Algothrithms/         # 核心算法实现
+    ├── agents/              # 智能体实现
+    │   ├── a2c.py           # A2C (Advantage Actor-Critic)
+    │   └── qac.py           # QAC (Soft Actor-Critic / Q-Value Actor-Critic)
+    ├── networks.py          # 神经网络定义
+    ├── utils.py             # 工具函数
+    └── main.py              # 训练入口
+```
 
-本项目的核心内容基于赵世钰老师的开源书籍，以下是原书的相关信息：
+## 已实现算法
+
+| 算法 | 文件 | 说明 |
+|------|------|------|
+| A2C | [a2c.py](RL_Algothrithms/agents/a2c.py) | Advantage Actor-Critic，同步版本 |
+| QAC | [qac.py](RL_Algothrithms/agents/qac.py) | Q-Value Actor-Critic，支持 GPU |
+
+## 环境配置
+
+```bash
+pip install torch numpy matplotlib gymnasium
+```
+
+## 快速开始
+
+```bash
+cd RL_Algothrithms
+python main.py --agent a2c    # 训练 A2C
+python main.py --agent qac     # 训练 QAC
+```
+
+## 关于原书
 
 - **书名**: Mathematical Foundations of Reinforcement Learning
 - **作者**: Shiyu Zhao (Westlake University)
-- **GitHub 仓库**: [Book-Mathematical-Foundation-of-Reinforcement-Learning](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning)
-- **配套视频**:
-  - [Bilibili (中文)](https://space.bilibili.com/2044042934)
-  - [YouTube (English)](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8)
+- **GitHub**: [MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning)
+- **B站**: [赵世钰老师频道](https://space.bilibili.com/2044042934)
+- **YouTube**: [课程列表](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8)
 
----
+## License
 
-## 📂 项目结构
-
-本项目主要包含以下内容：
-
-- `Lecture slides/`: 课程相关的幻灯片。
-- `Notebooks/`: 包含详细推导和实验过程的 Jupyter Notebooks。
-- `RawBook/`: 原书中的相关资源和代码。
-- `.gitignore`: Git 忽略文件，指定不需要提交到版本控制的文件或目录。
-- `README.md`: 项目说明文档。
-- `LICENSE`: 代码的开源协议说明。
-
-## 🛠️ 环境配置
-
-本项目使用 Python 进行开发。推荐使用 `uv` 或 `conda` 管理环境。
-
-```bash
-# 安装依赖 (示例)
-pip install numpy matplotlib jupyter
-```
-
-## 📝 开源协议 (License)
-
-### 关于代码
-
-本项目中由本人编写的复现代码遵循 **MIT License** 开源协议。这意味着你可以自由地使用、修改和分发这些代码，但请保留原作者的版权声明。
-
-### 关于笔记
-
-项目中的学习笔记内容仅供个人学习交流使用。
-
----
-
-## 🤝 致谢
-
-特别感谢赵世钰老师提供的精彩教材和开源资源，帮助我们深入理解强化学习的数学原理。
+MIT License（代码部分）

RL_Algothrithms/agents/a2c.py

@@ -0,0 +1,59 @@
+import torch
+import torch.optim as optim
+import torch.nn.functional as F
+import torch.distributions as distributions
+from networks import Actor, VCritic
+
+class A2CAgent:
+    def __init__(self, state_dim, action_dim, device, actor_lr=0.001, critic_lr=0.002, gamma=0.99):
+        self.device = device
+        self.gamma = gamma
+        
+        # A2C 使用 VCritic
+        self.actor = Actor(state_dim, action_dim).to(self.device)
+        self.critic = VCritic(state_dim).to(self.device)
+        
+        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
+        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=critic_lr)
+
+    def select_action(self, state):
+        with torch.no_grad():
+            state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
+            action_probs = self.actor(state_tensor)
+            action = distributions.Categorical(action_probs).sample().item()
+        return action
+
+    # 注意：A2C 的更新不需要 next_action，但为了与 QAC 的接口统一，此处用 *args 吸收多余参数
+    def update(self, state, action, reward, next_state, next_action, done):
+        state = torch.FloatTensor(state).unsqueeze(0).to(self.device)
+        next_state = torch.FloatTensor(next_state).unsqueeze(0).to(self.device)
+        reward = torch.FloatTensor([reward]).unsqueeze(0).to(self.device)
+        
+        # --- Critic 更新 ---
+        v_value = self.critic(state)
+        next_v_value = self.critic(next_state).detach()
+        
+        td_target = reward + self.gamma * next_v_value * (1 - int(done))
+        # 计算优势函数 (Advantage)
+        advantage = td_target - v_value
+        
+        critic_loss = F.mse_loss(v_value, td_target)
+        
+        self.critic_optimizer.zero_grad()
+        critic_loss.backward()
+        self.critic_optimizer.step()
+        
+        # --- Actor 更新 ---
+        action_probs = self.actor(state)
+        dist = distributions.Categorical(action_probs)
+        log_prob = dist.log_prob(torch.tensor([action]).to(self.device))
+
+        # 计算策略的熵，鼓励探索
+        entropy = dist.entropy()
+        
+        # Actor 梯度上升目标：ln(pi) * Advantage，方差更小
+        actor_loss = -(log_prob * advantage.detach()).mean()- 0.01 * entropy.mean()
+        
+        self.actor_optimizer.zero_grad()
+        actor_loss.backward()
+        self.actor_optimizer.step()

RL_Algothrithms/agents/qac.py

@@ -0,0 +1,59 @@
+import torch
+import torch.optim as optim
+import torch.nn.functional as F
+import torch.distributions as distributions
+from networks import Actor, QCritic
+
+class QACAgent:
+    def __init__(self, state_dim, action_dim, device, actor_lr=0.001, critic_lr=0.002, gamma=0.99):
+        # 接收设备参数，确保网络挂载在 GPU 或 CPU 上
+        self.device = device
+        self.gamma = gamma
+        
+        # 实例化网络并移动到指定设备
+        self.actor = Actor(state_dim, action_dim).to(self.device)
+        self.critic = QCritic(state_dim, action_dim).to(self.device)
+        
+        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=actor_lr)
+        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=critic_lr)
+
+    def select_action(self, state):
+        # 推理时禁用梯度图计算，加快速度
+        with torch.no_grad():
+            state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
+            action_probs = self.actor(state_tensor)
+            action = distributions.Categorical(action_probs).sample().item()
+        return action
+
+    def update(self, state, action, reward, next_state, next_action, done):
+        # 将数据转换为张量并送入 GPU
+        state = torch.FloatTensor(state).unsqueeze(0).to(self.device)
+        next_state = torch.FloatTensor(next_state).unsqueeze(0).to(self.device)
+        reward = torch.FloatTensor([reward]).unsqueeze(0).to(self.device)
+        
+        # --- Critic 更新 ---
+        q_values = self.critic(state)
+        current_q = q_values[0, action]
+        
+        next_q_values = self.critic(next_state).detach()
+        next_q = next_q_values[0, next_action]
+        
+        # 计算 TD 目标
+        td_target = reward + self.gamma * next_q * (1 - int(done))
+        critic_loss = F.mse_loss(current_q, td_target.squeeze())
+        
+        self.critic_optimizer.zero_grad()
+        critic_loss.backward()
+        self.critic_optimizer.step()
+        
+        # --- Actor 更新 ---
+        action_probs = self.actor(state)
+        dist = distributions.Categorical(action_probs)
+        log_prob = dist.log_prob(torch.tensor([action]).to(self.device))
+        
+        # Actor 梯度上升目标：ln(pi) * Q(s, a)
+        actor_loss = -(log_prob * current_q.detach())
+        
+        self.actor_optimizer.zero_grad()
+        actor_loss.backward()
+        self.actor_optimizer.step()

RL_Algothrithms/main.py

@@ -0,0 +1,71 @@
+import gymnasium as gym
+import torch
+from agents.qac import QACAgent
+from agents.a2c import A2CAgent
+from utils import plot_comparison
+
+def train_agent(env_name, agent_class, device, num_episodes=500):
+    """
+    通用的训练循环函数
+    """
+    env = gym.make(env_name)
+    state_dim = env.observation_space.shape[0] # type: ignore
+    action_dim = int(env.action_space.n)       # type: ignore
+    
+    # 实例化传入的算法代理
+    agent = agent_class(state_dim, action_dim, device)
+    
+    rewards_history = []
+    
+    for episode in range(num_episodes):
+        state, _ = env.reset()
+        episode_reward = 0
+        
+        # QAC 属于 Sarsa 类，需要提前采样第一个动作
+        action = agent.select_action(state)
+        
+        while True:
+            next_state, reward, terminated, truncated, _ = env.step(action)
+            done = terminated or truncated
+            
+            # 预采样下一个动作
+            next_action = agent.select_action(next_state)
+            
+            # 统一的接口调用更新
+            agent.update(state, action, reward, next_state, next_action, done)
+            
+            state = next_state
+            action = next_action
+            episode_reward += reward # type: ignore
+            
+            if done:
+                break
+                
+        rewards_history.append(episode_reward)
+        if (episode + 1) % 100 == 0:
+            print(f"[{agent_class.__name__}] 回合 {episode+1}/{num_episodes}, 近100回合均分: {sum(rewards_history[-100:])/100:.2f}")
+            
+    env.close()
+    return rewards_history
+
+if __name__ == "__main__":
+    # 检测 GPU
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    print(f"正在使用的计算设备: {device}")
+    
+    ENV_NAME = 'CartPole-v1'
+    EPISODES = 600
+    
+    print("\n--- 开始训练 QAC ---")
+    qac_rewards = train_agent(ENV_NAME, QACAgent, device, EPISODES)
+    
+    print("\n--- 开始训练 A2C ---")
+    a2c_rewards = train_agent(ENV_NAME, A2CAgent, device, EPISODES)
+    
+    # 收集结果并绘图对比
+    results = {
+        'QAC (High Variance)': qac_rewards,
+        'A2C (Low Variance)': a2c_rewards
+    }
+    
+    plot_comparison(results, window=50, save_path='qac_vs_a2c_gpu.png')

RL_Algothrithms/networks.py

@@ -0,0 +1,43 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+# 策略网络 (离散动作 Actor)
+class Actor(nn.Module):
+    def __init__(self, state_dim, action_dim):
+        super(Actor, self).__init__()
+        # 定义两层隐藏层
+        self.fc1 = nn.Linear(state_dim, 128)
+        self.fc2 = nn.Linear(128, action_dim)
+        
+    def forward(self, state):
+        x = F.relu(self.fc1(state))
+        # 输出动作的概率分布
+        action_probs = F.softmax(self.fc2(x), dim=-1)
+        return action_probs
+
+# Q值网络 (用于 QAC)
+class QCritic(nn.Module):
+    def __init__(self, state_dim, action_dim):
+        super(QCritic, self).__init__()
+        self.fc1 = nn.Linear(state_dim, 128)
+        self.fc2 = nn.Linear(128, action_dim)
+        
+    def forward(self, state):
+        x = F.relu(self.fc1(state))
+        # 输出各个动作的具体价值
+        q_values = self.fc2(x)
+        return q_values
+
+# V值网络 (用于 A2C，由于加入了基线，只需输出状态价值标量)
+class VCritic(nn.Module):
+    def __init__(self, state_dim):
+        super(VCritic, self).__init__()
+        self.fc1 = nn.Linear(state_dim, 128)
+        self.fc2 = nn.Linear(128, 1)
+        
+    def forward(self, state):
+        x = F.relu(self.fc1(state))
+        # 输出当前状态的价值评估
+        v_value = self.fc2(x)
+        return v_value

RL_Algothrithms/utils.py

@@ -0,0 +1,34 @@
+import matplotlib
+# 关键设置：针对 Linux 服务器无 GUI 环境，强制使用 Agg 后端进行纯文件渲染
+matplotlib.use('Agg') 
+import matplotlib.pyplot as plt
+import numpy as np
+
+def plot_comparison(results_dict, window=50, save_path='comparison_result.png'):
+    """
+    绘制并保存算法对比曲线
+    :param results_dict: 字典格式 {'QAC': [奖励列表], 'A2C': [奖励列表]}
+    :param window: 移动平均的窗口大小
+    :param save_path: 图片保存路径
+    """
+    plt.figure(figsize=(10, 6))
+    
+    for algo_name, rewards in results_dict.items():
+        # 绘制原始透明度较低的曲线
+        plt.plot(rewards, alpha=0.3, label=f'{algo_name} (Raw)')
+        
+        # 计算并绘制移动平均曲线，使趋势更平滑
+        if len(rewards) >= window:
+            moving_avg = np.convolve(rewards, np.ones(window)/window, mode='valid')
+            plt.plot(np.arange(window-1, len(rewards)), moving_avg, linewidth=2, label=f'{algo_name} (Avg {window})')
+            
+    plt.xlabel('Episode')
+    plt.ylabel('Total Reward')
+    plt.title('Algorithm Comparison: QAC vs A2C')
+    plt.legend()
+    plt.grid(True, alpha=0.3)
+    
+    # 将图像保存到服务器硬盘
+    plt.savefig(save_path, dpi=300)
+    plt.close()
+    print(f"对比图像已成功保存至: {save_path}")