97 lines
4.4 KiB
Markdown
97 lines
4.4 KiB
Markdown
# RL-Study
|
||
|
||
强化学习算法实现与学习笔记，基于赵世钰老师《Mathematical Foundations of Reinforcement Learning》。
|
||
|
||
## 项目结构
|
||
|
||
```
|
||
RL-Study/
|
||
├── Lecture slides/ # 课程幻灯片
|
||
│ ├── slidesForMyLectureVideos/ # 配套视频课件
|
||
│ └── slidesContinuouslyUpdated/ # 持续更新的课件
|
||
├── Notebooks/ # Jupyter 学习笔记
|
||
│ ├── C1.ipynb ~ C10.ipynb # 各章节推导与实验
|
||
│ ├── SAC.ipynb # SAC (Soft Actor-Critic) 算法
|
||
│ └── *_training_results.png # 训练结果可视化
|
||
├── RawBook/ # 原书资源
|
||
├── RL_Algothrithms/ # 核心算法实现
|
||
│ ├── agents/ # 智能体实现
|
||
│ │ ├── a2c.py # A2C (Advantage Actor-Critic)
|
||
│ │ ├── qac.py # QAC (Q-Value Actor-Critic)
|
||
│ │ ├── off_pac.py # Off-PAC (Off-Policy Actor-Critic)
|
||
│ │ ├── ddpg.py # DDPG (Deep Deterministic Policy Gradient)
|
||
│ │ └── dpac.py # DPAC (Deterministic Policy Actor-Critic)
|
||
│ ├── networks.py # 离散动作空间网络 (Actor, QCritic, VCritic)
|
||
│ ├── networks_cont.py # 连续动作空间网络 (ContActor, ContQCritic)
|
||
│ ├── utils.py # 工具函数
|
||
│ ├── main.py # 离散动作空间训练入口 (CartPole-v1)
|
||
│ ├── disp_main.py # 离散动作空间多算法对比
|
||
│ └── cont_main.py # 连续动作空间训练入口 (Pendulum-v1)
|
||
└── TRPO/ # TRPO (Trust Region Policy Optimization) 独立实现
|
||
├── models.py # ActorNet (高斯策略), CriticNet (值函数)
|
||
├── utils.py # RolloutBuffer, GAE, 共轭梯度, FVP
|
||
├── agent.py # TRPO 智能体
|
||
└── main.py # TRPO 训练入口 (Pendulum-v1)
|
||
```
|
||
|
||
## 已实现算法
|
||
|
||
### 离散动作空间（CartPole-v1）
|
||
|
||
| 算法 | 文件 | 说明 |
|
||
| ------- | ------------------------------------------------------ | ------------------------------------------------------ |
|
||
| A2C | [agents/a2c.py](RL_Algothrithms/agents/a2c.py) | Advantage Actor-Critic，同步版本，V-critic，带熵正则化 |
|
||
| QAC | [agents/qac.py](RL_Algothrithms/agents/qac.py) | Q-Value Actor-Critic，On-policy SARSA 风格，支持 GPU |
|
||
| Off-PAC | [agents/off_pac.py](RL_Algothrithms/agents/off_pac.py) | Off-Policy Actor-Critic，带重要性采样，epsilon 探索 |
|
||
|
||
### 连续动作空间（Pendulum-v1）
|
||
|
||
| 算法 | 文件 | 说明 |
|
||
| ---- | ------------------------------------------------ | -------------------------------------------------------------- |
|
||
| DDPG | [agents/ddpg.py](RL_Algothrithms/agents/ddpg.py) | Deep Deterministic Policy Gradient，离策略，带目标网络和软更新 |
|
||
| DPAC | [agents/dpac.py](RL_Algothrithms/agents/dpac.py) | Deterministic Policy Actor-Critic，在策略 |
|
||
| TRPO | [agent.py](TRPO/agent.py) | Trust Region Policy Optimization，共轭梯度法 + 线搜索 + GAE |
|
||
|
||
## 环境配置
|
||
|
||
```bash
|
||
pip install torch numpy matplotlib gymnasium
|
||
```
|
||
|
||
## 快速开始
|
||
|
||
### 离散动作空间（CartPole-v1）
|
||
|
||
```bash
|
||
cd RL_Algothrithms
|
||
python main.py --agent a2c # 训练 A2C
|
||
python main.py --agent qac # 训练 QAC
|
||
python disp_main.py # 多算法对比训练
|
||
```
|
||
|
||
### 连续动作空间（Pendulum-v1）
|
||
|
||
```bash
|
||
cd RL_Algothrithms
|
||
python cont_main.py # 训练 DDPG 和 DPAC
|
||
```
|
||
|
||
### TRPO
|
||
|
||
```bash
|
||
cd TRPO
|
||
python main.py # 训练 TRPO
|
||
```
|
||
|
||
## 关于原书
|
||
|
||
- **书名**: Mathematical Foundations of Reinforcement Learning
|
||
- **作者**: Shiyu Zhao (Westlake University)
|
||
- **GitHub**: [MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning](https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning)
|
||
- **B站**: [赵世钰老师频道](https://space.bilibili.com/2044042934)
|
||
- **YouTube**: [课程列表](https://youtube.com/playlist?list=PLEhdbSEZZbDaFWPX4gehhwB9vJZJ1DNm8)
|
||
|
||
## License
|
||
|
||
MIT License（代码部分）
|