376 lines
17 KiB
HTML
376 lines
17 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="zh-CN">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>同伦策略迭代 (Homotopic PI) — Deep NN 实现</title>
|
||
<script>
|
||
window.MathJax = {
|
||
tex: {
|
||
inlineMath: [['$', '$'], ['\\(', '\\)']],
|
||
displayMath: [['$$', '$$'], ['\\[', '\\]']],
|
||
tags: 'ams'
|
||
}
|
||
};
|
||
</script>
|
||
<script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||
<style>
|
||
:root {
|
||
--bg: #fdfdfc;
|
||
--card-bg: #ffffff;
|
||
--text: #2c2c2c;
|
||
--text-secondary: #5a5a5a;
|
||
--accent: #2563eb;
|
||
--accent-light: #eff6ff;
|
||
--border: #e5e7eb;
|
||
--radius: 10px;
|
||
}
|
||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||
body {
|
||
background: var(--bg);
|
||
color: var(--text);
|
||
font-family: 'Segoe UI', 'PingFang SC', 'Microsoft YaHei', 'Noto Sans SC', system-ui, -apple-system, sans-serif;
|
||
line-height: 1.78;
|
||
font-size: 16px;
|
||
-webkit-font-smoothing: antialiased;
|
||
}
|
||
.container { max-width: 820px; margin: 0 auto; padding: 48px 24px 80px; }
|
||
.paper-meta {
|
||
background: var(--card-bg);
|
||
border: 1px solid var(--border);
|
||
border-radius: var(--radius);
|
||
padding: 28px 32px;
|
||
margin-bottom: 40px;
|
||
}
|
||
.paper-meta h1 { font-size: 1.5rem; font-weight: 700; line-height: 1.4; margin-bottom: 6px; color: #111; }
|
||
.paper-meta .sub { font-size: 0.9rem; color: var(--text-secondary); }
|
||
h2 { font-size: 1.22rem; font-weight: 700; margin: 48px 0 16px; padding-bottom: 8px; border-bottom: 2px solid var(--border); color: #111; }
|
||
h3 { font-size: 1.05rem; font-weight: 700; margin: 34px 0 12px; color: #222; }
|
||
p { margin-bottom: 14px; }
|
||
.math-block { overflow-x: auto; padding: 4px 0; }
|
||
code {
|
||
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
|
||
font-size: 0.9em;
|
||
background: #f3f4f6;
|
||
padding: 1px 6px;
|
||
border-radius: 4px;
|
||
color: #1e40af;
|
||
}
|
||
pre {
|
||
background: #1e1e2e; color: #cdd6f4;
|
||
padding: 16px 20px; border-radius: var(--radius);
|
||
overflow-x: auto; font-size: 0.85rem; line-height: 1.6;
|
||
margin: 16px 0;
|
||
}
|
||
blockquote {
|
||
margin: 20px 0;
|
||
padding: 16px 22px;
|
||
border-radius: var(--radius);
|
||
border-left: 4px solid var(--accent);
|
||
background: var(--accent-light);
|
||
font-size: 0.94rem;
|
||
}
|
||
blockquote p { margin-bottom: 8px; }
|
||
blockquote p:last-child { margin-bottom: 0; }
|
||
blockquote strong { color: var(--accent); }
|
||
blockquote.danger { border-left-color: #dc2626; background: #fef2f2; }
|
||
blockquote.danger strong { color: #dc2626; }
|
||
blockquote.definition { border-left-color: #7c3aed; background: #faf5ff; }
|
||
blockquote.definition strong { color: #7c3aed; }
|
||
blockquote.algorithm { border-left-color: #059669; background: #ecfdf5; }
|
||
blockquote.algorithm strong { color: #059669; }
|
||
blockquote.proof { border-left-color: #d97706; background: #fffbeb; }
|
||
blockquote.proof strong { color: #d97706; }
|
||
blockquote.key { border-left-color: #db2777; background: #fdf2f8; }
|
||
blockquote.key strong { color: #db2777; }
|
||
blockquote.impl { border-left-color: #0891b2; background: #ecfeff; }
|
||
blockquote.impl strong { color: #0891b2; }
|
||
.danger-box {
|
||
margin: 18px 0;
|
||
padding: 16px 22px;
|
||
border-radius: var(--radius);
|
||
border: 1px solid #fecaca;
|
||
background: #fef2f2;
|
||
font-size: 0.94rem;
|
||
}
|
||
.danger-box .label { font-weight: 700; font-size: 0.85rem; text-transform: uppercase; letter-spacing: 0.04em; color: #dc2626; margin-bottom: 6px; }
|
||
ol.alg { counter-reset: step; list-style: none; padding-left: 0; margin: 12px 0; }
|
||
ol.alg > li { counter-increment: step; position: relative; padding-left: 42px; margin-bottom: 14px; }
|
||
ol.alg > li::before {
|
||
content: counter(step);
|
||
position: absolute; left: 0; top: -1px;
|
||
width: 28px; height: 28px;
|
||
background: var(--accent); color: #fff;
|
||
font-size: 0.82rem; font-weight: 700;
|
||
border-radius: 50%;
|
||
display: flex; align-items: center; justify-content: center;
|
||
}
|
||
ol.alg ul { margin: 6px 0 0 0; padding-left: 20px; list-style: disc; color: var(--text-secondary); font-size: 0.93rem; }
|
||
ol.alg ul li { margin-bottom: 4px; }
|
||
hr { border: none; border-top: 1px solid var(--border); margin: 40px 0; }
|
||
.footer { margin-top: 56px; padding-top: 20px; border-top: 1px solid var(--border); font-size: 0.82rem; color: #b0b0b0; }
|
||
@media (max-width: 640px) {
|
||
.container { padding: 24px 16px 48px; }
|
||
.paper-meta { padding: 20px; }
|
||
h2 { font-size: 1.12rem; }
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="container">
|
||
|
||
<div class="paper-meta">
|
||
<h1>Adaptive Optimal Control of Unknown Nonlinear Systems via Homotopy-Based Policy Iteration</h1>
|
||
<div class="sub">Chen et al., IEEE Trans. Autom. Control, Vol. 69, No. 5, pp. 3396–3403, May 2024</div>
|
||
<div class="sub" style="margin-top:8px;"><strong>本实现：</strong>Deep NN 函数逼近器 (CriticNN + ActorNN)，梯度下降训练 (Adam)</div>
|
||
</div>
|
||
|
||
<!-- ═══ 一 ═══ -->
|
||
<h2>一、这篇文章解决了什么问题？</h2>
|
||
|
||
<p>用强化学习做最优控制，本质上是在<strong>模型信息不完全</strong>的情况下，通过数据学习一个最优策略。Policy Iteration (PI) 的前提是必须先有一个 <strong>admissible policy</strong> 来启动迭代。但求解 admissible policy 本身需要模型信息——形成循环依赖。</p>
|
||
|
||
<div class="danger-box">
|
||
<div class="label">PI 的初始化困境</div>
|
||
PI 要求初始策略是 admissible 的 → 没有模型就算不出 admissible 策略 → 没有 admissible 策略就无法启动 PI → 无法启动 PI 就学不出最优策略。
|
||
</div>
|
||
|
||
<blockquote class="definition">
|
||
<strong>什么是 Admissible Policy？</strong><br><br>
|
||
给定广义代价函数 $$ J(x_0) = \int_0^\infty \left[ Q(x) + u^T R(x) u \right] dt $$
|
||
称 $u(x)$ 是 <strong>admissible</strong> 的，当且仅当：对任意初始状态 $x_0$，$J(x_0)$ 有限，且系统状态被驱动到原点。
|
||
</blockquote>
|
||
|
||
<hr>
|
||
|
||
<!-- ═══ 二 ═══ -->
|
||
<h2>二、核心思想：同伦 (Homotopy)</h2>
|
||
|
||
<p>对原始系统动态方程做<strong>同伦变换</strong>，人为引入阻尼项：</p>
|
||
|
||
$$ \underbrace{\dot{x} = f(x) + g(x)u}_{\text{原始系统}} \quad\longrightarrow\quad \underbrace{\dot{x} = f(x) - L_i x + g(x)u_i(x)}_{\text{同伦系统}} $$
|
||
|
||
<p>加上 $-L_i x$ 之后，即使 <strong>$u = 0$</strong>（零控制），系统也是稳定的。$u=0$ 天然就是 admissible 的，PI 可以直接启动。然后逐步减小 $L_i$，让同伦系统逼近原始系统。当 $L_i$ 归零时，得到原系统下的 admissible 策略。</p>
|
||
|
||
<blockquote class="key">
|
||
<strong>关键点</strong><br>
|
||
同伦项 $-L_i x$ 是<strong>人工阻尼</strong>让系统自治稳定，不是去「抵消」未知动态 $f,g$。$f,g$ 的未知性通过 off-policy 积分消除 + 深度神经网络逼近来绕过。
|
||
</blockquote>
|
||
|
||
<hr>
|
||
|
||
<!-- ═══ 三 ═══ -->
|
||
<h2>三、数学原理</h2>
|
||
|
||
<h3>3.1 问题设定</h3>
|
||
|
||
$$ \dot{x} = f(x) + g(x)u $$
|
||
|
||
$$ \|f(z)-f(y)\| \le L_f\|z-y\|, \qquad \|g(z)-g(y)\| \le L_g\|z-y\| $$
|
||
|
||
控制目标：极小化 $ J(x_0) = \int_0^\infty \left[ Q(x) + u^T R(x) u \right] dt $
|
||
|
||
<strong>Hamiltonian 方程</strong>：
|
||
$$ 0 = \left( \frac{\partial V}{\partial x} \right)^T \bigl( f(x) + g(x)u \bigr) + Q(x) + u^T R(x) u \tag{5} $$
|
||
|
||
<strong>最优策略解析形式</strong>：
|
||
$$ u^*(x) = -\frac{1}{2} R^{-1}(x) g^T(x) \frac{\partial V(x)}{\partial x} \tag{6} $$
|
||
|
||
<strong>HJB 方程</strong>（$V(0)=0$）：
|
||
$$ 0 = \left( \frac{\partial V}{\partial x} \right)^T f(x) + Q(x) - \frac{1}{4} \left( \frac{\partial V}{\partial x} \right)^T g R^{-1} g^T \frac{\partial V}{\partial x} \tag{7} $$
|
||
|
||
<h3>3.2 传统 PI 方法</h3>
|
||
|
||
<blockquote class="algorithm">
|
||
<strong>Policy Iteration</strong>
|
||
<ol class="alg">
|
||
<li><strong>策略评估</strong><br>给定 $u_i^0$，求解 $V_i^0$（$V_i^0(0)=0$）：
|
||
$$ 0 = \left( \frac{\partial V_i^0}{\partial x} \right)^T \bigl( f + g u_i^0 \bigr) + Q + (u_i^0)^T R u_i^0 \tag{8} $$
|
||
</li>
|
||
<li><strong>策略改进</strong><br>
|
||
$$ u_{i+1}^0(x) = -\frac{1}{2} R^{-1} g^T \frac{\partial V_i^0}{\partial x} \tag{9} $$
|
||
</li>
|
||
</ol>
|
||
</blockquote>
|
||
|
||
<p>PI 保证 $V_{i+1}^0(x) \le V_i^0(x)$，收敛到最优解。前提：<strong>$u_0^0$ 必须是 admissible 的</strong>。</p>
|
||
|
||
<h3>3.3 同伦 PI 方法</h3>
|
||
|
||
<blockquote class="definition">
|
||
<strong>Definition 2 (Lipschitz Admissibility)</strong><br>
|
||
零控制 $u=0$ 是 <strong>Lipschitz admissible</strong> 的，如果存在 $L$ 使得 $\dot{x} = f(x) - Lx + g(x)u$ 渐近稳定。充分条件：$L > L_f$。
|
||
</blockquote>
|
||
|
||
<ul style="margin-bottom:14px; padding-left:22px;">
|
||
<li><strong>策略评估</strong>（同伦 Bellman 方程）：
|
||
$$ 0 = \left( \frac{\partial V_i}{\partial x} \right)^T \bigl[ f - L_i x + g u_i \bigr] + Q + u_i^T R u_i \tag{12} $$
|
||
</li>
|
||
<li><strong>策略改进</strong>：
|
||
$$ u_{i+1}(x) = -\frac{1}{2} R^{-1} g^T \frac{\partial V_i}{\partial x} \tag{13} $$
|
||
</li>
|
||
<li><strong>步长选取</strong>（保证收敛）：
|
||
$$ (1-\gamma)Q\gamma_{i+1} \;\le\; \left( \frac{\partial V_i}{\partial x} \right)^T x \;\le\; (1-\gamma)(Q + u_{i+1}^T R u_{i+1}) + (u_i - u_{i+1})^T R (u_i - u_{i+1}) \tag{14} $$
|
||
</li>
|
||
</ul>
|
||
|
||
<blockquote class="proof">
|
||
<strong>收敛性保证</strong>
|
||
<ol style="margin:0;padding-left:20px;">
|
||
<li>$u_{i+1}$ 在同伦系统下是 admissible 的。</li>
|
||
<li>若 $(\frac{\partial V_i}{\partial x})^T x \ge \beta V_i(x)$，有限步内 $L_i = 0$。</li>
|
||
</ol>
|
||
</blockquote>
|
||
|
||
<hr>
|
||
|
||
<!-- ═══ 四 ═══ -->
|
||
<h2>四、数据驱动的同伦 PI — Deep NN 实现</h2>
|
||
|
||
<h3>4.1 Off-Policy 积分消除</h3>
|
||
|
||
<p>将系统重写为：$\dot{x} = \underbrace{f - L_i x + g u_i}_{\text{同伦系统}} \;+\; L_i x + g v_i \tag{18}$，其中 $v_i = u - u_i$ 是实际执行的 control（PE + 旧策略）与目标策略的偏差。</p>
|
||
|
||
<p>沿轨迹积分消去 $f,g$：</p>
|
||
|
||
$$ V_i(x(t_{k+1})) - V_i(x(t_k)) - \int_{t_k}^{t_{k+1}} \left(\frac{\partial V_i}{\partial x}\right)^T L_i x\,dt
|
||
= -\int_{t_k}^{t_{k+1}} \Bigl[ Q + u_i^T R u_i + 2u_{i+1}^T R v_i \Bigr] dt \tag{20} $$
|
||
|
||
<blockquote class="key">
|
||
<strong>为什么能绕过 $f$ 和 $g$？</strong><br>
|
||
$f,g$ 被 $V_i(x(t_{k+1})) - V_i(x(t_k))$ 隐式替代，等式仅含状态采样和待求函数。
|
||
</blockquote>
|
||
|
||
<h3>4.2 Deep NN 函数逼近</h3>
|
||
|
||
<p><strong>本实现用深度神经网络替代多项式基函数</strong>，彻底规避多项式带来的多重共线性问题（原多项式 cond(A) ~ 1e9，矩阵奇异）。</p>
|
||
|
||
<blockquote class="impl">
|
||
<strong>CriticNN — 值函数 $V(x) \ge 0$</strong><br><br>
|
||
<pre>Linear(2, 32, bias=False) → Tanh
|
||
Linear(32, 32, bias=False) → Tanh
|
||
Linear(32, 2, bias=False) # 原始输出 z₁, z₂
|
||
V(x) = z₁(x)² + z₂(x)² # 结构保证 V≥0, V(0)=0</pre>
|
||
<code>bias=False</code> 全程保证 $V(0)=0$（结构保证）。两个输出头使原点 Hessian 可达 rank-2。
|
||
</blockquote>
|
||
|
||
<blockquote class="impl">
|
||
<strong>ActorNN — 策略 $u(x): \mathbb{R}^2 \to \mathbb{R}$</strong><br><br>
|
||
<pre>Linear(2, 32, bias=True) → Tanh
|
||
Linear(32, 32, bias=True) → Tanh
|
||
Linear(32, 1, bias=True) # 有符号控制输出</pre>
|
||
</blockquote>
|
||
|
||
<h3>4.3 解耦损失函数（梯度下降替代矩阵求逆）</h3>
|
||
|
||
<blockquote class="impl">
|
||
<strong>Critic Loss — Bellman 残差</strong><br>
|
||
$$ \text{Loss}_C = \mathbb{E}\left[ \left( \Delta V_k - \textstyle\int \nabla V \cdot (Lx)dt + \textstyle\int (Q + R\hat{u}^2)dt + \textstyle\int 2R\hat{u}_{\text{new}} v\,dt \right)^2 \right] $$
|
||
Actor 输出被 <code>.detach()</code>，梯度仅流向 Critic。
|
||
</blockquote>
|
||
|
||
<blockquote class="impl">
|
||
<strong>Actor Loss — 最优策略回归</strong><br>
|
||
$$ \text{Loss}_A = \mathbb{E}\left[ \left( u_{\text{NN}}(x) - \left(-\frac{1}{2}R^{-1}g^T(x)\nabla V(x)\right) \right)^2 \right] $$
|
||
对倒立摆 ($J=1, R=1$)：$u^* = -0.5 \cdot \partial V / \partial x_2$<br>
|
||
Critic 梯度被 <code>.detach()</code>，梯度仅流向 Actor。两个网络各有独立 Adam 优化器。
|
||
</blockquote>
|
||
|
||
<blockquote class="key">
|
||
<strong>为什么解耦？</strong><br>
|
||
原论文用线性系统 $A\theta = b$ 联合求解 Critic 和 Actor 权重。对于 NN，联合训练 Bellman 残差会导致 Actor 梯度通过积分项传播，噪声大且不稳定。解耦后 Actor 直接学习最优控制律 $u^* = -(1/2)R^{-1}g^T\nabla V$（策略改进公式 13），训练稳定且收敛快。
|
||
</blockquote>
|
||
|
||
<h3>4.4 算法三阶段</h3>
|
||
|
||
<blockquote class="algorithm">
|
||
<strong>Algorithm 1：完整执行流程</strong>
|
||
<ol class="alg">
|
||
<li>
|
||
<strong>Phase One — 寻找 L₀</strong>
|
||
<ul>
|
||
<li>用 $u =$ PE 信号（零策略）采集一条轨迹</li>
|
||
<li>扫描所有 $L \in [L_{\text{start}}, L_{\text{max}}]$，每个 L 训练独立 trainer</li>
|
||
<li>选取 <strong>Bellman 残差最低</strong> 的 L（而非第一个正定的 L）</li>
|
||
<li>Critic 必须通过正定性检验：Hessian at origin PD + $V(x) \ge 0$ on trajectory</li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<strong>Phase Two — 同伦收缩至 $L=0$</strong>
|
||
<ul>
|
||
<li>用当前策略 + PE 采集新轨迹</li>
|
||
<li>解耦训练：Critic 极小化 Bellman 残差，Actor 回归最优控制律</li>
|
||
<li>用 safety constraint (14) 计算步长 $\alpha$，$L \leftarrow \max(0, L - \alpha)$</li>
|
||
<li>PD 不通过 → 回退到 best weights；连续拒绝 > 15 次 → 提前终止</li>
|
||
</ul>
|
||
</li>
|
||
<li>
|
||
<strong>Phase Three — 标准 PI（$L=0$）</strong>
|
||
<ul>
|
||
<li>设 $L = 0$，用 Phase Two 策略作初始</li>
|
||
<li>交替 Critic 评估 + Actor 改进，直到 loss 收敛</li>
|
||
<li>输出 $u^*(x)$ 和 $V^*(x)$</li>
|
||
</ul>
|
||
</li>
|
||
</ol>
|
||
</blockquote>
|
||
|
||
<hr>
|
||
|
||
<!-- ═══ 五 ═══ -->
|
||
<h2>五、实现架构</h2>
|
||
|
||
<table style="width:100%; border-collapse:collapse; margin:16px 0; font-size:0.93rem;">
|
||
<tr style="background:#f3f4f6;">
|
||
<th style="padding:8px 12px; text-align:left; border:1px solid var(--border);">模块</th>
|
||
<th style="padding:8px 12px; text-align:left; border:1px solid var(--border);">文件</th>
|
||
<th style="padding:8px 12px; text-align:left; border:1px solid var(--border);">职责</th>
|
||
</tr>
|
||
<tr>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><strong>NN 模型</strong></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><code>hpi/nn_models.py</code></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);">CriticNN, ActorNN, compute_q, check_positive_definite, check_lyapunov_decrease</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><strong>训练器</strong></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><code>hpi/nn_trainer.py</code></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);">NNTrainer: 解耦 Critic/Actor 损失 + Adam 优化</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><strong>控制器</strong></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><code>hpi/hpi_controller.py</code></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);">三阶段 HPI 主循环 + safety constraint</td>
|
||
</tr>
|
||
<tr>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><strong>数据采集</strong></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);"><code>hpi/data_collector.py</code></td>
|
||
<td style="padding:8px 12px; border:1px solid var(--border);">倒立摆仿真 + PE 信号生成</td>
|
||
</tr>
|
||
</table>
|
||
|
||
<hr>
|
||
|
||
<!-- ═══ 六 ═══ -->
|
||
<h2>六、总结</h2>
|
||
|
||
<h3>本文贡献</h3>
|
||
$$ \text{同伦阻尼 } L_i x \;\xrightarrow{\text{使 }u=0\text{ 稳定}}\; \text{Lipschitz admissible} \;\xrightarrow{\text{逐步归零 }L_i}\; \text{admissible} \;\xrightarrow{\text{传统 PI}}\; \text{最优策略} $$
|
||
|
||
<h3>本实现的改进</h3>
|
||
<ol style="padding-left:22px; margin-bottom:14px;">
|
||
<li><strong>Deep NN 替代多项式基函数</strong>：消除多重共线性 (cond(A) ~ 1e9)，系统矩阵不再奇异</li>
|
||
<li><strong>梯度下降替代矩阵求逆</strong>：Adam 优化 Bellman 残差 + 策略回归，规避病态线性系统</li>
|
||
<li><strong>解耦 Actor-Critic 训练</strong>：Actor 直接回归 $u^* = -1/(2R)g^T\nabla V$，比联合求解更稳定</li>
|
||
<li><strong>Phase One 全扫描</strong>：扫描所有 L 取最低 loss，而非第一个正定的 L</li>
|
||
</ol>
|
||
|
||
<div class="footer">
|
||
笔记整理自 Chen et al., "Adaptive Optimal Control of Unknown Nonlinear Systems via Homotopy-Based Policy Iteration," IEEE TAC, 2024.<br>
|
||
本实现使用 Deep NN (PyTorch) 替代多项式基函数，梯度下降替代矩阵求逆。<br>
|
||
HPI 在所有测试初始状态下均稳定倒立摆，性能优于 LQR 基线。
|
||
</div>
|
||
|
||
</div>
|
||
</body>
|
||
</html>
|