720 lines
26 KiB
Markdown
720 lines
26 KiB
Markdown
# 神经网络架构 & C++ 在线推理指南
|
||
|
||
## 1. 系统总览
|
||
|
||
整个推理流水线由 **两个神经网络** 串联组成：
|
||
|
||
```
|
||
游戏状态 (OpenSpiel / Botzone)
|
||
│
|
||
├─── 提取牌面 ──→ CardModel ──→ pred_histogram [50] ──┐
|
||
│ ├─ concat → [55]
|
||
└─── 提取局势 ──→ env_features [5] ──┘
|
||
│
|
||
CFRNetwork
|
||
│
|
||
┌────────────┴────────────┐
|
||
│ │
|
||
regret_head [5] policy_head [5]
|
||
│ │
|
||
Regret Matching Softmax(masked)
|
||
│ │
|
||
current_strategy [5] avg_strategy [5]
|
||
```
|
||
|
||
**在线推理只需要 `avg_strategy`** —— 它是经过 Softmax 归一化的动作概率分布，从中采样即可得到最终动作。
|
||
|
||
---
|
||
|
||
## 2. CardModel 架构
|
||
|
||
### 2.1 用途
|
||
|
||
将扑克手牌（2 张底牌 + 0~5 张公共牌）编码为一个 **50 维胜率直方图**，作为 CFRNetwork 的输入之一。
|
||
|
||
### 2.2 网络结构
|
||
|
||
```
|
||
输入:
|
||
x_hole: [batch, 2] int64 — 2 张底牌 ID (0-51)
|
||
x_board: [batch, 5] int64 — 5 张公共牌 ID (0-51, 不足用 52 填充)
|
||
|
||
Embedding: nn.Embedding(53, 64, padding_idx=52)
|
||
- 53 个 token: 52 张牌 + 1 个 PAD(52)
|
||
- padding_idx=52 表示 PAD token 的 embedding 恒为 0
|
||
|
||
编码:
|
||
hole_emb = Embedding(x_hole).sum(dim=1) → [batch, 64]
|
||
board_emb = Embedding(x_board).sum(dim=1) → [batch, 64]
|
||
combined = cat([hole_emb, board_emb]) → [batch, 128]
|
||
|
||
Backbone MLP (每层: Linear → ReLU → LayerNorm):
|
||
128 → 512 → ReLU → LayerNorm(512)
|
||
512 → 512 → ReLU → LayerNorm(512)
|
||
512 → 256 → ReLU → LayerNorm(256)
|
||
|
||
Equity Head (在线推理不需要):
|
||
256 → 32 → ReLU → 1 → Sigmoid → [batch, 1] 标量胜率
|
||
|
||
Histogram Head (在线推理只需要这个):
|
||
256 → 64 → ReLU → 50 → Softmax → [batch, 50] 胜率直方图
|
||
|
||
输出:
|
||
pred_equity: [batch, 1] 胜率标量 (0~1)
|
||
pred_histogram: [batch, 50] 胜率直方图 (和为1) ← 这是 CFRNetwork 的 card_features
|
||
```
|
||
|
||
### 2.3 权重参数明细
|
||
|
||
| 层名 (state_dict key) | 形状 | 参数量 |
|
||
|---|---|---|
|
||
| `embedding.weight` | [53, 64] | 3,392 |
|
||
| `backbone.0.weight` (Linear 128→512) | [512, 128] | 65,536 |
|
||
| `backbone.0.bias` | [512] | 512 |
|
||
| `backbone.2.weight` (LayerNorm 512) | [512] | 512 |
|
||
| `backbone.2.bias` (LayerNorm 512) | [512] | 512 |
|
||
| `backbone.3.weight` (Linear 512→512) | [512, 512] | 262,144 |
|
||
| `backbone.3.bias` | [512] | 512 |
|
||
| `backbone.5.weight` (LayerNorm 512) | [512] | 512 |
|
||
| `backbone.5.bias` (LayerNorm 512) | 512 |
|
||
| `backbone.6.weight` (Linear 512→256) | [256, 512] | 131,072 |
|
||
| `backbone.6.bias` | [256] | 256 |
|
||
| `backbone.8.weight` (LayerNorm 256) | [256] | 256 |
|
||
| `backbone.8.bias` (LayerNorm 256) | 256 |
|
||
| `equity_head.0.weight` (Linear 256→32) | [32, 256] | 8,192 |
|
||
| `equity_head.0.bias` | [32] | 32 |
|
||
| `equity_head.2.weight` (Linear 32→1) | [1, 32] | 32 |
|
||
| `equity_head.2.bias` | [1] | 1 |
|
||
| `histogram_head.0.weight` (Linear 256→64) | [64, 256] | 16,384 |
|
||
| `histogram_head.0.bias` | [64] | 64 |
|
||
| `histogram_head.2.weight` (Linear 64→50) | [50, 64] | 3,200 |
|
||
| `histogram_head.2.bias` | [50] | 50 |
|
||
| **合计** | | **~426,803** |
|
||
|
||
### 2.4 牌面 ID 编码规则
|
||
|
||
```
|
||
card_id = rank * 4 + suit
|
||
|
||
rank: 0=2, 1=3, 2=4, 3=5, 4=6, 5=7, 6=8, 7=9, 8=T, 9=J, 10=Q, 11=K, 12=A
|
||
suit: 0=c(梅花), 1=d(方块), 2=h(红心), 3=s(黑桃)
|
||
|
||
示例: Ac = 12*4+0 = 48, Ks = 11*4+3 = 47, 2c = 0*4+0 = 0
|
||
PAD_TOKEN = 52 (embedding 恒为零向量)
|
||
```
|
||
|
||
---
|
||
|
||
## 3. CFRNetwork 架构
|
||
|
||
### 3.1 用途
|
||
|
||
接受牌面特征 + 局势特征，输出 **5 个动作** 的遗憾值和策略 logits，经 Regret Matching 和 Softmax 得到动作概率分布。
|
||
|
||
### 3.2 网络结构
|
||
|
||
```
|
||
输入:
|
||
card_features: [batch, 50] — CardModel 输出的胜率直方图
|
||
env_features: [batch, 5] — 归一化后的局势特征
|
||
|
||
拼接:
|
||
x = cat([card_features, env_features]) → [batch, 55]
|
||
|
||
Backbone MLP (每层: Linear → ReLU):
|
||
55 → 256 → ReLU
|
||
256 → 256 → ReLU
|
||
256 → 128 → ReLU
|
||
|
||
Regret Head (在线推理可跳过):
|
||
128 → 5 (无激活函数, regret 可为负数)
|
||
|
||
Policy Head:
|
||
128 → 5 (输出 logits, 后续过 Softmax)
|
||
|
||
输出:
|
||
regrets: [batch, 5] 遗憾值原始输出
|
||
policy_logits: [batch, 5] 策略 logits
|
||
```
|
||
|
||
### 3.3 权重参数明细
|
||
|
||
| 层名 (state_dict key) | 形状 | 参数量 |
|
||
|---|---|---|
|
||
| `backbone.0.weight` (Linear 55→256) | [256, 55] | 14,080 |
|
||
| `backbone.0.bias` | [256] | 256 |
|
||
| `backbone.2.weight` (Linear 256→256) | [256, 256] | 65,536 |
|
||
| `backbone.2.bias` | [256] | 256 |
|
||
| `backbone.4.weight` (Linear 256→128) | [128, 256] | 32,768 |
|
||
| `backbone.4.bias` | [128] | 128 |
|
||
| `regret_head.weight` (Linear 128→5) | [5, 128] | 640 |
|
||
| `regret_head.bias` | [5] | 5 |
|
||
| `policy_head.weight` (Linear 128→5) | [5, 128] | 640 |
|
||
| `policy_head.bias` | [5] | 5 |
|
||
| **合计** | | **~114,314** |
|
||
|
||
> 注意: `backbone` 是 `nn.Sequential`，索引 0,2,4 是 Linear 层，索引 1,3,5 是 ReLU（无参数）。
|
||
|
||
### 3.4 动作空间
|
||
|
||
| 索引 | 名称 | 含义 |
|
||
|---|---|---|
|
||
| 0 | FOLD | 弃牌 |
|
||
| 1 | CALL | 跟注/过牌 |
|
||
| 2 | HALF_POT | 加注 = 1/2 跟注后底池 |
|
||
| 3 | FULL_POT | 加注 = 1.0 跟注后底池 |
|
||
| 4 | ALL_IN | 全押 |
|
||
|
||
---
|
||
|
||
## 4. 推理流水线详解
|
||
|
||
### 4.1 输入特征构造
|
||
|
||
#### 4.1.1 env_features [5] 构造
|
||
|
||
```
|
||
env_features = [
|
||
pot / 20000.0, // 底池归一化
|
||
p0_stack / 20000.0, // 玩家0剩余筹码归一化
|
||
p1_stack / 20000.0, // 玩家1剩余筹码归一化
|
||
street / 3.0, // 轮次归一化 (0=Preflop, 1=Flop, 2=Turn, 3=River)
|
||
position, // 当前行动玩家 (0.0 或 1.0)
|
||
]
|
||
```
|
||
|
||
#### 4.1.2 card_features [50] 构造
|
||
|
||
```
|
||
1. 提取当前玩家的 2 张底牌 ID → hole_cards[2]
|
||
2. 提取公共牌 ID (0~5张) → board_cards[0..5]
|
||
3. 公共牌不足 5 张时用 PAD_TOKEN=52 填充 → x_board[5]
|
||
4. 送入 CardModel:
|
||
- hole_emb = Embedding(hole_cards).sum(行向量求和) → [64]
|
||
- board_emb = Embedding(x_board).sum(行向量求和) → [64]
|
||
- combined = cat([hole_emb, board_emb]) → [128]
|
||
- features = Backbone(combined) → [256]
|
||
- histogram = HistogramHead(features) → [50]
|
||
5. card_features = histogram (50维胜率直方图)
|
||
```
|
||
|
||
#### 4.1.3 legal_mask [5] 构造
|
||
|
||
```
|
||
legal_mask = [fold_ok, call_ok, half_pot_ok, full_pot_ok, allin_ok]
|
||
每个元素为 0 或 1，标识该 CFR 动作是否合法。
|
||
|
||
构造规则:
|
||
FOLD(0): 引擎合法动作包含 action 0
|
||
CALL(1): 引擎合法动作包含 action 1
|
||
ALL_IN(4): 引擎存在 >1 的加注动作
|
||
HALF_POT(2)/FULL_POT(3): 需同时满足:
|
||
- 存在加注动作
|
||
- 当前 street 加注次数 < 2 (RAISE_CAP)
|
||
- 计算目标贡献额 >= 最小加注额
|
||
- 目标贡献额映射后不等于 ALL-IN 动作
|
||
```
|
||
|
||
### 4.2 从 policy_logits 到动作概率 (avg_strategy)
|
||
|
||
这是在线推理的核心逻辑，只需使用 `policy_head` 的输出：
|
||
|
||
```
|
||
1. logits = policy_head(backbone_output) → [5]
|
||
2. masked_logits = logits
|
||
对 legal_mask[i]==0 的位置, 令 masked_logits[i] = -1e9
|
||
3. avg_strategy = softmax(masked_logits) → [5]
|
||
所有合法动作概率 > 0, 非法动作概率 ≈ 0
|
||
4. 从 avg_strategy 的合法动作中采样
|
||
```
|
||
|
||
### 4.3 降噪 (可选但推荐)
|
||
|
||
训练代码中的实践经验：将概率低于 3% 的动作直接置零后重新归一化，防止神经网络底噪导致异常 All-in。
|
||
|
||
---
|
||
|
||
## 5. C++ 实现指南
|
||
|
||
### 5.1 推荐方案: 手写前向传播 + 加载权重
|
||
|
||
由于网络结构简单（纯 MLP，无卷积/注意力），**不需要 LibTorch**，直接用 Eigen 或手写矩阵乘法即可，部署体积小、推理快。
|
||
|
||
### 5.2 权重文件格式
|
||
|
||
PyTorch 的 `.pt` 文件本质是 Python pickle 序列化的 dict。C++ 直接读取比较麻烦，推荐两步转换：
|
||
|
||
**Step 1: Python 导出为二进制**
|
||
|
||
```python
|
||
import torch
|
||
import struct
|
||
|
||
def export_weights_bin(state_dict, output_path):
|
||
"""将 state_dict 导出为 C++ 可直接读取的二进制文件。"""
|
||
with open(output_path, 'wb') as f:
|
||
# 写入张量数量
|
||
f.write(struct.pack('I', len(state_dict)))
|
||
|
||
for name, tensor in state_dict.items():
|
||
# 写入名字长度 + 名字
|
||
name_bytes = name.encode('utf-8')
|
||
f.write(struct.pack('I', len(name_bytes)))
|
||
f.write(name_bytes)
|
||
|
||
# 写入维度数量
|
||
shape = tensor.shape
|
||
f.write(struct.pack('I', len(shape)))
|
||
|
||
# 写入每个维度大小
|
||
for dim in shape:
|
||
f.write(struct.pack('I', dim))
|
||
|
||
# 写入数据 (float32)
|
||
data = tensor.float().numpy().tobytes()
|
||
f.write(data)
|
||
|
||
# 导出 CardModel 权重
|
||
card_sd = torch.load("card_model/data/best_card_model.pt", map_location="cpu", weights_only=False)
|
||
if "model_state_dict" in card_sd:
|
||
card_sd = card_sd["model_state_dict"]
|
||
export_weights_bin(card_sd, "weights/card_model.bin")
|
||
|
||
# 导出 CFRNetwork 权重
|
||
cfr_sd = torch.load("botzone_cfr_net.pt", map_location="cpu", weights_only=False)
|
||
if "model_state_dict" in cfr_sd:
|
||
cfr_sd = cfr_sd["model_state_dict"]
|
||
export_weights_bin(cfr_sd, "weights/cfr_net.bin")
|
||
```
|
||
|
||
**Step 2: C++ 读取二进制权重**
|
||
|
||
```cpp
|
||
struct Tensor {
|
||
std::string name;
|
||
std::vector<int> shape;
|
||
std::vector<float> data;
|
||
};
|
||
|
||
std::unordered_map<std::string, Tensor> load_weights(const std::string& path) {
|
||
std::unordered_map<std::string, Tensor> weights;
|
||
std::ifstream f(path, std::ios::binary);
|
||
|
||
uint32_t num_tensors;
|
||
f.read((char*)&num_tensors, 4);
|
||
|
||
for (uint32_t i = 0; i < num_tensors; i++) {
|
||
Tensor t;
|
||
uint32_t name_len;
|
||
f.read((char*)&name_len, 4);
|
||
t.name.resize(name_len);
|
||
f.read(t.name.data(), name_len);
|
||
|
||
uint32_t ndim;
|
||
f.read((char*)&ndim, 4);
|
||
t.shape.resize(ndim);
|
||
for (uint32_t d = 0; d < ndim; d++)
|
||
f.read((char*)&t.shape[d], 4);
|
||
|
||
int total = 1;
|
||
for (int d : t.shape) total *= d;
|
||
t.data.resize(total);
|
||
f.read((char*)t.data.data(), total * sizeof(float));
|
||
|
||
weights[t.name] = std::move(t);
|
||
}
|
||
return weights;
|
||
}
|
||
```
|
||
|
||
### 5.3 C++ 前向传播实现
|
||
|
||
#### 5.3.1 基础算子
|
||
|
||
```cpp
|
||
// 矩阵乘法: y = W * x + b (W: [out, in], x: [in], b: [out])
|
||
void linear(const float* W, const float* b, const float* x,
|
||
float* y, int in_dim, int out_dim) {
|
||
for (int i = 0; i < out_dim; i++) {
|
||
float sum = b[i];
|
||
for (int j = 0; j < in_dim; j++) {
|
||
sum += W[i * in_dim + j] * x[j];
|
||
}
|
||
y[i] = sum;
|
||
}
|
||
}
|
||
|
||
// ReLU
|
||
void relu(float* x, int dim) {
|
||
for (int i = 0; i < dim; i++)
|
||
x[i] = std::max(0.0f, x[i]);
|
||
}
|
||
|
||
// LayerNorm: y = (x - mean) / sqrt(var + eps) * gamma + beta
|
||
void layer_norm(const float* gamma, const float* beta,
|
||
const float* x, float* y, int dim, float eps = 1e-5f) {
|
||
float mean = 0.0f;
|
||
for (int i = 0; i < dim; i++) mean += x[i];
|
||
mean /= dim;
|
||
|
||
float var = 0.0f;
|
||
for (int i = 0; i < dim; i++) var += (x[i] - mean) * (x[i] - mean);
|
||
var /= dim;
|
||
|
||
float inv_std = 1.0f / std::sqrt(var + eps);
|
||
for (int i = 0; i < dim; i++)
|
||
y[i] = gamma[i] * (x[i] - mean) * inv_std + beta[i];
|
||
}
|
||
|
||
// Sigmoid
|
||
void sigmoid(float* x, int dim) {
|
||
for (int i = 0; i < dim; i++)
|
||
x[i] = 1.0f / (1.0f + std::exp(-x[i]));
|
||
}
|
||
|
||
// Softmax
|
||
void softmax(float* x, int dim) {
|
||
float max_val = *std::max_element(x, x + dim);
|
||
float sum = 0.0f;
|
||
for (int i = 0; i < dim; i++) {
|
||
x[i] = std::exp(x[i] - max_val);
|
||
sum += x[i];
|
||
}
|
||
for (int i = 0; i < dim; i++) x[i] /= sum;
|
||
}
|
||
```
|
||
|
||
#### 5.3.2 CardModel 前向传播
|
||
|
||
```cpp
|
||
class CardModelInference {
|
||
public:
|
||
// 预分配缓冲区
|
||
float hole_emb[64]; // embedding sum
|
||
float board_emb[64]; // embedding sum
|
||
float combined[128]; // concat
|
||
float backbone_buf[3][512]; // 各隐藏层
|
||
float hist_fc1[64];
|
||
float hist_out[50];
|
||
|
||
// 权重引用 (从 load_weights 获取)
|
||
const float* emb_weight; // [53, 64]
|
||
float backbone_w[3], backbone_b[3]; // Linear 权重/偏置
|
||
float ln_gamma[3], ln_beta[3]; // LayerNorm 参数
|
||
float hist_w1, hist_b1; // 256→64
|
||
float hist_w2, hist_b2; // 64→50
|
||
|
||
void forward(const int* hole_cards, // [2] int, 0-51
|
||
const int* board_cards, // [5] int, 0-51 (不足填52)
|
||
float* histogram) { // 输出 [50]
|
||
|
||
// 1. Embedding lookup + sum
|
||
// hole_emb = emb_weight[hole_cards[0]] + emb_weight[hole_cards[1]]
|
||
memset(hole_emb, 0, 64 * sizeof(float));
|
||
for (int c = 0; c < 2; c++) {
|
||
const float* emb = emb_weight + hole_cards[c] * 64;
|
||
for (int i = 0; i < 64; i++)
|
||
hole_emb[i] += emb[i];
|
||
}
|
||
// board_emb = sum of emb_weight[board_cards[i]], PAD(52) 的 embedding 全为0
|
||
memset(board_emb, 0, 64 * sizeof(float));
|
||
for (int c = 0; c < 5; c++) {
|
||
if (board_cards[c] == 52) continue; // PAD, skip
|
||
const float* emb = emb_weight + board_cards[c] * 64;
|
||
for (int i = 0; i < 64; i++)
|
||
board_emb[i] += emb[i];
|
||
}
|
||
|
||
// 2. Concat [hole_emb | board_emb]
|
||
memcpy(combined, hole_emb, 64 * sizeof(float));
|
||
memcpy(combined + 64, board_emb, 64 * sizeof(float));
|
||
|
||
// 3. Backbone: Linear → ReLU → LayerNorm × 3
|
||
int in_dim = 128;
|
||
int hidden_dims[3] = {512, 512, 256};
|
||
const float* input = combined;
|
||
|
||
for (int layer = 0; layer < 3; layer++) {
|
||
linear(backbone_w[layer], backbone_b[layer],
|
||
input, backbone_buf[layer], in_dim, hidden_dims[layer]);
|
||
relu(backbone_buf[layer], hidden_dims[layer]);
|
||
layer_norm(ln_gamma[layer], ln_beta[layer],
|
||
backbone_buf[layer], backbone_buf[layer], hidden_dims[layer]);
|
||
in_dim = hidden_dims[layer];
|
||
input = backbone_buf[layer];
|
||
}
|
||
|
||
// 4. Histogram head: 256 → 64 (ReLU) → 50 (Softmax)
|
||
linear(hist_w1, hist_b1, backbone_buf[2], hist_fc1, 256, 64);
|
||
relu(hist_fc1, 64);
|
||
linear(hist_w2, hist_b2, hist_fc1, hist_out, 64, 50);
|
||
softmax(hist_out, 50);
|
||
|
||
memcpy(histogram, hist_out, 50 * sizeof(float));
|
||
}
|
||
};
|
||
```
|
||
|
||
#### 5.3.3 CFRNetwork 前向传播 (只走 policy_head)
|
||
|
||
```cpp
|
||
class CFRNetInference {
|
||
public:
|
||
float concat_buf[55]; // card_features[50] + env_features[5]
|
||
float backbone_buf[3]; // 三层隐藏层
|
||
float logits[5]; // policy_head 输出
|
||
float strategy[5]; // 最终动作概率
|
||
|
||
// 权重
|
||
const float* backbone_w[3]; const float* backbone_b[3]; // 三层 Linear
|
||
const float* policy_w; const float* policy_b; // 128→5
|
||
|
||
void forward(const float* card_features, // [50]
|
||
const float* env_features, // [5]
|
||
const int* legal_mask, // [5], 0或1
|
||
float* out_strategy) { // 输出 [5]
|
||
|
||
// 1. Concat
|
||
memcpy(concat_buf, card_features, 50 * sizeof(float));
|
||
memcpy(concat_buf + 50, env_features, 5 * sizeof(float));
|
||
|
||
// 2. Backbone: Linear → ReLU × 3
|
||
int in_dim = 55;
|
||
int hidden_dims[3] = {256, 256, 128};
|
||
const float* input = concat_buf;
|
||
float* output = nullptr;
|
||
|
||
// 需要为每层分配缓冲区, 这里简化表示
|
||
// layer 0: 55 → 256
|
||
float h0[256];
|
||
linear(backbone_w[0], backbone_b[0], input, h0, 55, 256);
|
||
relu(h0, 256);
|
||
|
||
// layer 1: 256 → 256
|
||
float h1[256];
|
||
linear(backbone_w[1], backbone_b[1], h0, h1, 256, 256);
|
||
relu(h1, 256);
|
||
|
||
// layer 2: 256 → 128
|
||
float h2[128];
|
||
linear(backbone_w[2], backbone_b[2], h1, h2, 256, 128);
|
||
relu(h2, 128);
|
||
|
||
// 3. Policy head: 128 → 5
|
||
linear(policy_w, policy_b, h2, logits, 128, 5);
|
||
|
||
// 4. Masked Softmax
|
||
for (int i = 0; i < 5; i++) {
|
||
if (legal_mask[i] == 0)
|
||
logits[i] = -1e9f; // 非法动作设大负数
|
||
}
|
||
softmax(logits, 5);
|
||
|
||
memcpy(out_strategy, logits, 5 * sizeof(float));
|
||
}
|
||
};
|
||
```
|
||
|
||
#### 5.3.4 完整推理流程
|
||
|
||
```cpp
|
||
// === Step 1: 从游戏状态提取信息 ===
|
||
int hole_cards[2] = { /* 底牌 ID 0-51 */ };
|
||
int board_cards[5] = { /* 公共牌 ID, 不足5张用52填充 */ };
|
||
float env_features[5] = {
|
||
pot / 20000.0f,
|
||
p0_stack / 20000.0f,
|
||
p1_stack / 20000.0f,
|
||
street / 3.0f,
|
||
(float)position
|
||
};
|
||
int legal_mask[5] = { /* 由 BetTranslator 逻辑计算 */ };
|
||
|
||
// === Step 2: CardModel 前向传播 ===
|
||
float card_features[50];
|
||
card_model.forward(hole_cards, board_cards, card_features);
|
||
|
||
// === Step 3: CFRNetwork 前向传播 ===
|
||
float strategy[5];
|
||
cfr_net.forward(card_features, env_features, legal_mask, strategy);
|
||
|
||
// === Step 4: 采样 (可选: 先降噪) ===
|
||
// 降噪: 将 <3% 的概率置零后重新归一化
|
||
float threshold = 0.03f;
|
||
float sum = 0.0f;
|
||
for (int i = 0; i < 5; i++) {
|
||
if (legal_mask[i] && strategy[i] < threshold)
|
||
strategy[i] = 0.0f;
|
||
sum += strategy[i];
|
||
}
|
||
if (sum > 0) for (int i = 0; i < 5; i++) strategy[i] /= sum;
|
||
|
||
// 从合法动作中按概率采样
|
||
int chosen = sample_from_distribution(strategy, legal_mask);
|
||
|
||
// === Step 5: CFR 动作 → 引擎动作 (BetTranslator) ===
|
||
int engine_action = cfr_to_engine(state, chosen);
|
||
```
|
||
|
||
### 5.4 权重加载映射表
|
||
|
||
C++ 加载权重时，需要按照 PyTorch `state_dict` 的 key 名映射到对应的层：
|
||
|
||
#### CardModel 权重 key → 用途
|
||
|
||
| state_dict key | 形状 | 用途 |
|
||
|---|---|---|
|
||
| `embedding.weight` | [53, 64] | Embedding 查找表 |
|
||
| `backbone.0.weight` | [512, 128] | Linear 128→512 |
|
||
| `backbone.0.bias` | [512] | |
|
||
| `backbone.2.weight` | [512, 512] | LayerNorm gamma (错! 见下方) |
|
||
| `backbone.2.bias` | [512] | LayerNorm beta |
|
||
| `backbone.3.weight` | [512, 512] | Linear 512→512 |
|
||
| `backbone.3.bias` | [512] | |
|
||
| `backbone.5.weight` | [512, 512] | LayerNorm gamma |
|
||
| `backbone.5.bias` | [512] | LayerNorm beta |
|
||
| `backbone.6.weight` | [256, 512] | Linear 512→256 |
|
||
| `backbone.6.bias` | [256] | |
|
||
| `backbone.8.weight` | [256] | LayerNorm gamma |
|
||
| `backbone.8.bias` | [256] | LayerNorm beta |
|
||
| `histogram_head.0.weight` | [64, 256] | Linear 256→64 |
|
||
| `histogram_head.0.bias` | [64] | |
|
||
| `histogram_head.2.weight` | [50, 64] | Linear 64→50 |
|
||
| `histogram_head.2.bias` | [50] | |
|
||
| `equity_head.*` | - | 在线推理不需要 |
|
||
|
||
> **重要**: CardModel 的 `backbone` 是 `nn.Sequential`，层的索引对应关系为:
|
||
> - `backbone.0` = Linear(128, 512)
|
||
> - `backbone.1` = ReLU (无参数)
|
||
> - `backbone.2` = LayerNorm(512)
|
||
> - `backbone.3` = Linear(512, 512)
|
||
> - `backbone.4` = ReLU (无参数)
|
||
> - `backbone.5` = LayerNorm(512)
|
||
> - `backbone.6` = Linear(512, 256)
|
||
> - `backbone.7` = ReLU (无参数)
|
||
> - `backbone.8` = LayerNorm(256)
|
||
|
||
#### CFRNetwork 权重 key → 用途
|
||
|
||
| state_dict key | 形状 | 用途 |
|
||
|---|---|---|
|
||
| `backbone.0.weight` | [256, 55] | Linear 55→256 |
|
||
| `backbone.0.bias` | [256] | |
|
||
| `backbone.2.weight` | [256, 256] | Linear 256→256 |
|
||
| `backbone.2.bias` | [256] | |
|
||
| `backbone.4.weight` | [128, 256] | Linear 256→128 |
|
||
| `backbone.4.bias` | [128] | |
|
||
| `regret_head.weight` | [5, 128] | 在线推理不需要 |
|
||
| `regret_head.bias` | [5] | 在线推理不需要 |
|
||
| `policy_head.weight` | [5, 128] | Linear 128→5 |
|
||
| `policy_head.bias` | [5] | |
|
||
|
||
> **重要**: CFRNetwork 的 `backbone` 索引:
|
||
> - `backbone.0` = Linear(55, 256)
|
||
> - `backbone.1` = ReLU (无参数)
|
||
> - `backbone.2` = Linear(256, 256)
|
||
> - `backbone.3` = ReLU (无参数)
|
||
> - `backbone.4` = Linear(256, 128)
|
||
> - `backbone.5` = ReLU (无参数)
|
||
|
||
---
|
||
|
||
## 6. 可优化项
|
||
|
||
### 6.1 是否需要 CardModel?
|
||
|
||
**需要**。CardModel 是推理流水线的必要组成部分，它将离散的牌面信息编码为 50 维连续向量，CFRNetwork 依赖这个输入。如果去掉 CardModel，你需要另外设计牌面编码方式，且训练好的 CFRNetwork 权重将无法使用。
|
||
|
||
### 6.2 是否需要 Regret Head?
|
||
|
||
**在线推理不需要**。Regret Head 用于训练时的 Regret Matching，在线推理只使用 Policy Head + Softmax 得到 `avg_strategy`。可以不加载 `regret_head` 的权重以节省内存。
|
||
|
||
### 6.3 是否需要 Equity Head?
|
||
|
||
**在线推理不需要**。Equity Head 只输出标量胜率用于监控，推理时只需要 Histogram Head 的 50 维输出。
|
||
|
||
### 6.4 替代方案: LibTorch
|
||
|
||
如果不想手写前向传播，可以使用 LibTorch (PyTorch C++ API) 直接加载 `.pt` 权重并执行推理。优点是代码量少，缺点是依赖体积大 (~200MB+)。
|
||
|
||
### 6.5 替代方案: ONNX Runtime
|
||
|
||
可以将两个模型导出为 ONNX 格式，用 ONNX Runtime C++ API 推理。兼顾易用性和性能。
|
||
|
||
```python
|
||
# 导出 CardModel 到 ONNX
|
||
card_model = CardModel()
|
||
card_model.load_state_dict(torch.load("card_model/data/best_card_model.pt", map_location="cpu", weights_only=False))
|
||
card_model.eval()
|
||
dummy_hole = torch.randint(0, 52, (1, 2))
|
||
dummy_board = torch.randint(0, 52, (1, 5))
|
||
torch.onnx.export(card_model, (dummy_hole, dummy_board), "card_model.onnx",
|
||
input_names=["x_hole", "x_board"],
|
||
output_names=["pred_equity", "pred_histogram"])
|
||
|
||
# 导出 CFRNetwork 到 ONNX
|
||
cfr_net = CFRNetwork()
|
||
cfr_net.load_state_dict(torch.load("botzone_cfr_net.pt", map_location="cpu", weights_only=False))
|
||
cfr_net.eval()
|
||
dummy_card = torch.randn(1, 50)
|
||
dummy_env = torch.randn(1, 5)
|
||
torch.onnx.export(cfr_net, (dummy_card, dummy_env), "cfr_net.onnx",
|
||
input_names=["card_features", "env_features"],
|
||
output_names=["regrets", "policy_logits"])
|
||
```
|
||
|
||
---
|
||
|
||
## 7. 数据流总结
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────┐
|
||
│ 在线推理数据流 │
|
||
├─────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ 游戏状态 │
|
||
│ ├── hole_cards[2] (int, 0-51) │
|
||
│ ├── board_cards[5] (int, 0-51 or 52=PAD) │
|
||
│ ├── pot, stacks, street, position │
|
||
│ └── legal_actions (引擎原生) │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌─────────────────┐ ┌──────────────────┐ │
|
||
│ │ CardModel │ │ Feature Builder │ │
|
||
│ │ Embedding(53,64)│ │ pot/20000 │ │
|
||
│ │ sum → cat │ │ p0_stack/20000 │ │
|
||
│ │ MLP 128→512→ │ │ p1_stack/20000 │ │
|
||
│ │ 512→256 │ │ street/3.0 │ │
|
||
│ │ +LayerNorm │ │ position (0|1) │ │
|
||
│ │ HistHead 256→ │ └────────┬─────────┘ │
|
||
│ │ 64→50+Sofmax │ │ │
|
||
│ └────────┬────────┘ │ │
|
||
│ │ │ │
|
||
│ card_features[50] env_features[5] │
|
||
│ │ │ │
|
||
│ └──────────┬───────────┘ │
|
||
│ ▼ │
|
||
│ ┌─────────────────────┐ │
|
||
│ │ CFRNetwork │ │
|
||
│ │ cat → [55] │ │
|
||
│ │ MLP 55→256→256→128 │ │
|
||
│ │ PolicyHead 128→5 │ │
|
||
│ └────────┬────────────┘ │
|
||
│ │ │
|
||
│ policy_logits[5] │
|
||
│ │ │
|
||
│ masked_fill(非法→-1e9) │
|
||
│ │ │
|
||
│ Softmax → avg_strategy[5] │
|
||
│ │ │
|
||
│ 降噪 (<3%→0, 重归一化) │
|
||
│ │ │
|
||
│ 按概率采样 → chosen_cfr_idx │
|
||
│ │ │
|
||
│ BetTranslator → engine_action │
|
||
│ │ │
|
||
│ 执行引擎动作 │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────┘
|
||
```
|