Compare commits

..

11 Commits

Author SHA1 Message Date
e2hang
48252e04d3 README: clone URL 改为本仓库 (huajishe/e2hang/Mortal-Rating) 2026-07-16 15:58:50 +08:00
e2hang
9f4ff8fdc6 无头化改造 + 移除 Turnstile 模拟点击 + 跨平台支持
- 删除 uc_gui_click_captcha / _poke_captcha,Turnstile 改为无头自动放行
- _wait_for_turnstile_token 改为纯轮询,新增 _acquire_turnstile_token 做 JS 重置 + 重开页面恢复
- headless 默认 True,--headless 改用 BooleanOptionalAction(--no-headless 调试)
- 新增 proxy: "direct" 配置,通过 chromium_arg --proxy-server=direct:// 绕过系统代理
  (mjai.ekyu.moe Turnstile 风控机房代理 IP,需直连)
- install_uc_driver.py 平台感知:Linux/Mac/Win 选对应 chromedriver 包和二进制名
- .gitignore 加 .DS_Store/venv/pytest_cache/个人配置
- README 加 Fedora 支持、proxy direct、headless 默认说明
- docs/headless-turnstile-troubleshooting.md 排查记录
2026-07-16 15:46:25 +08:00
Vesper
ca252c2981 feat: support KillerDucky report parsing 2026-07-13 21:31:20 +08:00
Vesper
4a4fd0ae60 feat: improve interactive analysis charts 2026-07-13 20:56:25 +08:00
Vesper
e987d0876a refactor: use riichi terminology in reports 2026-07-13 17:30:49 +08:00
Vesper
fa4653f028 feat: redesign analysis reports 2026-07-13 17:21:44 +08:00
Vesper
6e78f1a13e test: cover Tenhou edge cases 2026-07-13 17:03:41 +08:00
Vesper
5307913a01 feat: group results by data source 2026-07-13 17:00:50 +08:00
Vesper
8ba2887f46 docs: recommend configuration file usage 2026-07-13 16:57:01 +08:00
Solas Vesper
3d16f81c88 Merge pull request #1 from myouo/agent/add-tenhou-source
Add Tenhou log source and source modes
2026-07-13 16:53:00 +08:00
Vesper
4127a4c618 feat: add Tenhou log source 2026-07-13 16:48:45 +08:00
17 changed files with 3663 additions and 588 deletions

31
.gitignore vendored
View File

@@ -1,7 +1,30 @@
# 运行产物
results/
__pycache__/
*.pyc
*.log
downloaded_files/
latest_logs/
batchmortal_bench_*/
batchmortal_bench_*/
# Python
__pycache__/
*.pyc
*.pyo
.venv/
venv/
env/
.pytest_cache/
*.egg-info/
build/
dist/
# 个人配置(含玩家昵称等私有信息,不提交)
config.yaml
config.yml
config.toml
sstpp.yaml
# 编辑器 / 系统
.DS_Store
.idea/
.vscode/
*.swp
*.swo

239
README.md
View File

@@ -1,169 +1,166 @@
# Batch Mortal Analysis
`batchmortal` 是一个基于 Python 和 SeleniumBase 的批量牌谱分析脚本。它会`amae-koromo` 取雀魂对局记录,再自动打开 `mjai.ekyu.moe` 提交牌谱、等待分析完成,并把结果导出为 `xlsx``csv`
`batchmortal` 是一个批量牌谱分析脚本`amae-koromo` 取雀魂对局,或从 `nodocchi.moe` 获取天凤四人半庄对局,再通过 SeleniumBase 将牌谱提交到 `mjai.ekyu.moe`,最后导出 CSV/XLSX 结果与可选图表
## 环境要求
- Python 3.8+
- Python 3.10+
- Google Chrome
- 可访问目标站点的网络环境
- 能够访问所选数据源及 `mjai.ekyu.moe` 的网络环境
安装依赖:
支持 macOS、Linux含 Fedora和 Windows。无头模式下不需要图形界面也不需要任何系统辅助功能/屏幕录制权限——Turnstile 通过浏览器指纹自动放行不再有模拟点击。Fedora 安装 Chrome`sudo dnf install google-chrome-stable`(需先启用第三方仓库)或从官网下载 rpm。
安装:
```bash
git clone https://github.com/myouo/batchmortal.git
cd batchmortal
git clone https://huajishe.fun/git/e2hang/Mortal-Rating.git
cd Mortal-Rating
pip install -r requirements.txt
```
## 基本用法
## 推荐:使用配置文件
项目提供完整示例配置 [`config.example.yaml`](config.example.yaml)。先编辑其中的 `mode` 和对应玩家 ID再运行
```bash
python main.py -p <玩家昵称> [选项]
# 或者
python main.py -a <数字账号ID> [选项]
python main.py --config config.example.yaml
```
### 使用配置文件(推荐)
建议先开启 `dry_run: true`,确认玩家、模式和提取出的牌谱链接正确;确认后改回 `false` 进行 Mortal 分析。
你可以通过配置文件来简化命令行输入。在项目根目录下创建一个 `config.yaml``config.toml` 文件(或参考 [`config.default.yaml`](config.default.yaml))。
示例 `config.yaml`
### 数据源配置
```yaml
nickname: "言乾"
limit: 10
modes: "12"
review_language: "zh-CN"
headless: true
save_screenshot: true
save_local_paipu: false
output: "xlsx"
# 同时只能选择一个数据源mj/0 为雀魂th/1 为天凤
mode: "th"
# mode: mj 时只读取这一段
mj:
nickname: ""
# account_id: 12345678
limit: 10
modes: "12"
# mode: th 时只读取这一段
th:
nickname: ""
limit: 10
modes: "4p-south"
```
带配置文件的运行方式
```bash
python main.py
```
*(你也可以通过 `python main.py --config my_config.yaml` 手动指定配置文件位置。命令行传入的参数会覆盖配置文件中的同名设置。)*
`mj:``th:` 可以同时保存两套参数,但每次运行只读取顶层 `mode` 选中的一段
常见示例(不使用配置文件):
- `mode`:数据源开关,推荐填写 `mj``th`;也兼容 `0``1`
- `modes`:当前数据源内部的对局类型筛选,与顶层 `mode` 含义不同;天凤固定使用 `4p-south`
- 雀魂可使用 `nickname`,也可使用数字 `account_id`;天凤必须使用 `nickname`
- `limit` 按实际对局模式限制获取数量。
### 对局模式
| 数据源 | `modes` 示例 | 说明 |
| :--- | :--- | :--- |
| 雀魂 `mj` | `9,12,16` | amae-koromo 的数字模式 ID例如 9 四人金南、12 四人玉南、16 四人王座南 |
| 天凤 `th` | `4p-south` | 仅接受四人半庄(四麻南场)牌谱 |
天凤的 `4p-south` 是本项目根据 Nodocchi 返回的 `playernum``playlength` 生成的统一筛选名,并非 Nodocchi API 直接返回的字符串。三麻和东风牌谱不在本项目的天凤分析范围内。
### 常用公共配置
完整字段和注释请直接查看 [`config.example.yaml`](config.example.yaml),常用字段包括:
| 配置项 | 作用 |
| :--- | :--- |
| `review_language` | 分析页面语言:`zh-CN`, `en`, `ja`, `ko` |
| `review_ui` | 结果页样式:`classic``killerducky`;默认保持 `classic` 兼容性 |
| `model_tag` | Mortal 模型版本 |
| `headless` | 是否无界面运行浏览器,默认 `true`;调试时可设 `false` |
| `proxy` | 代理 URL设为 `"direct"` 强制浏览器直连、绕过系统代理mjai.ekyu.moe 的 Turnstile 在机房代理 IP 下会拒绝自动放行,需直连) |
| `dry_run` | 只提取并打印牌谱链接,不启动浏览器 |
| `retry` | 单条牌谱失败后的重试次数 |
| `analyze_bad_move_rate` | 是否统计 5%/10% 两档恶手率 |
| `save_screenshot` | 是否保存分析结果截图 |
| `save_local_paipu` | 是否保存 Mortal 结果页 HTML |
| `output` | `csv``xlsx` |
| `plot` | `none`, `html`, `png`, `both` |
| `plot_limit` | 图表只使用最近 N 条结果;不填表示全部 |
## 命令行覆盖
配置文件是推荐入口;临时参数可以在命令行中覆盖:
```bash
python main.py -p 言乾 --modes 12 --limit 1 --headless --save-screenshot --output xlsx
# 使用配置文件,但临时切换到天凤并只提取链接
python main.py --config config.example.yaml --mode th -p ププリン --modes 4p-south --limit 10 --dry-run
# 使用配置文件,但临时分析指定雀魂玩家
python main.py --config config.example.yaml --mode mj -p 言乾 --modes 12 --limit 10
```
## 参数说明
主要参数:
### Target Options (目标参数)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `-p`, `--player` | 无 | 目标雀魂玩家昵称(必须提供 `-p``-a` |
| `-a`, `--account-id`| 无 | 直接指定玩家数字账号 ID。当与 `-p` 同时提供时,实际拉取以 `-a` 为准,但程序的目标名称仍沿用 `-p` 提供的昵称。若仅提供 `-a`,程序会通过 API 尝试获取真实昵称。 |
| 参数 | 说明 |
| :--- | :--- |
| `--config` | 指定 YAML/TOML 配置文件 |
| `--mode` | 唯一数据源:`mj`/`0``th`/`1` |
| `-p`, `--player` | 当前数据源的玩家昵称 |
| `-a`, `--account-id` | 雀魂数字账号 ID天凤不支持 |
| `--modes` | 逗号分隔的对局模式 |
| `--limit` | 每个实际模式最多获取的记录数 |
| `--review-ui` | 结果页样式:`classic``killerducky` |
| `--dry-run` | 只打印牌谱 URL |
| `--headless` / `--no-headless` | 强制开启/关闭无头模式(默认开) |
| `--proxy` | 代理 URL`direct` 强制直连绕过系统代理 |
| `--badmove` | 开启恶手率统计 |
| `--save-local` | 保存 Mortal 结果页 HTML |
| `--save-screenshot` | 保存结果截图 |
| `--plot` | 生成 HTML/PNG 图表 |
### Analysis Options (分析参数)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `--limit` | `10` | 每个 mode 最多拉取多少条记录 |
| `--modes` | `9` | 逗号分隔的 mode 列表,例如 `9四人金南,12四人玉南,16四人王座南` |
| `--model-tag` | `4.1b` | Mortal 分析模型版本 |
| `--review-language`, `--lang` | `zh-CN` | 跑谱页面语言,会写入 `mjai.ekyu.moe` 表单字段 `select[name="lang"]`。可选 `zh-CN`, `en`, `ja`, `ko` |
| `--retry` | `3` | 失败条目的重试次数。每次重试都会重新打开分析页并重新提交 |
| `--badmove` | `False` | 额外统计恶手率。开启后会在日志和结果文件中写入 5% 与 10% 两档恶手率 |
KillerDucky 页面将 Rating 和 AI 一致率显示在 About 中。项目实际从该页面引用的
`/report/*.json` 结构化数据读取这些字段;开启 `analyze_bad_move_rate` 后,也会根据每个
决策的 `actual_index` 与实际选择概率计算 5%/10% 恶手率。两种 UI 的恶手率口径一致。
### Browser & Network Options (浏览器与网络)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `--headless` | `False` | 后台无界面运行浏览器(强烈推荐) |
| `--proxy` | 系统代理 | 指定浏览器代理;不传时尝试自动读取系统代理 |
旧参数 `--source majsoul|tenhou` 仍可兼容使用,但不能和 `--mode` 同时出现;新配置统一推荐 `mode: mj|th`
### Output Options (输出与绘图)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `--output` | `xlsx` | 导出格式,可选 `xlsx``csv`默认xlsx |
| `--plot` | `none` | 生成折线图:`none`, `html`, `png`, 或 `both`。默认不生成 |
| `--save-screenshot` | `False` | 保存分析结果页面截图(举报时可用) |
| `--save-local` | `False` | 保存每条 Mortal 分析结果页的本地 HTML并在结果文件中记录路径 |
### Advanced Submission Options (高级配置)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `--unsafe-parallel-review` | `False` | 允许并发提交 review。理论上更快但在单代理环境下通常易失败已弃用不推荐 |
| `--submit-interval` | `6` | 受控模式下,两次提交之间的最小间隔秒数(已弃用,不推荐) |
| `--submit-cooldown` | `30` | 受控模式下,连续失败后的冷却秒数(已弃用,不推荐) |
| `--prewarm-standby` | `False` | 实验功能。使用两个持久窗口轮流接力,每次只让当前焦点窗口完整处理一条任务(推荐尝试) |
### General / Legacy Options (通用及历史参数)
| 参数 | 默认值 | 说明 |
| :--- | :--- | :--- |
| `--config` | 无 | 指定配置文件路径(支持 yaml/toml命令行参数会覆盖配置项 |
| `--dry-run` | `False` | 只拉取并打印牌谱 URL不启动浏览器 |
| `--no-manual-verification` | `False` | 兼容旧脚本保留参数,当前无作用(已弃用) |
| `--flare-url` | 无 | 兼容旧脚本保留参数,当前无作用(已弃用) |
## 运行模式建议
- 默认模式:单窗口串行,当前最稳。
- `--prewarm-standby`:实验功能。现在表示“双窗口轮转”,不是后台抢跑预热;在单代理环境下不保证比默认串行更快,但值得一试。
- `--unsafe-parallel-review`:不推荐在单系统代理环境下使用,通常会增加 Cloudflare/Turnstile 等待和失败率。
推荐先从默认模式开始:
```bash
python main.py -p 言乾 --limit 10 --modes 16 --headless
```
如果你要测试实验性的双窗口轮转:
注意:这是实验路径,建议只在你已经拿到默认串行基线后再做对比测试。
```bash
python main.py -p 言乾 --limit 10 --modes 16 --headless --prewarm-standby
```
## 绘图模块 (--plot)
使用 `--plot html``both` 将会在产出数据后调用底层的可视化引擎生成 `report_<nickname>.html`
示例:
```bash
python main.py -p "main()" --limit 20 --modes 16 --headless --retry 3 --badmove --prewarm-standby --save-screenshot --plot both
```
![report_main__](assets/report_main__.png)
## 浏览器提交模式
- 默认:单个持久浏览器串行处理,使用提交间隔和失败冷却,稳定性最好。
- `prewarm_standby: true`:使用两个持久窗口轮流处理任务;仍是受控提交,不保证更快。
- `unsafe_parallel_review: true`:绕过受控提交协调,不代表真正的多线程并发,可能更容易触发 Turnstile 或限流。
- `submit_interval`:受控模式下两次提交的最小间隔秒数。
- `submit_cooldown`:连续提交失败后的冷却秒数。
## 输出目录
结果默认写入
雀魂与天凤结果按来源并列保存
```text
results/<nickname>/
results/
├── majsoul/<nickname>/
└── tenhou/<nickname>/
```
常见产物包括
常见文件
- `results/<nickname>/results.xlsx``results/<nickname>/results.csv`
- `results/<nickname>/mode_<id>/<uuid>.png`
- `results/<nickname>/mode_<id>/<uuid>_error.png`
- `results/<nickname>/mode_<id>/<uuid>.html`
- `results.xlsx``results.csv`
- `mode_<id>/<uuid>.png`
- `mode_<id>/<uuid>_error.png`
- `mode_<id>/<uuid>.html`
- `report_<nickname>.html` / `report_<nickname>.png`
## 日志
导出结果包含 `source` 字段,用于标识 `majsoul``tenhou`。天凤模式目录示例为 `mode_4p-south`
运行日志会在每条输出前附带当前系统时间,便于定位慢点和错误发生时刻
可视化报告包含关键指标卡、Rating 单半庄值与半庄移动平均、按决策数加权的 AI 一致率、Rating 分布和低 Rating 牌谱检讨入口。缺失指标显示为 `—`,不会按 0 计入图表或汇总
## 说明
从旧版本升级时,请将原有的 `results/<nickname>/` 雀魂目录移动到 `results/majsoul/<nickname>/`;否则程序无法从新目录识别以前已经处理的牌谱。
- `xlsx` 写入逻辑已做批量化优化但整体耗时通常主要由浏览器提交、Cloudflare Turnstile 和远端分析生成决定。
- `--badmove` 只会在新分析的牌谱中统计恶手率;已经成功写入结果文件的牌谱会继续跳过,不会为了补恶手率自动重跑。
- `--save-local` 只会保存新分析的结果页;已经成功写入结果文件的牌谱会继续跳过,不会为了补本地 HTML 自动重跑。
- 在只有一个系统代理的情况下,多窗口或多线程通常不会线性提速。
## 注意事项
## 下一步
- 计划增加常见 id 与`match_id`的对应词典,以满足大家喜爱的主播/牌手改名而自己不愿意去翻的情景,如:{"火龙果说电影" : "8888621"}(主播名称与`account_id`的对应) 以便大家使用
- anything in issue...?
- Nodocchi 返回的 `tw` 是压缩座位排列;脚本会解码目标玩家视角,并且只接受 `tenhou.net` 正式牌谱 URL。
- Nodocchi 中没有 `url``tw` 的历史统计记录不会进入 Mortal 分析队列。
- 已成功写入结果文件的牌谱会跳过;失败记录仍可在后续运行中重试
- `--badmove`、本地 HTML 和截图只会应用于新执行的分析,不会自动重跑已经成功的牌谱。
- 总耗时通常取决于浏览器提交、Cloudflare Turnstile 和远端分析生成速度。
- Turnstile 通过无头浏览器指纹自动放行,不做模拟点击、不需要系统辅助权限;少数情况会卡住,脚本会先重置控件再重开页面,仍失败才走 `retry`。机房代理出口 IP 会被 Turnstile 触发交互挑战,需用 `proxy: "direct"` 走直连。
## License

View File

@@ -9,6 +9,7 @@ from seleniumbase import SB
REVIEW_BASE_URL = "https://mjai.ekyu.moe"
DEFAULT_REVIEW_LANGUAGE = "zh-CN"
DEFAULT_REVIEW_UI = "classic"
# Mirrors the reviewer form field `select[name="lang"]`.
# Supported values on mjai.ekyu.moe are:
@@ -45,6 +46,153 @@ REPORT_URL_FRAGMENT = "/report/"
BAD_MOVE_STRICT_LIMIT = 5
BAD_MOVE_LOOSE_LIMIT = 10
REVIEW_UI_ALIASES = {
"classic": "classic",
"killerducky": "killerducky",
"killer-ducky": "killerducky",
"kd": "killerducky",
}
def normalize_review_ui(review_ui):
if review_ui is None:
return DEFAULT_REVIEW_UI
key = str(review_ui).strip().lower().replace("_", "-")
normalized = REVIEW_UI_ALIASES.get(key)
if normalized:
return normalized
raise ValueError(
f"Unsupported review UI '{review_ui}'. Supported values: classic, killerducky"
)
PROXY_DIRECT = "direct"
def build_browser_kwargs(headless: bool, proxy: str | None) -> dict:
"""Translate headless/proxy into ``SB()`` kwargs.
A proxy value of ``"direct"`` (case-insensitive) forces Chrome to bypass
every proxy — including the macOS system proxy that Clash/etc. configure —
by passing ``--proxy-server=direct://``. This is required for mjai.ekyu.moe:
Cloudflare Turnstile flags datacenter proxy exit IPs and refuses to
auto-issue a token, so the browser must reach the site from the residential
IP. SeleniumBase's ``proxy=`` param cannot express this because its
``validate_proxy_string`` rejects ``direct://`` and falls back to None,
which leaves Chrome on the system proxy.
"""
kwargs = {"uc": True, "headless": headless}
if proxy and str(proxy).strip().lower() == PROXY_DIRECT:
kwargs["chromium_arg"] = "--proxy-server=direct://"
elif proxy:
kwargs["proxy"] = proxy
return kwargs
def parse_killerducky_metadata(data):
"""Convert KillerDucky's report JSON into the metadata shape used by results.py."""
data = data if isinstance(data, dict) else {}
review = data.get("review")
review = review if isinstance(review, dict) else {}
total_matches = review.get("total_matches")
total_reviewed = review.get("total_reviewed")
matches_total = ""
if (
isinstance(total_matches, (int, float))
and isinstance(total_reviewed, (int, float))
and total_reviewed > 0
):
rate = 100 * total_matches / total_reviewed
matches_total = f"{int(total_matches)}/{int(total_reviewed)} = {rate:.3f}%"
rating = review.get("rating")
formatted_rating = ""
if isinstance(rating, (int, float)):
formatted_rating = f"{rating * 100:.3f}"
def text(value):
return "" if value is None else str(value)
return {
"engine": text(data.get("engine")),
"model tag": text(review.get("model_tag")),
"rating": formatted_rating,
"matches/total": matches_total,
"temperature": text(review.get("temperature")),
"game length": text(data.get("game_length")),
"player id": text(data.get("player_id")),
"review duration": text(data.get("review_time")),
}
def parse_killerducky_bad_move_stats(
data,
strict_limit=BAD_MOVE_STRICT_LIMIT,
loose_limit=BAD_MOVE_LOOSE_LIMIT,
):
"""Calculate bad-move rates from KillerDucky's structured Mortal decisions."""
data = data if isinstance(data, dict) else {}
review = data.get("review")
review = review if isinstance(review, dict) else {}
total_reviewed = review.get("total_reviewed")
denominator = total_reviewed if isinstance(total_reviewed, int) and total_reviewed >= 0 else None
strict_count = 0
loose_count = 0
mismatch_count = 0
unparsed_count = 0
kyokus = review.get("kyokus")
for kyoku in kyokus if isinstance(kyokus, list) else []:
if not isinstance(kyoku, dict):
continue
entries = kyoku.get("entries")
for entry in entries if isinstance(entries, list) else []:
if not isinstance(entry, dict) or entry.get("is_equal") is not False:
continue
mismatch_count += 1
actual_index = entry.get("actual_index")
details = entry.get("details")
if (
not isinstance(actual_index, int)
or not isinstance(details, list)
or actual_index < 0
or actual_index >= len(details)
or not isinstance(details[actual_index], dict)
):
unparsed_count += 1
continue
probability = details[actual_index].get("prob")
if not isinstance(probability, (int, float)):
unparsed_count += 1
continue
probability_percent = probability * 100
if probability_percent <= strict_limit:
strict_count += 1
if probability_percent <= loose_limit:
loose_count += 1
def format_rate(count):
if denominator is None or denominator <= 0:
return ""
return f"{100 * count / denominator:.3f}%"
return {
"badMoveRate5": format_rate(strict_count),
"badMoveCount5": str(strict_count),
"badMoveRate10": format_rate(loose_count),
"badMoveCount10": str(loose_count),
"badMoveDenominator": "" if denominator is None else str(denominator),
"badMoveOrderLossCount": str(mismatch_count),
"badMoveUnparsedCount": str(unparsed_count),
}
def normalize_review_language(language):
if language is None:
@@ -178,10 +326,12 @@ class BrowserAutomator:
submission_coordinator=None,
controlled_submission=True,
review_language=DEFAULT_REVIEW_LANGUAGE,
review_ui=DEFAULT_REVIEW_UI,
):
self.headless = headless
self.proxy = proxy
self.review_language = normalize_review_language(review_language)
self.review_ui = normalize_review_ui(review_ui)
self.review_url = build_review_url(self.review_language)
self.controlled_submission = controlled_submission
if controlled_submission:
@@ -195,7 +345,7 @@ class BrowserAutomator:
break
try:
with SB(uc=True, headless=self.headless, proxy=self.proxy) as sb:
with SB(**build_browser_kwargs(self.headless, self.proxy)) as sb:
tasks_processed = 0
while True:
@@ -259,7 +409,7 @@ class BrowserAutomator:
def iter_alternating_windows(self, tasks, max_retries=3):
pending = deque(tasks)
with SB(uc=True, headless=self.headless, proxy=self.proxy) as sb:
with SB(**build_browser_kwargs(self.headless, self.proxy)) as sb:
slots = [
{
"name": "window-a",
@@ -306,7 +456,7 @@ class BrowserAutomator:
def iter_dual_window_pipeline(self, tasks, max_retries=3):
pending = deque(tasks)
with SB(uc=True, headless=self.headless, proxy=self.proxy) as sb:
with SB(**build_browser_kwargs(self.headless, self.proxy)) as sb:
active_slot = {
"name": "active",
"handle": sb.driver.current_window_handle,
@@ -419,10 +569,9 @@ class BrowserAutomator:
else:
logging.info(f"{log_prefix} Unthrottled mode, starting Turnstile")
self._prepare_review_form(sb)
self._poke_captcha(sb)
token_started_at = time.perf_counter()
self._wait_for_turnstile_token(sb, log_prefix, timeout=35)
self._acquire_turnstile_token(sb, task, log_prefix)
token_wait_seconds = time.perf_counter() - token_started_at
logging.info(f"{log_prefix} Turnstile token ready in {token_wait_seconds:.1f}s")
@@ -568,7 +717,7 @@ class BrowserAutomator:
logging.info(f"{log_prefix} {slot['name']} tab entering Turnstile")
token_started_at = time.perf_counter()
self._wait_for_turnstile_token(sb, log_prefix, timeout=35)
self._acquire_turnstile_token(sb, task, log_prefix)
token_wait_seconds = time.perf_counter() - token_started_at
logging.info(f"{log_prefix} Turnstile token ready in {token_wait_seconds:.1f}s")
@@ -693,7 +842,6 @@ class BrowserAutomator:
self._open_fresh_review_page(sb, log_prefix)
self._populate_form(sb, task["paipu_url"], task["model_tag"])
self._prepare_review_form(sb)
self._poke_captcha(sb)
def _prime_rotation_slot(self, sb, slot, label):
if not slot.get("handle"):
@@ -774,7 +922,7 @@ class BrowserAutomator:
if "mjai.ekyu.moe" in current_url:
sb.execute_script("window.location.replace(arguments[0]);", self.review_url)
else:
sb.uc_open_with_reconnect(self.review_url, reconnect_time=2)
sb.uc_open_with_reconnect(self.review_url, reconnect_time=4)
except Exception:
sb.open(self.review_url)
@@ -826,6 +974,7 @@ class BrowserAutomator:
const paipuUrl = arguments[0];
const modelTag = arguments[1];
const reviewLanguage = arguments[2];
const reviewUi = arguments[3];
const dispatch = (el) => {
el.dispatchEvent(new Event('input', { bubbles: true }));
@@ -857,7 +1006,7 @@ class BrowserAutomator:
setSelect('select[name="engine"]', 'mortal');
setSelect('select[name="mortal-model-tag"]', modelTag);
setSelect('select[name="ui"]', 'classic');
setSelect('select[name="ui"]', reviewUi);
setSelect('select[name="lang"]', reviewLanguage);
const details = document.querySelector('details.details.mb-3');
@@ -870,7 +1019,7 @@ class BrowserAutomator:
showRating.click();
}
const form = document.querySelector(arguments[3]);
const form = document.querySelector(arguments[4]);
if (form) {
form.target = '_self';
}
@@ -880,6 +1029,7 @@ class BrowserAutomator:
paipu_url,
model_tag,
self.review_language,
self.review_ui,
FORM_SELECTOR,
)
@@ -887,10 +1037,13 @@ class BrowserAutomator:
raise RuntimeError("Could not populate review form")
def _wait_for_turnstile_token(self, sb, log_prefix, timeout):
deadline = time.time() + timeout
next_poke_at = time.time() + 8
recoveries = 0
"""Pure poller: returns when cf-turnstile-response has a token.
No GUI clicks, no widget reset, no page reload. Recovery is handled by
the caller (_acquire_turnstile_token) so this method stays a simple
timeout-bound poll.
"""
deadline = time.time() + timeout
while time.time() < deadline:
state = self._read_review_state(sb)
if state["token_length"] > 0:
@@ -902,19 +1055,65 @@ class BrowserAutomator:
):
raise RuntimeError(f"{log_prefix} Turnstile token was rejected before submission")
if time.time() >= next_poke_at:
recoveries += 1
logging.info(f"{log_prefix} Turnstile token still missing, retrying captcha click")
self._recover_turnstile_widget(sb)
self._poke_captcha(sb)
if recoveries >= 2:
raise RuntimeError(f"{log_prefix} Turnstile widget stalled before token issuance")
next_poke_at = time.time() + 8
time.sleep(0.5)
raise RuntimeError(f"{log_prefix} Timed out waiting for Turnstile token")
def _acquire_turnstile_token(
self,
sb,
task,
log_prefix,
per_attempt_timeout: float = 12.0,
max_reloads: int = 2,
):
"""Obtain a Turnstile token via headless UC-mode auto-pass.
No GUI clicks and no OS permissions are needed. On stall we first try a
cheap in-page JS reset of the Turnstile widget, then fall back to fully
reloading the review form (up to ``max_reloads`` times) to give
Cloudflare a fresh chance to auto-issue a token. Server-side rejections
(invalid/timeout-or-duplicate) are propagated without retry.
"""
js_reset_done = False
reloads = 0
while True:
try:
self._wait_for_turnstile_token(sb, log_prefix, per_attempt_timeout)
return
except RuntimeError as exc:
msg = str(exc).lower()
if "rejected" in msg or "rate" in msg:
raise
if not js_reset_done:
js_reset_done = True
logging.info(f"{log_prefix} Turnstile token missing, resetting widget")
try:
self._recover_turnstile_widget(sb)
except Exception:
pass
continue
if reloads >= max_reloads:
raise RuntimeError(
f"{log_prefix} Turnstile widget stalled before token issuance"
) from exc
reloads += 1
logging.info(
f"{log_prefix} Turnstile token still missing, "
f"reloading form ({reloads}/{max_reloads})"
)
try:
self._open_fresh_review_page(sb, log_prefix)
self._populate_form(sb, task["paipu_url"], task["model_tag"])
self._prepare_review_form(sb)
except Exception as reload_exc:
logging.warning(f"{log_prefix} Form reload failed: {reload_exc}")
def _submit_review(self, sb, log_prefix):
submitted = sb.execute_script(
"""
@@ -971,6 +1170,12 @@ class BrowserAutomator:
raise RuntimeError(f"{log_prefix} Review submission never left the form page")
def _extract_metadata(self, sb):
killerducky_data = self._extract_killerducky_data(sb)
if killerducky_data:
return parse_killerducky_metadata(killerducky_data)
if self.review_ui == "killerducky":
raise RuntimeError("Could not extract KillerDucky report JSON")
metadata = sb.execute_script(
"""
const data = {};
@@ -987,6 +1192,56 @@ class BrowserAutomator:
)
return metadata or {}
def _extract_killerducky_data(self, sb, include_entries=False):
try:
data = sb.execute_script(
"""
if (!(window.MM && window.MM.GS && window.MM.GS.fullData)) {
return null;
}
const source = window.MM.GS.fullData;
const review = source.review || {};
const includeEntries = arguments[0];
return {
engine: source.engine,
game_length: source.game_length,
review_time: source.review_time,
player_id: source.player_id,
review: {
model_tag: review.model_tag,
rating: review.rating,
temperature: review.temperature,
total_matches: review.total_matches,
total_reviewed: review.total_reviewed,
kyokus: includeEntries && Array.isArray(review.kyokus)
? review.kyokus.map((kyoku) => ({
entries: Array.isArray(kyoku.entries) ? kyoku.entries.map((entry) => {
const actualIndex = entry.actual_index;
const details = Array.isArray(entry.details)
? entry.details.map((detail, index) => (
index === actualIndex ? {prob: detail.prob} : null
))
: null;
return {
is_equal: entry.is_equal,
actual_index: actualIndex,
details,
};
}) : [],
}))
: [],
},
};
""",
include_entries,
)
return data if isinstance(data, dict) else None
except Exception as exc:
if self.review_ui == "killerducky":
raise RuntimeError("Could not read KillerDucky report JSON") from exc
return None
def _save_local_paipu(self, sb, filepath, source_url, log_prefix):
os.makedirs(os.path.dirname(filepath), exist_ok=True)
html = sb.execute_script(
@@ -1031,6 +1286,10 @@ class BrowserAutomator:
def _extract_bad_move_stats(self, sb, log_prefix):
try:
killerducky_data = self._extract_killerducky_data(sb, include_entries=True)
if killerducky_data:
return parse_killerducky_bad_move_stats(killerducky_data)
stats = sb.execute_script(
"""
const strictLimit = arguments[0];
@@ -1139,7 +1398,22 @@ class BrowserAutomator:
def _wait_for_result_or_error(self, sb, log_prefix, timeout):
deadline = time.time() + timeout
while time.time() < deadline:
if sb.is_element_present(RESULT_SELECTOR):
if self.review_ui == "killerducky":
try:
killerducky_ready = sb.execute_script(
"""
return !!(
window.MM && window.MM.GS && window.MM.GS.fullData
&& window.MM.GS.fullData.review
&& document.querySelector('.about-metadata table')
);
"""
)
except Exception:
killerducky_ready = False
if killerducky_ready:
return
elif sb.is_element_present(RESULT_SELECTOR):
return
state = self._read_review_state(sb)
@@ -1174,6 +1448,18 @@ class BrowserAutomator:
def _expand_metadata_panel(self, sb, log_prefix):
try:
if self.review_ui == "killerducky":
sb.execute_script(
"""
const modal = document.getElementById('about-modal');
if (modal && !modal.open) {
modal.showModal();
}
"""
)
time.sleep(0.5)
return
is_open = sb.execute_script(
"""
const details = document.querySelector('body > details:nth-child(6)');
@@ -1186,12 +1472,6 @@ class BrowserAutomator:
except Exception as exc:
logging.warning(f"{log_prefix} Could not expand metadata menu: {exc}")
def _poke_captcha(self, sb):
try:
sb.uc_gui_click_captcha()
except Exception:
pass
def _recover_turnstile_widget(self, sb):
try:
sb.execute_script(

View File

@@ -1,6 +1,79 @@
import os
import sys
MODE_TO_SOURCE = {
"mj": "majsoul",
"th": "tenhou",
}
SOURCE_MODE_ALIASES = {
"mj": "mj",
"majsoul": "mj",
"mahjong-soul": "mj",
"0": "mj",
"th": "th",
"tenhou": "th",
"1": "th",
}
def normalize_source_mode(value) -> str:
"""Normalize source selectors to the canonical config modes ``mj`` or ``th``."""
if value is None:
return "mj"
key = str(value).strip().lower().replace("_", "-")
mode = SOURCE_MODE_ALIASES.get(key)
if not mode:
raise ValueError(
f"Unsupported source mode '{value}'. Use 'mj' for Mahjong Soul or 'th' for Tenhou."
)
return mode
def source_for_mode(mode) -> str:
return MODE_TO_SOURCE[normalize_source_mode(mode)]
def resolve_mode_config(config: dict, requested_mode=None) -> tuple[str, str, dict]:
"""
Select exactly one source-specific config section.
New configs use ``mode: mj|th`` with ``mj:`` and ``th:`` mappings. The old
top-level ``source: majsoul|tenhou`` format remains readable for backward
compatibility. Defining both selectors is rejected so the active source is
always unambiguous.
"""
config = config or {}
if not isinstance(config, dict):
raise ValueError("Config root must be a mapping/object.")
configured_mode = config.get("mode")
legacy_source = config.get("source")
if configured_mode is not None and legacy_source is not None:
raise ValueError(
"Config defines both 'mode' and legacy 'source'. "
"Keep exactly one selector: 'mode: mj' or 'mode: th'."
)
selector = requested_mode
if selector is None:
selector = configured_mode if configured_mode is not None else legacy_source
mode = normalize_source_mode(selector)
section = config.get(mode, {})
if section is None:
section = {}
if not isinstance(section, dict):
raise ValueError(f"Config section '{mode}' must be a mapping/object.")
effective_config = dict(config)
effective_config.update(section)
if mode == "th" and "modes" not in section and "tenhou_modes" in config:
effective_config["modes"] = config["tenhou_modes"]
return mode, MODE_TO_SOURCE[mode], effective_config
def load_config(config_path: str = None) -> dict:
"""
Load configuration from a YAML or TOML file.

View File

@@ -6,6 +6,7 @@ import openpyxl
CSV_COLUMNS = [
"nickname",
"source",
"mode",
"uuid",
"paipuUrl",
@@ -46,7 +47,7 @@ def parse_metadata(metadata: dict) -> dict:
return value
return ""
ai_consistency = get(["\u4e00\u81f4\u7387", "Match Rate"])
ai_consistency = get(["\u4e00\u81f4\u7387", "Match Rate", "Matches/total"])
numerator, denominator, rate = "", "", ""
# Matches strings like "195/271 = 71.956%".

306
batchmortal/tenhou.py Normal file
View File

@@ -0,0 +1,306 @@
import logging
import re
import urllib.parse
from collections import defaultdict
from datetime import datetime, timezone
import requests
NODOCCHI_API_URL = "https://nodocchi.moe/api/listuser.php"
REQUEST_HEADERS = {"Accept": "application/json"}
SESSION = requests.Session()
TENHOU_MODE_ALIASES = {
"all": "all",
"*": "all",
"4": "4p",
"4p": "4p",
"yonma": "4p",
"四麻": "4p",
"4e": "4p-east",
"4p-east": "4p-east",
"四东": "4p-east",
"四東": "4p-east",
"4s": "4p-south",
"4p-south": "4p-south",
"四南": "4p-south",
"3": "3p",
"3p": "3p",
"sanma": "3p",
"三麻": "3p",
"3e": "3p-east",
"3p-east": "3p-east",
"三东": "3p-east",
"三東": "3p-east",
"3s": "3p-south",
"3p-south": "3p-south",
"三南": "3p-south",
}
TENHOU_MODE_CHOICES = (
"all",
"4p",
"4p-east",
"4p-south",
"3p",
"3p-east",
"3p-south",
)
TENHOU_LOG_ID_RE = re.compile(
r"^\d{10}gm-[0-9a-z]{4}-[0-9a-z]+-[0-9a-z]{8}$",
re.IGNORECASE,
)
def fetch_tenhou_player_records(player_name: str) -> tuple[str, list[dict]]:
"""Fetch a Tenhou player's records from nodocchi.moe."""
try:
response = SESSION.get(
NODOCCHI_API_URL,
params={"name": player_name},
timeout=30,
headers=REQUEST_HEADERS,
)
response.raise_for_status()
data = response.json()
except Exception as exc:
raise RuntimeError(
f"Nodocchi API request failed while searching for '{player_name}': {exc}"
) from exc
if not isinstance(data, dict):
raise ValueError("Unexpected Nodocchi API response: expected an object.")
retry_after = data.get("retry")
if retry_after:
raise RuntimeError(
f"Nodocchi API asked the client to retry after {retry_after} seconds."
)
records = data.get("list")
if not isinstance(records, list):
raise ValueError("Unexpected Nodocchi API response: missing 'list' field.")
if not records:
raise ValueError(f"Tenhou player not found or has no records: '{player_name}'.")
resolved_name = data.get("name")
if not isinstance(resolved_name, str) or not resolved_name:
resolved_name = player_name
logging.info(
"[Nodocchi] Found Tenhou player '%s' with %d records.",
resolved_name,
len(records),
)
return resolved_name, records
def normalize_tenhou_modes(value: str | list[str] | tuple[str, ...]) -> tuple[str, ...]:
if isinstance(value, str):
raw_modes = value.split(",")
else:
raw_modes = value
normalized = []
for raw_mode in raw_modes:
key = str(raw_mode).strip().lower().replace("_", "-")
if not key:
continue
mode = TENHOU_MODE_ALIASES.get(key)
if not mode:
supported = ", ".join(TENHOU_MODE_CHOICES)
raise ValueError(
f"Unsupported Tenhou mode '{raw_mode}'. Supported values: {supported}"
)
if mode not in normalized:
normalized.append(mode)
if not normalized:
return ("all",)
if "all" in normalized:
return ("all",)
return tuple(normalized)
def tenhou_record_mode(record: dict) -> str:
try:
player_num = int(record.get("playernum"))
play_length = int(record.get("playlength"))
except (TypeError, ValueError) as exc:
raise ValueError("Tenhou record is missing a valid playernum/playlength.") from exc
if player_num not in (3, 4) or play_length not in (1, 2):
raise ValueError(
f"Unsupported Tenhou record mode: playernum={player_num}, playlength={play_length}."
)
length_name = {1: "east", 2: "south"}[play_length]
return f"{player_num}p-{length_name}"
def tenhou_mode_matches(record_mode: str, selected_modes: tuple[str, ...]) -> bool:
if "all" in selected_modes:
return True
return any(
record_mode == selected_mode
or (selected_mode in ("3p", "4p") and record_mode.startswith(f"{selected_mode}-"))
for selected_mode in selected_modes
)
def find_player_order(record: dict, player_name: str) -> int | None:
try:
player_num = int(record.get("playernum"))
except (TypeError, ValueError):
return None
for order in range(1, player_num + 1):
if record.get(f"player{order}") == player_name:
return order
return None
def decode_tenhou_viewpoint(encoded_tw, player_order: int | None) -> int | None:
"""
Decode Nodocchi's packed seat permutation.
This mirrors the site's frontend: each two-bit group stores the result
order for one Tenhou seat. The seat whose value equals order - 1 is the
requested player's ``tw`` value.
"""
try:
packed = int(encoded_tw)
except (TypeError, ValueError):
return None
if player_order not in (1, 2, 3, 4):
return None
expected_order = player_order - 1
for seat in range(4):
if (packed >> (seat * 2)) & 3 == expected_order:
return seat
return None
def parse_tenhou_log_url(url: str) -> tuple[str, str] | None:
"""Return ``(log_id, canonical_base_url)`` for an official Tenhou log URL."""
if not isinstance(url, str) or not url:
return None
parsed = urllib.parse.urlsplit(url)
if parsed.scheme not in ("http", "https"):
return None
if (parsed.hostname or "").lower() not in ("tenhou.net", "www.tenhou.net"):
return None
if parsed.path not in ("/0", "/0/"):
return None
log_values = urllib.parse.parse_qs(parsed.query).get("log", [])
if len(log_values) != 1 or not TENHOU_LOG_ID_RE.fullmatch(log_values[0]):
return None
log_id = log_values[0]
canonical_host = (parsed.hostname or "tenhou.net").lower()
canonical_url = urllib.parse.urlunsplit(
(
parsed.scheme,
canonical_host,
"/0/",
urllib.parse.urlencode({"log": log_id}),
"",
)
)
return log_id, canonical_url
def format_tenhou_timestamp(timestamp) -> str:
try:
timestamp = int(timestamp)
except (TypeError, ValueError):
return ""
if timestamp <= 0:
return ""
return (
datetime.fromtimestamp(timestamp, timezone.utc)
.astimezone()
.strftime("%Y-%m-%d %H:%M:%S")
)
def build_tenhou_paipu_urls(
records: list[dict],
player_name: str,
modes: str | list[str] | tuple[str, ...] = "all",
limit: int | None = None,
) -> list[dict]:
"""Build recent, player-focused Tenhou paipu URLs from Nodocchi records."""
selected_modes = normalize_tenhou_modes(modes)
per_mode_count: defaultdict[str, int] = defaultdict(int)
results = []
seen_log_ids = set()
skipped_without_url = 0
skipped_invalid = 0
def start_time(record):
try:
return int(record.get("starttime", 0))
except (TypeError, ValueError):
return 0
for record in sorted(records, key=start_time, reverse=True):
try:
mode = tenhou_record_mode(record)
except ValueError:
skipped_invalid += 1
continue
if not tenhou_mode_matches(mode, selected_modes):
continue
if limit is not None and per_mode_count[mode] >= limit:
continue
raw_url = record.get("url")
if not raw_url:
skipped_without_url += 1
continue
parsed_url = parse_tenhou_log_url(raw_url)
if not parsed_url:
skipped_invalid += 1
continue
player_order = find_player_order(record, player_name)
viewpoint = decode_tenhou_viewpoint(record.get("tw"), player_order)
if viewpoint is None:
skipped_invalid += 1
continue
log_id, base_url = parsed_url
if log_id in seen_log_ids:
continue
start_timestamp = start_time(record)
try:
duration_seconds = max(0, int(record.get("during", 0))) * 60
except (TypeError, ValueError):
duration_seconds = 0
results.append(
{
"source": "tenhou",
"mode": mode,
"uuid": log_id,
"paipuUrl": f"{base_url}&tw={viewpoint}",
"startTime": format_tenhou_timestamp(start_timestamp),
"endTime": format_tenhou_timestamp(start_timestamp + duration_seconds),
}
)
seen_log_ids.add(log_id)
per_mode_count[mode] += 1
logging.info(
"[Nodocchi] Built %d Tenhou links (unavailable=%d, invalid=%d).",
len(results),
skipped_without_url,
skipped_invalid,
)
return results

File diff suppressed because it is too large Load Diff

View File

@@ -3,15 +3,25 @@
# 或者直接在主目录下放置名为 config.yaml 或 config.toml 的文件
# 命令行传入的参数会覆盖这里的配置
# 目标雀魂玩家昵称(必须提供,可以在命令行中通过位置参数指定,也可以在此处指定)
nickname: ""
# 数据源模式,同时只能选择一个:
# mj或 0 -> 雀魂 / amae-koromo
# th或 1 -> 天凤 / nodocchi.moe
mode: "th"
# 每个 mode 最多拉取多少条记录 (默认: 10)
limit: 10
# 雀魂配置;仅在 mode: "mj" 时读取
mj:
nickname: ""
# account_id: 12345678 # 可用数字账号 ID 替代 nickname
limit: 10
# 常见 modes: 9四人金南, 12四人玉南, 16四人王座南
modes: "12"
# 逗号分隔的 mode 列表 (默认: "9")
# 常见 mode: 9四人金南, 12四人玉南, 16四人王座南
modes: "12"
# 天凤配置;仅在 mode: "th" 时读取
th:
nickname: ""
limit: 10
# 天凤仅接受四人半庄(四麻南场),固定使用 4p-south
modes: "4p-south"
# Mortal 分析模型版本 (默认: "4.1b")
model_tag: "4.1b"
@@ -26,8 +36,13 @@ model_tag: "4.1b"
# ko -> 한국어,页面 /ko.html
review_language: "zh-CN"
# 分析结果页面样式:
# killerducky -> KillerDucky 交互界面;从页面引用的结构化 JSON 读取 Rating、AI 一致率和恶手率
# classic -> 经典长页面;保留原有 DOM 解析方式
review_ui: "killerducky"
# 是否额外分析恶手率。默认关闭,开启后会统计 5% 和 10% 两档恶手率
analyze_bad_move_rate: false
analyze_bad_move_rate: true
# 是否在后台以无头模式运行浏览器 (强烈推荐 true)
headless: true
@@ -36,10 +51,10 @@ headless: true
dry_run: false
# 是否保存分析结果页面的截图 (举报时有用)
save_screenshot: true
save_screenshot: false
# 是否保存 Mortal 分析结果页面到本地 HTML
save_local_paipu: false
save_local_paipu: true
# 导出格式,可选 "xlsx" 或 "csv" (默认: "xlsx")
output: "xlsx"
@@ -48,17 +63,21 @@ output: "xlsx"
retry: 3
# 指定浏览器的代理服务器,例如 "http://127.0.0.1:7890"
# 设为 "direct" 可强制浏览器直连、绕过系统代理(含 Clash 设置的系统代理)。
# mjai.ekyu.moe 的 Cloudflare Turnstile 会对机房代理出口 IP 触发交互挑战、
# 拒绝自动放行;走直连(住宅 IP才能在无头下自动拿到 token。
# 如果留空或注释掉,会自动尝试检测系统代理
# proxy: "http://127.0.0.1:7890"
# proxy: "direct"
# 实验性功能:双窗口接力机制 (默认: false)
# 双窗口接力机制 (默认: false)
prewarm_standby: true
# 危险的高并发请求容易导致Cloudflare卡盾拦截 (弃用推荐置空或保留为false)
# 跳过受控提交协调;不代表真正并发,可能更容易触发 Cloudflare/Turnstile
unsafe_parallel_review: false
# 绘图选项,支持 "none", "html", "png", "both" (默认: "none")
plot: "none"
plot: "both"
# 画图时取最近多少条数据 (默认: 全部)
# 设置后仅取最新的 N 条数据用于绘图

View File

@@ -0,0 +1,127 @@
# 无头化与 Turnstile 排查记录
本文记录将 batchmortal 改造为无头模式、移除模拟点击、并解决 Cloudflare Turnstile 自动放行问题的完整排查过程。供后续维护或迁移到其他平台时参考。
## 背景
原实现通过 SeleniumBase 的 `sb.uc_gui_click_captcha()``pyautogui` 物理点击 Cloudflare Turnstile 复选框来获取 token。这条路径有三个硬伤
1. 必须授予 macOS 辅助功能Accessibility+ 屏幕录制Screen Recording权限。
2. 无头模式下没有可点击的 GUI根本无法工作。
3. `pyautogui` 会接管鼠标,运行期间不能并行使用电脑。
目标:无头、无点击、无系统权限依赖。
## 排查过程
### 第一步:确认服务器是否强制校验 Turnstile
直接 `curl -X POST https://mjai.ekyu.moe/review`(不带 token
```
HTTP/2 403
invalid captcha response
```
结论HTTP 层绕不开 Turnstile必须拿到合法 token。表单 POST 到 `/review`token 字段是 `cf-turnstile-response`,由 Cloudflare 的 `turnstile/v0/api.js` 在浏览器端生成。
### 第二步:无头 + 自动放行(初步方案)
删掉所有 `_poke_captcha` / `uc_gui_click_captcha` 调用,把 `_wait_for_turnstile_token` 改成纯轮询器,新增 `_acquire_turnstile_token` 做恢复(先 JS 重置 `window.turnstile.reset()`,再最多 2 次重开页面)。`_open_fresh_review_page``uc_open_with_reconnect``reconnect_time` 从 2 调到 4给 Turnstile JS 更多时间在断开驱动期间自动下发 token。
实跑 `--limit 2`(无头 + 系统代理开着):**100% 失败**。每次都 12s 轮询无 token → JS 重置 → 重开页面 → 仍无 token → stall。6 次重试全挂。
### 第三步:隔离变量——是“无头被识破”还是“代理被风控”?
`--no-headless --limit 1`(有头 + 同一代理):**同样 100% 失败**,模式完全一致。
结论:**不是无头的问题**。有头无头在代理下都挂。最大嫌疑变成代理出口 IP 被 Cloudflare 风控(机房 IP 触发交互挑战,而非 managed 模式自动放行)。
### 第四步:测直连
`curl --noproxy '*' https://mjai.ekyu.moe/` 返回 HTTP 2001.1sCloudflare 边缘 IP。站点直连可达代理并非访问必需。
让用户关掉 Clash 系统代理,跑 `--limit 1` 无头直连:
```
[Proxy] No system proxy detected, running directly.
[1/1][...] Submit slot granted, starting Turnstile
[1/1][...] Turnstile token missing, resetting widget
[1/1][...] Turnstile token still missing, reloading form (1/2)
[1/1][...] Turnstile token ready in 26.8s ← 成功
[1/1][...] OK rating=89.869 match=80.420%
```
**直连下无头自动放行成功**。根因确认:代理出口 IP 被 Turnstile 风控。
### 第五步:自动化直连——`proxy: "direct"`
不能要求用户每次手动关 Clash 系统代理。需要让浏览器强制直连、绕过系统代理。
#### 为什么不能直接用 SeleniumBase 的 `proxy=` 参数
SeleniumBase 的 `validate_proxy_string``seleniumbase/core/proxy_helper.py`)只接受 `ip:port``scheme://host:port` 格式,`"direct://"` 通不过校验,会被置为 `None`。而 `proxy=None` 意味着不传 `--proxy-server` 给 Chrome —— Chrome 就回退到系统代理,反而走回 Clash。所以“不传 proxy”不等于“直连”。
#### 解法:用 `chromium_arg` 注入 `--proxy-server=direct://`
Chrome 的 `--proxy-server=direct://` 会强制忽略所有代理(含系统代理),用 `direct`无代理scheme 直连。SeleniumBase 的 `SB()``chromium_arg` 参数可以塞额外 Chrome 启动参数。
`batchmortal/browser.py` 加模块级 helper
```python
def build_browser_kwargs(headless, proxy):
kwargs = {"uc": True, "headless": headless}
if proxy and str(proxy).strip().lower() == "direct":
kwargs["chromium_arg"] = "--proxy-server=direct://"
elif proxy:
kwargs["proxy"] = proxy
return kwargs
```
`main.py``detect_proxy` 识别 `"direct"` 关键字(大小写不敏感)返回 `"direct"`,所有 `SB()` 调用统一走 `build_browser_kwargs`
#### 验证
Clash 系统代理开着 + `proxy: "direct"` + 无头 + 全量 40 条:**38/38 成功0 失败**。
- 36/38 局 token 秒出0.0~2.1s)—— 浏览器会话热起来后指纹/cookie 被信任
- 2/38 局 stall冷启动 + 中途偶发),都经 JS 重置 + 1 次重开页面恢复,没有任何一局需要外层 `retry`
- 全程零点击、零系统权限弹窗、无头、Clash 开着
## 关键代码位置
- `batchmortal/browser.py:build_browser_kwargs` —— proxy→SB kwargs 翻译,`"direct"``chromium_arg`
- `batchmortal/browser.py:_acquire_turnstile_token` —— 纯轮询 + JS 重置 + 重开页面恢复,无点击
- `batchmortal/browser.py:_wait_for_turnstile_token` —— 纯轮询器(检测 token / 检测 `invalid captcha response` / 超时)
- `main.py:detect_proxy` —— 识别 `"direct"` 关键字
- `install_uc_driver.py:_platform_archive_info` —— 跨平台 driver 包选择
## 平台兼容性Fedora/Linux
核心运行路径跨平台:`detect_proxy``urllib.request.getproxies()`Linux 读环境变量/GSettings`browser.py` 全是 JS + SeleniumBase 调用,文件路径用 `os.path.join``uc_name``os.name` 分流。无头模式不需要 X server。
**唯一需要修的是 `install_uc_driver.py`**:原版硬编码 Windows 的 `chromedriver-win64.zip` / `chromedriver.exe`,在 Linux 上会下载 Windows 二进制。改成按平台选 archive 和二进制名:
| 平台 | archive | 二进制 | uc_driver 名 |
| :--- | :--- | :--- | :--- |
| Windows | `chromedriver-win64.zip` | `chromedriver.exe` | `uc_driver.exe` |
| Linux | `chromedriver-linux64.zip` | `chromedriver` | `uc_driver`(需 `chmod 755` |
| macOS | `chromedriver-mac-{arm64,x64}.zip` | `chromedriver` | `uc_driver` |
Chrome 版本检测也加了 `google-chrome-stable`Fedora/Debian 常见名)和 mac app 路径回退。旧版 Chrome<115在 npmmirror 没有 Linux/Mac 包,返回 None 交给 SeleniumBase 自带下载兜底。
## 常见问题速查
| 症状 | 原因 | 解决 |
| :--- | :--- | :--- |
| Turnstile 每局都 stall、100% 失败 | 浏览器走了机房代理 IP | 配 `proxy: "direct"` 走直连 |
| 直连下偶发 stall~5% | Turnstile 偶尔不出 token | 内置恢复会自动 JS 重置 + 重开页面,无需干预 |
| `proxy: "direct"` 不生效、仍走代理 | Clash 用 TUN 模式(网络层拦截) | 切回系统代理模式,或临时关 ClashTUN 模式下 Chrome 的 `--proxy-server` 绕不过 |
| 无头下 token 一直拿不到 | 先确认是不是代理问题(有头也测一次) | 有头无头都挂 = 代理问题;仅无头挂 = 指纹问题 |
| Fedora 上 uc_driver 不可执行 | 旧版 install_uc_driver 下了 Windows 包 | 已修;或直接删 `seleniumbase/drivers/uc_driver*` 让 SB 自带下载 |
## 注意事项
- **TUN 模式陷阱**:如果 Clash 用 TUN 模式(而非系统代理模式),`--proxy-server=direct://` 绕不过网络层拦截,`proxy: "direct"` 会失效。此时只能切回系统代理模式或关 Clash。判断方法`scutil --proxy` 显示 `HTTPEnable: 0` 但网络仍走代理,多半是 TUN。
- **断点续跑**`get_processed_uuids` 会跳过已成功写入的 UUID所以即使中途网络抖动或被 Turnstile 卡住导致中断,重跑会从未处理的牌谱继续,不会重复分析。
- **不要把 `proxy: "direct"` 和真正的代理混用**`"direct"` 是关键字,不是 URL。配了 `"direct"` 就不会再走任何代理。

View File

@@ -19,28 +19,64 @@ def get_chrome_version():
major_version = int(full_version.split('.')[0])
return major_version, full_version
except Exception:
pass
# Linux / Mac 回退方案
candidates = [
["google-chrome", "--version"],
["google-chrome-stable", "--version"],
["/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", "--version"],
]
for cmd in candidates:
try:
# Linux / Mac (回退方案)
output = subprocess.check_output(["google-chrome", "--version"]).decode("utf-8")
output = subprocess.check_output(cmd).decode("utf-8")
full_version = re.search(r'([\d\.]+)', output).group(1)
major_version = int(full_version.split('.')[0])
return major_version, full_version
except Exception:
return None, None
continue
return None, None
def _platform_archive_info(major_version):
"""根据平台和 Chrome 主版本返回 (archive_name, arch_dir, binary_suffix)。
binary_suffix 为 '.exe' 表示 Windows'' 表示 Linux/Macchromedriver 无后缀)。
"""
is_windows = os.name == 'nt'
is_mac = platform.system() == 'Darwin'
if major_version >= 115:
if is_windows:
is_64bit = platform.architecture()[0] == '64bit'
return ("chromedriver-win64.zip" if is_64bit else "chromedriver-win32.zip",
"win64" if is_64bit else "win32", ".exe")
if is_mac:
is_arm = platform.machine() in ("arm64", "aarch64")
return ("chromedriver-mac-arm64.zip" if is_arm else "chromedriver-mac-x64.zip",
"mac-arm64" if is_arm else "mac-x64", "")
# Linux
return "chromedriver-linux64.zip", "linux64", ""
# Chrome 114 及以下
if is_windows:
return "chromedriver_win32.zip", None, ".exe"
# 旧版 Chrome 在 npmmirror 上只有 win32 包Linux/Mac 无对应旧版包;
# 让上层用 None 触发 "未找到" 路径,交给 SeleniumBase 自带下载兜底
return None, None, ""
def get_npm_mirror_download_url(major_version, full_version):
"""根据 Chrome 版本从国内 npmmirror 获取 ChromeDriver 下载链接"""
archive_name, arch_dir, _ = _platform_archive_info(major_version)
if archive_name is None:
print(f"❌ 错误npmmirror 未提供 Chrome {major_version} 在当前平台的包,跳过(交给 SeleniumBase 自带下载)。")
return None
# Chrome 115 及以上更改了分发路径
if major_version >= 115:
base_url = "https://registry.npmmirror.com/-/binary/chrome-for-testing/"
is_64bit = platform.architecture()[0] == '64bit'
archive_name = "chromedriver-win64.zip" if is_64bit else "chromedriver-win32.zip"
arch_dir = "win64" if is_64bit else "win32"
else:
# Chrome 114 及以下
base_url = "https://registry.npmmirror.com/-/binary/chromedriver/"
archive_name = "chromedriver_win32.zip"
arch_dir = None
try:
print(f"正在向镜像站请求可用版本列表: {base_url}")
@@ -112,28 +148,37 @@ def install_uc_driver():
# 获取当前 Python 环境中 seleniumbase 的 driver 存放目录
sb_drivers_dir = os.path.join(seleniumbase.__path__[0], 'drivers')
os.makedirs(sb_drivers_dir, exist_ok=True)
# 对于 Windows,最终的文件名是 uc_driver.exe
uc_driver_path = os.path.join(sb_drivers_dir, 'uc_driver.exe')
# Windows uc_driver 带后缀Linux/Mac 无后缀
_, _, binary_suffix = _platform_archive_info(major)
uc_driver_name = 'uc_driver.exe' if binary_suffix == '.exe' else 'uc_driver'
uc_driver_path = os.path.join(sb_drivers_dir, uc_driver_name)
# 压缩包内目标二进制名(应对不同版本的目录嵌套)
target_binary = 'chromedriver.exe' if binary_suffix == '.exe' else 'chromedriver'
try:
print("📦 正在解压并配置...")
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
exe_name = None
# 在压缩包中寻找 chromedriver.exe (应对不同版本的目录嵌套)
for name in zip_ref.namelist():
if name.lower().endswith('chromedriver.exe'):
# 取路径最后一段比较,兼容 chromedriver-linux64/chromedriver 这类嵌套
last_segment = name.rstrip('/').split('/')[-1].lower()
if last_segment == target_binary.lower():
exe_name = name
break
if not exe_name:
print("❌ 错误:未在压缩包中找到 chromedriver.exe")
print(f"❌ 错误:未在压缩包中找到 {target_binary}")
return
# 直接提取内容并写入到期望的 uc_driver.exe 文件位置
# 直接提取内容并写入到期望的 uc_driver 文件位置
with zip_ref.open(exe_name) as source, open(uc_driver_path, "wb") as target:
shutil.copyfileobj(source, target)
# Linux/Mac 下需要可执行权限
if binary_suffix != '.exe':
os.chmod(uc_driver_path, 0o755)
print(f"🎉 成功uc_driver.exe 已自动安装至:")
print(f"🎉 成功!{uc_driver_name} 已自动安装至:")
print(f" -> {uc_driver_path}")
print("现在程序可以正常运行了。")
except Exception as e:

214
main.py
View File

@@ -7,11 +7,27 @@ import urllib.request
from datetime import datetime, timezone
from batchmortal.api import build_paipu_urls, get_player_records, search_player, get_player_nickname_by_id
from batchmortal.browser import BrowserAutomator, ReviewSubmissionCoordinator, normalize_review_language
from batchmortal.browser import (
BrowserAutomator,
ReviewSubmissionCoordinator,
build_browser_kwargs,
normalize_review_language,
normalize_review_ui,
)
from batchmortal.results import ResultWriter, parse_metadata, get_processed_uuids, read_result_rows
from batchmortal.tenhou import (
build_tenhou_paipu_urls,
fetch_tenhou_player_records,
normalize_tenhou_modes,
)
from batchmortal.visualize import plot_results
from seleniumbase import SB
from batchmortal.config import load_config
from batchmortal.config import (
load_config,
normalize_source_mode,
resolve_mode_config,
source_for_mode,
)
def configure_logging():
@@ -34,12 +50,29 @@ def parse_review_language(value):
raise argparse.ArgumentTypeError(str(exc)) from exc
def parse_review_ui(value):
try:
return normalize_review_ui(value)
except ValueError as exc:
raise argparse.ArgumentTypeError(str(exc)) from exc
def parse_args():
pre_parser = argparse.ArgumentParser(add_help=False)
pre_parser.add_argument("--config", help="Path to config file (yaml or toml)")
pre_parser.add_argument("--mode")
pre_parser.add_argument("--source", "--platform", dest="legacy_source")
pre_args, _ = pre_parser.parse_known_args()
config = load_config(pre_args.config)
raw_config = load_config(pre_args.config)
requested_mode = pre_args.mode or pre_args.legacy_source
try:
config_mode, _, config = resolve_mode_config(
raw_config,
requested_mode=requested_mode,
)
except ValueError as exc:
pre_parser.error(str(exc))
parser = argparse.ArgumentParser(
description="Batch Mortal Analysis Script (Python/SeleniumBase Edition)",
@@ -48,6 +81,20 @@ def parse_args():
# -- General Options --
parser.add_argument("--config", help="Path to config file (yaml or toml)")
source_group = parser.add_mutually_exclusive_group()
source_group.add_argument(
"--mode",
choices=["mj", "th", "0", "1"],
default=config_mode,
help="Exclusive source mode: mj/0 for Mahjong Soul, th/1 for Tenhou",
)
source_group.add_argument(
"--source", "--platform",
choices=["majsoul", "tenhou"],
default=None,
help="Legacy source selector; use --mode for new configurations",
dest="legacy_source",
)
dry_run_default = config.get("dry_run", False)
parser.add_argument(
"--dry-run", "--dry_run",
@@ -60,7 +107,7 @@ def parse_args():
# -- Target Options --
target_group = parser.add_argument_group("Target Options")
target_group.add_argument(
"-p", "-u", "--player", dest="player", default=config.get("player") or config.get("nickname"), help="Player nickname"
"-p", "-u", "--player", dest="player", default=config.get("player") or config.get("nickname"), help="Player nickname on the selected source"
)
target_group.add_argument(
"-a", "--account-id", "--account_id", dest="account_id", type=int, default=config.get("account_id"), help="Directly specify player account ID"
@@ -72,7 +119,9 @@ def parse_args():
"--limit", type=int, default=config.get("limit", 10), help="Max records per mode"
)
analysis_group.add_argument(
"--modes", default=str(config.get("modes", "9")), help="Comma-separated mode IDs"
"--modes",
default=None,
help="Comma-separated modes: Mahjong Soul numeric IDs, or Tenhou all/4p/4p-east/4p-south/3p/3p-east/3p-south",
)
analysis_group.add_argument(
"--model-tag", "--model_tag", default=config.get("model_tag", "4.1b"), help="Mortal network version", dest="model_tag"
@@ -90,6 +139,19 @@ def parse_args():
help="Review page language; writes the mjai.ekyu.moe form field select[name='lang']",
dest="review_language"
)
review_ui_config = config.get("review_ui", config.get("ui"))
try:
review_ui_default = normalize_review_ui(review_ui_config)
except ValueError as exc:
parser.error(str(exc))
analysis_group.add_argument(
"--review-ui", "--review_ui", "--ui",
default=review_ui_default,
type=parse_review_ui,
metavar="{classic,killerducky}",
help="Review result UI; KillerDucky metadata and bad-move data are parsed from report JSON",
dest="review_ui",
)
analysis_group.add_argument(
"--retry", type=int, default=config.get("retry", 3), help="Retry failed review items this many times"
)
@@ -104,16 +166,18 @@ def parse_args():
# -- Browser / Network Options --
browser_group = parser.add_argument_group("Browser & Network Options")
headless_default = config.get("headless", False)
headless_default = config.get("headless", True)
browser_group.add_argument(
"--headless",
action="store_true" if not headless_default else "store_false",
action=argparse.BooleanOptionalAction,
default=headless_default,
help="Run browser headlessly",
help="Run browser headlessly (default: on)",
dest="headless"
)
browser_group.add_argument(
"--proxy", default=config.get("proxy"), help="Proxy URL (e.g. http://127.0.0.1:7890)"
"--proxy",
default=config.get("proxy"),
help='Proxy URL (e.g. http://127.0.0.1:7890), or "direct" to bypass all proxies including the system proxy (needed for mjai.ekyu.moe Turnstile)',
)
# -- Output Options --
@@ -184,33 +248,65 @@ def parse_args():
)
args = parser.parse_args()
if not args.player and not args.account_id:
parser.error("-p/--player or -a/--account-id is required either via command line arguments or config file")
if args.legacy_source:
args.mode = normalize_source_mode(args.legacy_source)
else:
args.mode = normalize_source_mode(args.mode)
args.source = source_for_mode(args.mode)
if args.modes is None:
fallback = "all" if args.source == "tenhou" else "9"
args.modes = str(config.get("modes", fallback))
if args.source == "tenhou":
if not args.player:
parser.error("Tenhou source requires -p/--player (a Tenhou player name)")
if args.account_id:
parser.error("--account-id is only supported by the majsoul source")
elif not args.player and not args.account_id:
parser.error("Mahjong Soul source requires -p/--player or -a/--account-id")
args.target_name = args.player if args.player else str(args.account_id)
return args
def build_output_path(nickname: str, output_format: str) -> tuple[str, str]:
def build_output_path(nickname: str, output_format: str, source: str = "majsoul") -> tuple[str, str]:
safe_nick = "".join(
c if c.isalnum() or c in ("_", "-", "\u4e00", "\u9fa5") else "_"
for c in nickname
)
output_root = os.path.join(os.path.dirname(os.path.abspath(__file__)), "results", safe_nick)
results_root = os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"results",
source,
)
output_root = os.path.join(results_root, safe_nick)
out_path = os.path.join(output_root, f"results.{output_format}")
return output_root, out_path
def detect_proxy(explicit_proxy: str | None) -> str | None:
if explicit_proxy:
if explicit_proxy.strip().lower() == "direct":
return "direct"
return explicit_proxy
sys_proxies = urllib.request.getproxies()
return sys_proxies.get("https") or sys_proxies.get("http")
def collect_tasks(account_id: int, modes: list[int], limit: int, output_root: str, processed_uuids: set) -> list[dict]:
def finalize_tasks(tasks: list[dict]) -> list[dict]:
total_tasks = len(tasks)
for index, task in enumerate(tasks, start=1):
task["idx"] = index
task["total"] = total_tasks
short_url = task["uuid"].split("-")[-1]
task["log_prefix"] = f"[{index}/{total_tasks}][{short_url}]"
return tasks
def collect_majsoul_tasks(account_id: int, modes: list[int], limit: int, output_root: str, processed_uuids: set) -> list[dict]:
tasks = []
for mode in modes:
log_line(f"[Mode {mode}] Fetching records...")
@@ -232,6 +328,7 @@ def collect_tasks(account_id: int, modes: list[int], limit: int, output_root: st
continue
tasks.append(
{
"source": "majsoul",
"mode": mode,
"uuid": item["uuid"],
"paipu_url": item["paipuUrl"],
@@ -241,24 +338,48 @@ def collect_tasks(account_id: int, modes: list[int], limit: int, output_root: st
}
)
total_tasks = len(tasks)
for index, task in enumerate(tasks, start=1):
task["idx"] = index
task["total"] = total_tasks
short_url = task["uuid"].split("-")[-1]
task["log_prefix"] = f"[{index}/{total_tasks}][{short_url}]"
return finalize_tasks(tasks)
return tasks
def collect_tenhou_tasks(
records: list[dict],
player_name: str,
modes: tuple[str, ...],
limit: int,
output_root: str,
processed_uuids: set,
) -> list[dict]:
items = build_tenhou_paipu_urls(records, player_name, modes=modes, limit=limit)
tasks = []
for item in items:
if item["uuid"] in processed_uuids:
log_line(f"[Skip] uuid={item['uuid']} already processed.")
continue
mode = item["mode"]
tasks.append(
{
"source": "tenhou",
"mode": mode,
"uuid": item["uuid"],
"paipu_url": item["paipuUrl"],
"start_time": item.get("startTime", ""),
"end_time": item.get("endTime", ""),
"mode_dir": os.path.join(output_root, f"mode_{mode}"),
}
)
return finalize_tasks(tasks)
def print_summary(args, modes):
log_line("=== Batch Mortal Analysis ===")
target_display = args.target_name + (f" (ID: {args.account_id})" if args.account_id and args.target_name != str(args.account_id) else "")
log_line(f" Target: {target_display}")
log_line(f" Mode: {args.mode} ({args.source})")
log_line(f" Modes: {modes}")
log_line(f" Limit: {args.limit} per mode")
log_line(f" ModelTag: {args.model_tag}")
log_line(f" Language: {args.review_language}")
log_line(f" ReviewUI: {args.review_ui}")
log_line(f" Headless: {args.headless}")
log_line(f" DryRun: {args.dry_run}")
log_line(f" Retry: {args.retry}")
@@ -350,6 +471,7 @@ def consume_result_event(args, writer: ResultWriter, result_event: dict, stats:
timestamp = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
base_row = {
"nickname": args.target_name,
"source": task.get("source", args.source),
"mode": task["mode"],
"uuid": task["uuid"],
"paipuUrl": task["paipu_url"],
@@ -444,7 +566,7 @@ def run_parallel_analysis(
log_line("[Serial] Starting analysis with 1 persistent browser")
try:
with SB(uc=True, headless=automator.headless, proxy=automator.proxy) as sb:
with SB(**build_browser_kwargs(automator.headless, automator.proxy)) as sb:
for task in tasks:
result_event = None
for attempt in range(args.retry + 1):
@@ -527,13 +649,26 @@ def main():
start_time = time.time()
args = parse_args()
args.retry = max(0, args.retry)
modes = [int(mode.strip()) for mode in args.modes.split(",")]
try:
if args.source == "tenhou":
modes = normalize_tenhou_modes(args.modes)
else:
modes = [int(mode.strip()) for mode in args.modes.split(",") if mode.strip()]
if not modes:
raise ValueError("At least one Mahjong Soul mode is required.")
except ValueError as exc:
logging.error(f"[FATAL] {exc}")
sys.exit(2)
if not args.dry_run:
ensure_uc_driver()
account_id = None
tenhou_records = None
try:
if args.account_id:
if args.source == "tenhou":
args.target_name, tenhou_records = fetch_tenhou_player_records(args.player)
elif args.account_id:
account_id = args.account_id
if not args.player:
# Attempt to fetch nickname to use as the target
@@ -549,16 +684,28 @@ def main():
print_summary(args, modes)
output_root, out_path = build_output_path(args.target_name, args.output)
output_root, out_path = build_output_path(args.target_name, args.output, args.source)
processed_uuids = get_processed_uuids(out_path, args.output)
proxy = detect_proxy(args.proxy)
if proxy:
if proxy == "direct":
logging.info("[Proxy] Forcing direct connection (bypassing system proxy) for browser.")
elif proxy:
logging.info(f"[Proxy] Using proxy for browser: {proxy}")
else:
logging.info("[Proxy] No system proxy detected, running directly.")
tasks = collect_tasks(account_id, modes, args.limit, output_root, processed_uuids)
if args.source == "tenhou":
tasks = collect_tenhou_tasks(
tenhou_records,
args.target_name,
modes,
args.limit,
output_root,
processed_uuids,
)
else:
tasks = collect_majsoul_tasks(account_id, modes, args.limit, output_root, processed_uuids)
total_processed = 0
total_failed = 0
analysis_stats = create_analysis_stats()
@@ -573,6 +720,7 @@ def main():
headless=args.headless,
proxy=proxy,
review_language=args.review_language,
review_ui=args.review_ui,
submission_coordinator=None,
controlled_submission=False,
)
@@ -586,6 +734,7 @@ def main():
headless=args.headless,
proxy=proxy,
review_language=args.review_language,
review_ui=args.review_ui,
submission_coordinator=submission_coordinator,
controlled_submission=True,
)
@@ -599,6 +748,7 @@ def main():
headless=args.headless,
proxy=proxy,
review_language=args.review_language,
review_ui=args.review_ui,
submission_coordinator=submission_coordinator,
controlled_submission=True,
)
@@ -616,7 +766,13 @@ def main():
log_line(f" Time: {elapsed:.2f}s")
if not args.dry_run:
log_line(f" Output: {out_path}")
plot_results(args.target_name, args.plot, args.output, plot_limit=args.plot_limit)
plot_results(
args.target_name,
args.plot,
args.output,
plot_limit=args.plot_limit,
output_root=output_root,
)
log_line("============")

View File

@@ -1,4 +1,10 @@
from batchmortal.browser import build_review_url, normalize_review_language
import pytest
from batchmortal.browser import (
build_review_url,
normalize_review_language,
normalize_review_ui,
)
def test_normalize_review_language():
@@ -17,6 +23,19 @@ def test_build_review_url():
assert build_review_url("ko") == "https://mjai.ekyu.moe/ko.html"
def test_normalize_review_ui():
assert normalize_review_ui(None) == "classic"
assert normalize_review_ui("classic") == "classic"
assert normalize_review_ui("killerducky") == "killerducky"
assert normalize_review_ui("killer-ducky") == "killerducky"
assert normalize_review_ui("kd") == "killerducky"
def test_normalize_review_ui_rejects_unknown_values():
with pytest.raises(ValueError, match="Unsupported review UI"):
normalize_review_ui("foo")
if __name__ == "__main__":
test_normalize_review_language()
test_build_review_url()

View File

@@ -0,0 +1,79 @@
import pytest
from batchmortal.config import (
normalize_source_mode,
resolve_mode_config,
source_for_mode,
)
CONFIG = {
"mode": "mj",
"review_language": "zh-CN",
"mj": {
"nickname": "雀魂玩家",
"limit": 70,
"modes": "12",
},
"th": {
"nickname": "天凤玩家",
"limit": 10,
"modes": "4p-south",
},
}
def test_mj_and_th_sections_are_selected_exclusively():
mode, source, selected = resolve_mode_config(CONFIG)
assert (mode, source) == ("mj", "majsoul")
assert selected["nickname"] == "雀魂玩家"
assert selected["modes"] == "12"
mode, source, selected = resolve_mode_config(CONFIG, requested_mode="th")
assert (mode, source) == ("th", "tenhou")
assert selected["nickname"] == "天凤玩家"
assert selected["modes"] == "4p-south"
def test_numeric_and_long_source_aliases_remain_supported():
assert normalize_source_mode(0) == "mj"
assert normalize_source_mode(1) == "th"
assert normalize_source_mode("majsoul") == "mj"
assert normalize_source_mode("tenhou") == "th"
assert source_for_mode("th") == "tenhou"
def test_normalize_source_mode_defaults_to_mj_for_none():
assert normalize_source_mode(None) == "mj"
@pytest.mark.parametrize("value", ["foo", 2])
def test_normalize_source_mode_rejects_unsupported_values(value):
with pytest.raises(ValueError, match="Unsupported source mode"):
normalize_source_mode(value)
def test_legacy_source_config_is_still_readable():
mode, source, selected = resolve_mode_config(
{
"source": "tenhou",
"nickname": "旧配置玩家",
"tenhou_modes": "4p",
}
)
assert (mode, source) == ("th", "tenhou")
assert selected["nickname"] == "旧配置玩家"
assert selected["modes"] == "4p"
def test_conflicting_config_selectors_are_rejected():
with pytest.raises(ValueError, match="both 'mode' and legacy 'source'"):
resolve_mode_config({"mode": "mj", "source": "tenhou"})
with pytest.raises(ValueError, match="both 'mode' and legacy 'source'"):
resolve_mode_config({"mode": "mj", "source": "majsoul"})
def test_source_section_must_be_a_mapping():
with pytest.raises(ValueError, match="section 'th'"):
resolve_mode_config({"mode": "th", "th": "not-a-mapping"})

102
tests/test_killerducky.py Normal file
View File

@@ -0,0 +1,102 @@
from batchmortal.browser import (
parse_killerducky_bad_move_stats,
parse_killerducky_metadata,
)
from batchmortal.results import parse_metadata
def make_entry(actual_probability, *, is_equal=False, actual_index=1):
details = [
{"prob": 0.8},
{"prob": actual_probability},
]
return {
"is_equal": is_equal,
"actual_index": actual_index,
"details": details,
}
def test_parse_killerducky_about_metadata():
data = {
"engine": "Mortal",
"game_length": "Hanchan",
"review_time": "2s 709ms",
"player_id": 2,
"review": {
"model_tag": "4.1b",
"rating": 0.8422832528256039,
"temperature": 0.1,
"total_matches": 131,
"total_reviewed": 173,
},
}
metadata = parse_killerducky_metadata(data)
assert metadata == {
"engine": "Mortal",
"model tag": "4.1b",
"rating": "84.228",
"matches/total": "131/173 = 75.723%",
"temperature": "0.1",
"game length": "Hanchan",
"player id": "2",
"review duration": "2s 709ms",
}
assert parse_metadata(metadata) == {
"modelTag": "4.1b",
"rating": "84.228",
"aiConsistencyRate": "75.723%",
"aiConsistencyNumerator": "131",
"aiConsistencyDenominator": "173",
"temperature": "0.1",
"gameLength": "Hanchan",
"playerId": "2",
"reviewDuration": "2s 709ms",
}
def test_parse_killerducky_bad_moves_uses_actual_choice_probability():
invalid_entry = {
"is_equal": False,
"actual_index": 4,
"details": [{"prob": 0.9}],
}
data = {
"review": {
"total_reviewed": 4,
"kyokus": [
{
"entries": [
make_entry(0.01, is_equal=True, actual_index=0),
make_entry(0.04),
make_entry(0.08),
invalid_entry,
]
}
],
}
}
assert parse_killerducky_bad_move_stats(data) == {
"badMoveRate5": "25.000%",
"badMoveCount5": "1",
"badMoveRate10": "50.000%",
"badMoveCount10": "2",
"badMoveDenominator": "4",
"badMoveOrderLossCount": "3",
"badMoveUnparsedCount": "1",
}
def test_parse_killerducky_bad_moves_handles_missing_review_data():
assert parse_killerducky_bad_move_stats({}) == {
"badMoveRate5": "",
"badMoveCount5": "0",
"badMoveRate10": "",
"badMoveCount10": "0",
"badMoveDenominator": "",
"badMoveOrderLossCount": "0",
"badMoveUnparsedCount": "0",
}

View File

@@ -0,0 +1,14 @@
from pathlib import Path
import pytest
from main import build_output_path
@pytest.mark.parametrize("source", ["majsoul", "tenhou"])
def test_results_are_grouped_by_source(source):
output_root, output_file = build_output_path("测试 玩家", "xlsx", source)
root_path = Path(output_root)
assert root_path.parts[-3:] == ("results", source, "测试_玩家")
assert Path(output_file) == root_path / "results.xlsx"

179
tests/test_tenhou.py Normal file
View File

@@ -0,0 +1,179 @@
import pytest
from batchmortal.tenhou import (
build_tenhou_paipu_urls,
decode_tenhou_viewpoint,
format_tenhou_timestamp,
normalize_tenhou_modes,
parse_tenhou_log_url,
tenhou_mode_matches,
tenhou_record_mode,
)
PLAYER = "ププリン"
def make_record(
log_id: str,
*,
starttime: int,
playernum: int = 4,
playlength: int = 2,
url_host: str = "tenhou.net",
) -> dict:
record = {
"starttime": starttime,
"during": 32,
"playernum": playernum,
"playlength": playlength,
"player1": PLAYER,
"player2": "Tatsuno7",
"player3": "Navitas",
"tw": "198",
"url": f"http://{url_host}/0/?log={log_id}",
}
if playernum == 4:
record["player4"] = "tca00"
return record
def test_decode_nodocchi_viewpoint_from_real_example():
# Nodocchi returns 198 for this game. ププリン is player1 (first place),
# and the site's two-bit permutation decoding resolves that to Tenhou seat 2.
assert decode_tenhou_viewpoint("198", 1) == 2
def test_decode_tenhou_viewpoint_rejects_non_numeric_tw():
assert decode_tenhou_viewpoint("abc", 1) is None
assert decode_tenhou_viewpoint(None, 1) is None
@pytest.mark.parametrize("player_order", [0, 5, None])
def test_decode_tenhou_viewpoint_rejects_invalid_player_order(player_order):
assert decode_tenhou_viewpoint("198", player_order) is None
def test_decode_tenhou_viewpoint_returns_none_for_unmapped_packed_value():
assert decode_tenhou_viewpoint("999", 1) is None
def test_build_real_example_paipu_url():
record = make_record(
"2026071212gm-00a9-0000-2dab9d24",
starttime=1783828560,
)
assert build_tenhou_paipu_urls([record], PLAYER) == [
{
"source": "tenhou",
"mode": "4p-south",
"uuid": "2026071212gm-00a9-0000-2dab9d24",
"paipuUrl": "http://tenhou.net/0/?log=2026071212gm-00a9-0000-2dab9d24&tw=2",
"startTime": format_tenhou_timestamp(1783828560),
"endTime": format_tenhou_timestamp(1783828560 + 32 * 60),
}
]
def test_build_tenhou_paipu_urls_deduplicates_log_ids():
log_id = "2026071212gm-00a9-0000-11111111"
records = [
make_record(log_id, starttime=200),
make_record(log_id, starttime=100),
]
items = build_tenhou_paipu_urls(records, PLAYER)
assert len(items) == 1
assert items[0]["uuid"] == log_id
assert items[0]["startTime"] == format_tenhou_timestamp(200)
def test_build_tenhou_paipu_urls_skips_invalid_viewpoints():
valid_log_id = "2026071212gm-00a9-0000-11111111"
invalid_tw = make_record(
"2026071212gm-00a9-0000-22222222",
starttime=200,
)
invalid_tw["tw"] = "999"
missing_player = make_record(
"2026071212gm-00a9-0000-33333333",
starttime=100,
)
missing_player["player1"] = "another-player"
items = build_tenhou_paipu_urls(
[make_record(valid_log_id, starttime=300), invalid_tw, missing_player],
PLAYER,
)
assert [item["uuid"] for item in items] == [valid_log_id]
def test_mode_filter_and_limit_are_applied_per_actual_mode():
records = [
make_record("2026071213gm-00a9-0000-aaaaaaaa", starttime=300, playlength=2),
make_record("2026071212gm-00a9-0000-bbbbbbbb", starttime=200, playlength=2),
make_record("2026071211gm-00a9-0000-cccccccc", starttime=100, playlength=1),
]
items = build_tenhou_paipu_urls(records, PLAYER, modes="4p", limit=1)
assert [item["uuid"] for item in items] == [
"2026071213gm-00a9-0000-aaaaaaaa",
"2026071211gm-00a9-0000-cccccccc",
]
assert [item["mode"] for item in items] == ["4p-south", "4p-east"]
def test_non_tenhou_urls_are_not_mistaken_for_tenhou_records():
record = make_record(
"2026071212gm-00a9-0000-2dab9d24",
starttime=1783828560,
url_host="game.maj-soul.com",
)
assert build_tenhou_paipu_urls([record], PLAYER) == []
assert parse_tenhou_log_url(record["url"]) is None
def test_records_without_url_or_viewpoint_are_skipped():
no_url = make_record("2026071212gm-00a9-0000-2dab9d24", starttime=200)
no_url.pop("url")
no_viewpoint = make_record("2026071213gm-00a9-0000-aaaaaaaa", starttime=300)
no_viewpoint.pop("tw")
assert build_tenhou_paipu_urls([no_url, no_viewpoint], PLAYER) == []
def test_tenhou_mode_aliases_and_labels():
assert normalize_tenhou_modes("四南,3p-east") == ("4p-south", "3p-east")
assert normalize_tenhou_modes("*") == ("all",)
assert tenhou_record_mode({"playernum": 3, "playlength": 1}) == "3p-east"
def test_normalize_tenhou_modes_rejects_unsupported_label():
with pytest.raises(ValueError, match="Unsupported Tenhou mode 'foo'"):
normalize_tenhou_modes(("foo",))
@pytest.mark.parametrize("modes", ["", (), ("",), (" ",), (" ", "", " ")])
def test_normalize_tenhou_modes_empty_values_fall_back_to_all(modes):
assert normalize_tenhou_modes(modes) == ("all",)
def test_tenhou_mode_matches_normalized_modes():
four_player = normalize_tenhou_modes(("4p",))
assert tenhou_mode_matches("4p-east", four_player)
assert tenhou_mode_matches("4p-south", four_player)
assert not tenhou_mode_matches("3p-east", four_player)
assert not tenhou_mode_matches("3p-south", four_player)
east_only = normalize_tenhou_modes(("4p-east",))
assert tenhou_mode_matches("4p-east", east_only)
assert not tenhou_mode_matches("4p-south", east_only)
south_only = normalize_tenhou_modes(("4p-south",))
assert tenhou_mode_matches("4p-south", south_only)
assert not tenhou_mode_matches("4p-east", south_only)

223
tests/test_visualize.py Normal file
View File

@@ -0,0 +1,223 @@
import pytest
from batchmortal import visualize
from batchmortal.visualize import (
prepare_dashboard_data,
rolling_average,
)
def make_result(
index: int,
*,
rating: float,
ai_rate: str = "",
ai_numerator: str = "",
ai_denominator: str = "",
) -> dict:
return {
"nickname": "测试玩家",
"source": "majsoul",
"mode": "16",
"uuid": f"game-{index}",
"startTime": f"2026-07-{index:02d} 12:00:00",
"resultUrl": f"https://example.com/report/{index}",
"modelTag": "4.1b",
"rating": str(rating),
"aiConsistencyRate": ai_rate,
"aiConsistencyNumerator": ai_numerator,
"aiConsistencyDenominator": ai_denominator,
}
def test_rolling_average_requires_a_full_window():
assert rolling_average([1, 2, 3, 4, 5], 3) == [None, None, 2.0, 3.0, 4.0]
with pytest.raises(ValueError, match="positive"):
rolling_average([1, 2], 0)
def test_dashboard_data_keeps_missing_ai_null_and_weights_valid_samples():
missing_ai = make_result(1, rating=90)
measured = make_result(
2,
rating=92,
ai_rate="1%",
ai_numerator="8",
ai_denominator="10",
)
measured.update(
{
"badMoveCount5": "1",
"badMoveCount10": "2",
"badMoveDenominator": "20",
}
)
data = prepare_dashboard_data([missing_ai, measured])
assert data is not None
assert data["points"][0]["aiRate"] is None
assert data["points"][1]["aiRate"] == pytest.approx(80.0)
assert data["aiRate"] == pytest.approx(80.0)
assert data["aiDenominator"] == 10
assert data["aiWeighted"] is True
assert data["aiAxisMin"] == 60
assert data["badRate5"] == pytest.approx(5.0)
assert data["badRate10"] == pytest.approx(10.0)
assert data["badRateAxisMax"] == 12
assert data["badRateAxisInterval"] == 3
assert data["trendWindow"] is None
assert data["histogram"] == []
def test_dashboard_data_uses_recent_comparison_and_rolling_trend():
records = [make_result(index, rating=79 + index) for index in range(1, 21)]
data = prepare_dashboard_data(records)
assert data is not None
assert data["trendWindow"] == 10
assert data["ratingMean"] == pytest.approx(89.5)
assert data["comparisonWindow"] == 10
assert data["recentAverage"] == pytest.approx(94.5)
assert data["comparisonDelta"] == pytest.approx(10.0)
assert data["ratingRolling"][8] is None
assert data["ratingRolling"][9] == pytest.approx(84.5)
assert data["ratingDenseLower"] == pytest.approx(84.75)
assert data["ratingDenseUpper"] == pytest.approx(94.25)
assert "ratingBandLower" not in data
assert "ratingBandUpper" not in data
assert len(data["histogram"]) == 10
assert data["histogram"][0]["lower"] == 80
assert data["histogram"][-1]["upper"] == 100
def test_rating_batches_are_newest_anchored_in_twenty_game_windows():
records = [make_result(index, rating=80 + index) for index in range(1, 46)]
data = prepare_dashboard_data(records)
assert data is not None
assert [batch["count"] for batch in data["ratingBatches"]] == [5, 20, 20]
assert [batch["startLabel"] for batch in data["ratingBatches"]] == [
"#1",
"#6",
"#26",
]
assert [batch["endLabel"] for batch in data["ratingBatches"]] == [
"#5",
"#25",
"#45",
]
assert data["ratingBatches"][-1]["ratingMean"] == pytest.approx(115.5)
def test_bad_rate_axis_uses_padded_readable_quarters():
assert visualize._rate_axis_scale([1.0, 14.4828]) == (16.0, 4.0)
assert visualize._rate_axis_scale([0.0]) == (1.0, 0.25)
assert visualize._rate_axis_scale([]) == (10.0, 2.5)
def test_generate_html_contains_professional_sections_and_escapes_nickname(
monkeypatch,
tmp_path,
):
records = [
make_result(
index,
rating=88 + index,
ai_numerator=str(80 + index),
ai_denominator="100",
)
for index in range(1, 11)
]
monkeypatch.setattr(
visualize,
"read_results",
lambda nickname, output_format, output_root=None: records,
)
output_path = tmp_path / "report.html"
result = visualize.generate_html("<测试玩家>", str(output_path))
rendered = output_path.read_text(encoding="utf-8")
assert result == str(output_path)
assert "&lt;测试玩家&gt;" in rendered
assert "Rating 推移" in rendered
assert "AI 一致率推移" in rendered
assert "Rating 分布" in rendered
assert "检讨候选" in rendered
assert "半庄移动平均" in rendered
assert '"trendWindow":10' in rendered
assert "recentFocusMark" not in rendered
assert "recentFocusArea" not in rendered
assert "ratingContextArea" in rendered
assert "ratingDensityArea" in rendered
assert "全样本中间50%" in rendered
assert "density-key" in rendered
assert "Rating 中间50%" in rendered
assert '"ratingDenseLower"' in rendered
assert '"ratingDenseUpper"' in rendered
assert "trendReferenceLine" in rendered
assert "recentMarkArea" not in rendered
assert "bandRange" not in rendered
assert "smoothMonotone" in rendered
assert "rawEndLabel" in rendered
assert "trendEndLabel" not in rendered
assert "rollingBandSeries" not in rendered
assert "confidenceBandSeries" not in rendered
assert "95% 置信区间" not in rendered
assert '"ratingConfidenceLower"' not in rendered
assert '"aiConfidenceLower"' not in rendered
assert "rawSeriesData" in rendered
assert "selectedRawSeriesData" in rendered
assert "rating-selected-raw" in rendered
assert "rating-bad-rate-10-bars" in rendered
assert "rating-bad-rate-5-bars" in rendered
assert "badRateBarData" in rendered
assert "badRateXAxisConfig" in rendered
assert '"badRateAxisMax"' in rendered
assert '"badRateAxisInterval"' in rendered
assert "Rating、5% 与 10% 恶手率推移" in rendered
assert "position: 'left'" in rendered
assert "rating-batch-summary" in rendered
assert "bindBatchSelection" in rendered
assert "refreshBatchSelection" in rendered
assert rendered.count("markArea: ratingContextArea()") == 4
assert rendered.count("barMaxWidth: 7") == 2
assert '"ratingBatches"' in rendered
assert "rating-bin-highlight" in rendered
assert "rating-distribution-bars" in rendered
assert "distributionBarData" in rendered
assert "selectedRatingBinIndexes" in rendered
assert "ratingSelectionPalette" in rendered
assert "sourceEvent.ctrlKey" in rendered
assert "sourceEvent.metaKey" in rendered
assert "symbolSize: 12" in rendered
assert "rgba(255, 255, 255, 0)" not in rendered
assert "color: '#ffffff', borderColor: colors.indigoDark, borderWidth: 2" in rendered
assert "borderWidth: 2.4" not in rendered
assert "rating-selection-clear" in rendered
assert "取消选择" in rendered
assert "selectedRatingBinIndexes.length === 1 && existingPosition === 0" in rendered
assert "updateRatingSelectionControl" in rendered
assert "batch.startIndex" in rendered
assert "ai-selected-raw" in rendered
assert "rawPointData" in rendered
assert 'data-trend-view="overview"' in rendered
assert 'data-trend-view="detail"' in rendered
assert "header-metrics" in rendered
assert "header-metrics-grid" in rendered
assert "header-metrics-vertical" not in rendered
assert "header-metric-primary" in rendered
assert "全样本 Rating 平均值" in rendered
assert "header-dual-metric" in rendered
assert "review-snapshot" not in rendered
assert '"ratingBandLower"' not in rendered
assert "半庄中间50%" not in rendered
assert "观察常见水平" not in rendered
assert "section-note" not in rendered
assert "趋势预测线" not in rendered
assert "linear-gradient" not in rendered
assert "$payload" not in rendered