上海 9929 优选低延迟路径： GOST MTCP + ECMP 启动预热

测试机器环境：利群上海 9929 、xTom 软银 JP 。

0.优选后测试结果截图

RTT 延迟不抽卡，维持稳定不波动不断流。单线程测速能跑起来。

1. 问题背景

部分跨境/跨运营商线路 [本文特指上海 9929 宽带，存在多条 4837 & 9929 混合 ECMP： 据传联通将该业务外包给第三方公司，在后端接入了多条 AS4837 （普通联通骨干网）和 AS9929 （联通精品网）线路] 存在明显的 ECMP （ Equal-Cost Multi-Path ）：ICMP ping 稳定，但不同 TCP 连接的 RTT 会落在两个甚至多个明显不同的档位（例如一部分 ~33ms ，一部分 ~51ms ）。每次新建 TCP 都可能重新参与 ECMP 哈希，导致普通端口转发下不同业务连接分别落在快路/慢路，延迟随连接波动。

2. 核心思路

在 GOST MTCP 启动阶段选出一条低延迟外层 TCP ，后续所有业务长期复用这一条连接，而不是在运行期间持续探测、清理慢连接。

任意 TCP 客户端 → 前置 GOST :12000 → 单个长期 MTCP session （ 4 MiB Stream / 32 MiB Receive ）
  → 启动阶段检测并只保留低延迟外层 TCP → 远端 GOST :11000 → 目标 TCP 服务

GOST MTCP 把多个逻辑 TCP stream 复用在同一条外层 TCP 上，因此 ECMP 抽路的影响从"每个业务连接抽一次"收敛为"MTCP 外层连接建立时抽一次"。

本人曾尝试同时维持多条 MTCP TCP 、定期探测 RTT 、ss -K 踢慢路，但存在两个问题：一是 ss -K 会影响该 session 上承载的所有逻辑 stream ；二是踢掉后重建的连接会重新参与 ECMP ，仍可能抽到慢路，形成"抽路—清理"死循环。最终方案改为：只配置一个 MTCP node ，正常只允许一个 outer TCP ，仅在启动阶段做 ECMP 预热，命中快路后停止干预，运行期间不跑周期检测、不定时 kill 、不主动重建。

3. MTCP 缓冲区参数

单 stream 缓冲区过小会限制吞吐，需按 BDP 估算。例如 RTT≈33ms 、目标带宽≈400Mbps 时，BDP ≈ 400Mbps × 0.033s ≈ 1.65MiB ，因此 4 MiB Stream Buffer 较为宽裕：

mux.maxFrameSize: 32768        # 32 KiB
mux.maxReceiveBuffer: 33554432 # 32 MiB
mux.maxStreamBuffer: 4194304   # 4 MiB

4. 远端 GOST 配置

远端只需监听 :11000，实际转发目标由前置 GOST 的 forwarder 指定。

/root/gost/remote.yaml：

services:
- name: mtcp-server
  addr: :11000
  handler:
    type: relay
  listener:
    type: mtcp
    metadata:
      mux.version: 2
      mux.keepaliveDisabled: false
      mux.keepaliveInterval: 10s
      mux.keepaliveTimeout: 30s
      mux.maxFrameSize: 32768
      mux.maxReceiveBuffer: 33554432
      mux.maxStreamBuffer: 4194304

5. 前置 GOST 配置

前置监听 :12000，通过 chain-mtcp 走 MTCP dialer 连到远端。

services:
- name: tcp-entry
  addr: :12000
  handler:
    type: tcp
    chain: chain-mtcp
  listener:
    type: tcp
  forwarder:
    nodes:
    - name: target
      addr: 127.0.0.1:2345

chains:
- name: chain-mtcp
  hops:
  - name: remote
    nodes:
    - name: remote-mtcp
      addr: 203.0.113.20:11000
      connector:
        type: relay
      dialer:
        type: mtcp
        metadata:
          mux.version: 2
          mux.keepaliveDisabled: false
          mux.keepaliveInterval: 10s
          mux.keepaliveTimeout: 30s
          mux.maxFrameSize: 32768
          mux.maxReceiveBuffer: 33554432
          mux.maxStreamBuffer: 4194304

最终链路：TCP 客户端 → 前置:12000 → GOST TCP Handler → GOST MTCP Dialer → 203.0.113.20:11000 → 远端 GOST Relay → 127.0.0.1:2345（示例目标地址，可替换为任意内网/本机地址）。本质是 TCP → MTCP 隧道 → TCP。

6. 判断 ECMP 快路 / 慢路

ss -tin "dst 203.0.113.20:11000"

重点看 minrtt: 字段而非当前 rtt（当前 RTT 会受排队、突发流量、重传、ACK 延迟等临时因素影响，minrtt 更能反映这条 TCP 最初命中的基础路径）。例如某线路快路 32~34ms 、慢路 50~52ms ，可设 THRESH=40：minrtt < THRESH 判为 FAST ，否则判为 SLOW 。

7. 启动预热脚本 mtcp-prewarm.sh

脚本只负责启动阶段选路，逻辑要点：

• 目标地址上 ESTAB 连接数 = 0：主动 connect 触发 GOST 建立 MTCP ，等待。
• ESTAB = 1：读取 minrtt；< THRESH 则连续确认 STABLE_REQUIRED 次后判定 PREWARM SUCCESS；>= THRESH 则 ss -K 踢掉这条唯一的慢连接，等它彻底消失后重新触发连接，让 GOST 重新参与一次 ECMP 。
• ESTAB > 1：直接 PREWARM ERROR 退出，不自动挑一条 kill。因为设计目标是"一个 MTCP node + 一个 outer TCP"，出现多条说明运行态已偏离预期（可能是配置 reload 、dialer 被重新实例化、旧 session 未完全退出等），脚本不猜测该删哪条，避免误伤正在承载业务的 session ，提示手动 systemctl restart 对应 GOST 服务。

完整代码（支持多目标、支持自定义触发命令 PREWARM_CMD，含 {DST} {PORT} {TRIGGER} 占位符，各阈值/超时均可用环境变量覆盖）：

#!/usr/bin/env bash

# GOST MTCP ECMP 启动预热
#
# 默认用法：
#   ./mtcp-prewarm.sh <目标 IP> <目标端口>
#
# 多目标：
#   ./mtcp-prewarm.sh <IP1> <PORT1> <IP2> <PORT2> ...
#
# 为不同目标指定不同的本地触发入口：
#   ./mtcp-prewarm.sh \
#     203.0.113.20:[email protected]:12000 \
#     198.51.100.20:[email protected]:13000
#
# 默认触发入口：
#   PREWARM_ADDR=127.0.0.1:12000
#
# 自定义触发命令：
#   PREWARM_CMD='your command' ./mtcp-prewarm.sh ...
#
# PREWARM_CMD 可使用占位符：
#   {DST}      当前目标地址
#   {PORT}     当前目标端口
#   {TRIGGER}  当前本地触发入口
#
# 设计目标：
#   只在启动/维护阶段运行。
#   严格单连接模式：目标地址同时出现超过 1 条 ESTAB TCP 时立即报错退出，
#   不自动清理任何连接，避免把异常运行态误判为预热成功。
#   如果唯一 MTCP 外层 TCP 的 minrtt >= THRESH ，则断开并主动触发重建。
#   连续 STABLE_REQUIRED 次确认唯一连接低于阈值后退出，不再碰现有连接。

set -uo pipefail

THRESH="${THRESH:-40}"
MAX_ROUNDS="${MAX_ROUNDS:-30}"
SETTLE="${SETTLE:-2}"
EMPTY_WAIT="${EMPTY_WAIT:-1}"
STABLE_REQUIRED="${STABLE_REQUIRED:-2}"
PREWARM_ADDR="${PREWARM_ADDR:-127.0.0.1:12000}"
PREWARM_HOLD="${PREWARM_HOLD:-0.3}"
TRIGGER_TIMEOUT="${TRIGGER_TIMEOUT:-3}"
KILL_WAIT="${KILL_WAIT:-2}"
PREWARM_CMD="${PREWARM_CMD:-}"

ts() {
    date '+%F %T'
}

log() {
    echo "[$(ts)] $*"
}

usage() {
    cat <<'USAGE'
用法：
  ./mtcp-prewarm.sh <目标 IP> <目标端口>
  ./mtcp-prewarm.sh <IP1> <PORT1> <IP2> <PORT2> ...
  ./mtcp-prewarm.sh <目标 IP:端口 @触发 IP:端口> [...]

示例：
  ./mtcp-prewarm.sh 203.0.113.20 11000

  ./mtcp-prewarm.sh \
    203.0.113.20 11000 \
    198.51.100.20 11000

  ./mtcp-prewarm.sh \
    203.0.113.20:[email protected]:12000 \
    198.51.100.20:[email protected]:13000

可配置环境变量：
  THRESH=40
  MAX_ROUNDS=30
  SETTLE=2
  EMPTY_WAIT=1
  STABLE_REQUIRED=2
  PREWARM_ADDR=127.0.0.1:12000
  PREWARM_HOLD=0.3
  TRIGGER_TIMEOUT=3
  KILL_WAIT=2
  PREWARM_CMD='自定义触发命令'

PREWARM_CMD 占位符：
  {DST} {PORT} {TRIGGER}
USAGE
}

is_fast() {
    awk -v r="$1" -v t="$THRESH" 'BEGIN { exit !(r < t) }'
}

split_trigger() {
    local trigger="$1"
    TRIGGER_HOST="${trigger%:*}"
    TRIGGER_PORT="${trigger##*:}"

    if [[ -z "$TRIGGER_HOST" || -z "$TRIGGER_PORT" || "$TRIGGER_HOST" == "$TRIGGER_PORT" ]]; then
        return 1
    fi

    case "$TRIGGER_PORT" in
        ''|*[!0-9]*) return 1 ;;
    esac

    return 0
}

trigger_reconnect() {
    local dst="$1"
    local port="$2"
    local trigger="$3"
    local cmd=""

    log "${dst}:${port} trigger=${trigger}"

    if [[ -n "$PREWARM_CMD" ]]; then
        cmd="${PREWARM_CMD//\{DST\}/$dst}"
        cmd="${cmd//\{PORT\}/$port}"
        cmd="${cmd//\{TRIGGER\}/$trigger}"
        timeout "$TRIGGER_TIMEOUT" bash -lc "$cmd" >/dev/null 2>&1 || true
        return
    fi

    if ! split_trigger "$trigger"; then
        log "${dst}:${port} 无效触发入口: ${trigger}"
        return
    fi

    timeout "$TRIGGER_TIMEOUT" bash -c \
        "exec 3<>/dev/tcp/${TRIGGER_HOST}/${TRIGGER_PORT}; sleep ${PREWARM_HOLD}" \
        >/dev/null 2>&1 || true
}

count_established() {
    local dst="$1"
    local port="$2"

    ss -ntH "dst ${dst}:${port}" 2>/dev/null |
        awk '$1 == "ESTAB" { n++ } END { print n + 0 }'
}

show_established() {
    local dst="$1"
    local port="$2"

    ss -ntpH "dst ${dst}:${port}" 2>/dev/null |
    awk '$1 == "ESTAB" {
        printf("  local=%s peer=%s", $4, $5)
        for (i=6; i<=NF; i++) printf(" %s", $i)
        printf("\n")
    }'
}

get_flows() {
    local dst="$1"
    local port="$2"
    local lport=""
    local line=""
    local local_ep=""

    ss -tinH "dst ${dst}:${port}" 2>/dev/null |
    while IFS= read -r line; do
        if [[ "$line" == ESTAB* ]]; then
            local_ep=$(awk '{print $4}' <<<"$line")
            lport="${local_ep##*:}"
            continue
        fi

        if [[ -n "$lport" && "$line" =~ minrtt:([0-9.]+) ]]; then
            echo "${lport} ${BASH_REMATCH[1]}"
            lport=""
        fi
    done
}

kill_flow() {
    local dst="$1"
    local port="$2"
    local lport="$3"
    local minrtt="$4"

    log "${dst}:${port} KILL sport=${lport} minrtt=${minrtt}ms"
    ss -K "dst ${dst}:${port} sport = :${lport}" >/dev/null 2>&1 || true
}

prewarm_one() {
    local dst="$1"
    local port="$2"
    local trigger="$3"
    local stable=0
    local round total
    local flow lport minrtt
    local lock lockfd
    local socket_count kill_deadline

    case "$port" in
        ''|*[!0-9]*)
            log "错误: ${dst}:${port} 端口必须是数字"
            return 2
            ;;
    esac

    lock="/tmp/mtcp-prewarm-${dst//[^a-zA-Z0-9]/_}-${port}.lock"

    exec {lockfd}>"$lock"

    if ! flock -n "$lockfd"; then
        log "${dst}:${port} 已有预热任务运行，跳过"
        exec {lockfd}>&-
        return 0
    fi

    log "============================================================"
    log "MTCP ECMP prewarm start"
    log "target=${dst}:${port}"
    log "trigger=${trigger}"
    log "fast threshold=<${THRESH}ms"
    log "max rounds=${MAX_ROUNDS}"
    log "============================================================"

    for ((round=1; round<=MAX_ROUNDS; round++)); do

        socket_count="$(count_established "$dst" "$port")"

        if (( socket_count > 1 )); then
            log "============================================================"
            log "${dst}:${port} PREWARM ERROR"
            log "${dst}:${port} expected exactly 1 MTCP outer TCP, found ${socket_count}"
            log "${dst}:${port} multiple MTCP sessions detected"
            log "${dst}:${port} restart the corresponding GOST service"
            log "${dst}:${port} current established sockets:"

            show_established "$dst" "$port" |
            while IFS= read -r line; do
                log "$line"
            done

            log "============================================================"

            exec {lockfd}>&-
            return 3
        fi

        if (( socket_count == 0 )); then
            stable=0

            log "${dst}:${port} round=${round} no MTCP established flow"

            trigger_reconnect "$dst" "$port" "$trigger"

            sleep "$EMPTY_WAIT"
            continue
        fi

        mapfile -t flows < <(get_flows "$dst" "$port")

        total=${#flows[@]}

        if (( total == 0 )); then
            stable=0

            log "${dst}:${port} round=${round} one ESTAB flow exists but minrtt is not ready"

            sleep "$EMPTY_WAIT"
            continue
        fi

        if (( total > 1 )); then
            log "============================================================"
            log "${dst}:${port} PREWARM ERROR"
            log "${dst}:${port} inconsistent ss result"
            log "${dst}:${port} socket_count=${socket_count}, parsed_flows=${total}"
            log "${dst}:${port} no connection was modified"
            log "============================================================"

            exec {lockfd}>&-
            return 4
        fi

        flow="${flows[0]}"
        lport="${flow%% *}"
        minrtt="${flow##* }"

        log "---------------- ${dst}:${port} round ${round} ----------------"

        if is_fast "$minrtt"; then

            log "${dst}:${port} FAST sport=${lport} minrtt=${minrtt}ms"
            log "${dst}:${port} flows=1 fast=1 slow=0"

            ((stable++))

            log "${dst}:${port} single flow FAST, stable=${stable}/${STABLE_REQUIRED}"

            if (( stable >= STABLE_REQUIRED )); then

                socket_count="$(count_established "$dst" "$port")"

                if (( socket_count != 1 )); then
                    log "============================================================"
                    log "${dst}:${port} PREWARM ERROR"
                    log "${dst}:${port} final verification expected 1 MTCP outer TCP, found ${socket_count}"

                    if (( socket_count > 1 )); then
                        log "${dst}:${port} restart the corresponding GOST service"

                        show_established "$dst" "$port" |
                        while IFS= read -r line; do
                            log "$line"
                        done
                    fi

                    log "============================================================"

                    exec {lockfd}>&-
                    return 3
                fi

                log "${dst}:${port} PREWARM SUCCESS"
                log "${dst}:${port} verified single MTCP outer TCP with minrtt < ${THRESH}ms"

                exec {lockfd}>&-
                return 0
            fi

            sleep "$SETTLE"
            continue
        fi

        stable=0

        log "${dst}:${port} SLOW sport=${lport} minrtt=${minrtt}ms"
        log "${dst}:${port} flows=1 fast=0 slow=1"

        socket_count="$(count_established "$dst" "$port")"

        if (( socket_count != 1 )); then
            log "============================================================"
            log "${dst}:${port} PREWARM ERROR"
            log "${dst}:${port} flow count changed before kill"
            log "${dst}:${port} expected 1, found ${socket_count}"
            log "${dst}:${port} no connection was modified"

            if (( socket_count > 1 )); then
                log "${dst}:${port} restart the corresponding GOST service"

                show_established "$dst" "$port" |
                while IFS= read -r line; do
                    log "$line"
                done
            fi

            log "============================================================"

            exec {lockfd}>&-
            return 3
        fi

        kill_flow "$dst" "$port" "$lport" "$minrtt"

        kill_deadline=$((SECONDS + KILL_WAIT))

        while (( SECONDS < kill_deadline )); do

            socket_count="$(count_established "$dst" "$port")"

            if (( socket_count == 0 )); then
                break
            fi

            sleep 0.1
        done

        socket_count="$(count_established "$dst" "$port")"

        if (( socket_count > 1 )); then
            log "============================================================"
            log "${dst}:${port} PREWARM ERROR"
            log "${dst}:${port} multiple sessions appeared after kill: ${socket_count}"
            log "${dst}:${port} restart the corresponding GOST service"

            show_established "$dst" "$port" |
            while IFS= read -r line; do
                log "$line"
            done

            log "============================================================"

            exec {lockfd}>&-
            return 3
        fi

        if (( socket_count == 1 )); then
            log "============================================================"
            log "${dst}:${port} PREWARM ERROR"
            log "${dst}:${port} killed flow did not disappear within ${KILL_WAIT}s"
            log "${dst}:${port} no reconnect was triggered"
            log "${dst}:${port} check/restart the corresponding GOST service"

            show_established "$dst" "$port" |
            while IFS= read -r line; do
                log "$line"
            done

            log "============================================================"

            exec {lockfd}>&-
            return 5
        fi

        trigger_reconnect "$dst" "$port" "$trigger"

        log "${dst}:${port} waiting ${SETTLE}s for GOST reconnect..."

        sleep "$SETTLE"
    done

    log "============================================================"
    log "${dst}:${port} PREWARM TIMEOUT"

    socket_count="$(count_established "$dst" "$port")"

    log "${dst}:${port} established=${socket_count}"

    if (( socket_count > 1 )); then
        log "${dst}:${port} multiple MTCP sessions detected"
        log "${dst}:${port} restart the corresponding GOST service"

        show_established "$dst" "$port" |
        while IFS= read -r line; do
            log "$line"
        done
    fi

    mapfile -t flows < <(get_flows "$dst" "$port")

    for flow in "${flows[@]}"; do
        log "${dst}:${port} sport=${flow%% *} minrtt=${flow##* }ms"
    done

    log "============================================================"

    exec {lockfd}>&-
    return 1
}

declare -a TARGETS=()

if (( $# == 0 )); then
    usage
    exit 2
fi

if [[ "$1" == *"@"* ]]; then

    for spec in "$@"; do

        if [[ "$spec" != *"@"* ]]; then
            echo "错误: 混合参数格式不受支持"
            usage
            exit 2
        fi

        target="${spec%%@*}"
        trigger="${spec#*@}"

        dst="${target%:*}"
        port="${target##*:}"

        TARGETS+=("${dst}|${port}|${trigger}")
    done

else

    if (( $# % 2 != 0 )); then
        echo "错误: IP/端口参数必须成对出现"
        usage
        exit 2
    fi

    while (( $# > 0 )); do
        TARGETS+=("$1|$2|${PREWARM_ADDR}")
        shift 2
    done
fi

failed=0

for item in "${TARGETS[@]}"; do

    IFS='|' read -r dst port trigger <<<"$item"

    if ! prewarm_one "$dst" "$port" "$trigger"; then
        failed=1
    fi
done

if (( failed != 0 )); then
    log "one or more targets failed"
    exit 1
fi

log "all targets PREWARM SUCCESS"

exit 0

保存后校验语法：

chmod +x /root/mtcp-prewarm.sh
bash -n /root/mtcp-prewarm.sh

8. 使用方法

单目标手动执行：

THRESH=40 \
PREWARM_ADDR=127.0.0.1:12000 \
/root/mtcp-prewarm.sh 203.0.113.20 11000

若首次抽到慢路，脚本会 SLOW → KILL → 等旧连接消失 → 重新触发 直到命中快路并连续确认两次，输出 PREWARM SUCCESS。

多目标一次执行：

./mtcp-prewarm.sh \
  203.0.113.20 11000 \
  198.51.100.20 11000

不同目标使用不同本地触发入口：

./mtcp-prewarm.sh \
  203.0.113.20:[email protected]:12000 \
  198.51.100.20:[email protected]:13000

若普通 TCP connect 无法触发实际业务链，可用 PREWARM_CMD 自定义触发命令：

PREWARM_CMD="timeout 1 bash -c 'exec 3<>/dev/tcp/127.0.0.1/12000; sleep 0.3'" \
./mtcp-prewarm.sh 203.0.113.20 11000

不同线路 RTT 档位不同时，建议用独立 systemd service 分别设置各自的 THRESH（例如线路 A 快路 33ms/慢路 51ms 用 THRESH=40，线路 B 快路 46ms/慢路 60ms 用 THRESH=52）。

9. systemd 集成

长期运行建议用 systemd 管理 GOST：开机启动、异常退出自动拉起、日志统一记录，并在每次启动后自动执行一次 ECMP 预热（ExecStartPost）。

前置 GOST （主线路）：

cat >/etc/systemd/system/gost-mtcp-main.service <<'EOF'
[Unit]
Description=GOST MTCP Main
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
WorkingDirectory=/root/gost

ExecStart=/root/gost/gost -D -C /root/gost/cn.yaml

Environment="PREWARM_ADDR=127.0.0.1:12000"
Environment="THRESH=40"

ExecStartPost=/bin/bash -lc '/root/mtcp-prewarm.sh 203.0.113.20 11000 || true'

Restart=on-failure
RestartSec=2
TimeoutStartSec=120

KillSignal=SIGTERM

StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
EOF

启用与查看：

systemctl daemon-reload
systemctl enable --now gost-mtcp-main
systemctl status gost-mtcp-main --no-pager
journalctl -u gost-mtcp-main -n 100 -l --no-pager

ExecStartPost 末尾的 || true 是为了防止预热失败被 systemd 判定为启动失败，进而触发 Restart=on-failure 形成"启动→预热失败→service failed→重启→再次预热失败"的循环；加上 || true 后预热失败只写入 journal ，GOST 主进程照常运行，留给人工检查。

10. 验证与常见问题

验证只有一个 MTCP 且命中快路：

ss -ntp "dst 203.0.113.20:11000"   # ESTAB 数量应为 1
ss -tin "dst 203.0.113.20:11000"   # minrtt < THRESH

为什么 flows > 1 不自动处理？ 严格模式下同一目标出现 ≥2 条 outer TCP 属于异常运行态（而非单纯"抽到慢路"），脚本报 PREWARM ERROR 并停止，不猜测删哪条；更安全的处理是 systemctl restart 对应服务，让 ExecStartPost 重新走一次干净预热。

为什么不再用 cron 周期性踢慢路？ 一条 MTCP TCP 上可能承载多个业务 stream ，运行期间 ss -K 相当于主动重置整个 session 。原则改为：启动阶段允许检测/淘汰/重抽，正常运行阶段不主动干预；确需重建时用 systemctl restart 触发完整的启动预热流程。

运行一段时间后 RTT 变高怎么办？ 需区分两种情况：若还是原来那条 MTCP TCP （本地端口不变）且 minrtt 仍处于原来的低延迟档，只是当前 rtt 因拥塞/重传等临时抖动升高，无需处理；若本地端口已变化，说明 MTCP 外层 TCP 已重新建立并可能重新抽到慢路，此时也不应重新引入周期性 FastPath ，而是通过 systemctl restart 重新走一次启动预热。

为什么 mtr -T 仍可能看到多条路径？ mtr -T 每次探测都是新建 TCP 五元组，会重新参与 ECMP ，不会复用已建立的 GOST MTCP 连接，因此不能用它判断 MTCP 实际路径，应以 ss -tin 的 minrtt 为准。

文中公网 IP 、端口均为示例，实际使用请按自己线路的 RTT 分布调整。