Skip to content

fix(docker): taosd 被 OOM 杀掉时让容器退出，让 --restart 拉起来 (#34179) #35368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uk0 wants to merge 3 commits into
base: main
Choose a base branch
from
+115 −27
Open

fix(docker): taosd 被 OOM 杀掉时让容器退出，让 --restart 拉起来 (#34179) #35368

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
35e393a
fix(docker): tear container down when any TDengine child dies (#34179)
uk0 May 27, 2026
7002c43
fix(docker): install EXIT trap so cleanup covers all exit paths (#34179)
uk0 May 27, 2026
2cebb43
fix(docker): fail fast if taosd/sidecars die during startup (#34179)
uk0 May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 45 additions & 8 deletions packaging/docker/bin/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,19 @@ fi

NEEDS_INITDB=0

# PIDs of every long-running child we start. Used by the watchdog at the bottom
# of this script to take the whole container down if any single one of them
# dies (#34179). Without this, taosd being OOM-killed would leave the container
# alive (taosadapter still up) and Docker's --restart would never fire.
PIDS=""

# if dnode has been created or has mnode ep set or the host is first ep or not for cluster, just start.
if [ -f "$DATA_DIR/dnode/dnode.json" ] ||
[ -f "$DATA_DIR/dnode/mnodeEpSet.json" ] ||
[ "$FQDN" = "$FIRST_EP_HOST" ]; then
echo "start taosd with mnode ep set"
taosd &
PIDS="$PIDS $!"
while true; do
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

@gemini-code-assist gemini-code-assist Bot May 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

在 taosd 启动后的等待循环中，如果 taosd 因为配置错误或端口冲突等原因在启动时立即崩溃，taos --check 将永远无法成功，导致此 while true 循环变成死循环，容器将无限期挂起而无法触发 Docker 的 --restart 策略。

建议在循环中保存 taosd 的 PID，并在每次循环时检查其是否仍在运行。如果已退出，则应立即报错并退出容器，实现快速失败（Fail-Fast）。

Suggested change
taosd &
PIDS="$PIDS $!"
while true; do
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")
taosd &
TAOSD_PID=$!
PIDS="$PIDS $TAOSD_PID"
while true; do
if ! kill -0 "$TAOSD_PID" 2>/dev/null; then
echo "taosd died during startup"
exit 1
fi
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

@uk0 uk0 May 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recheck

echo ${es}
Expand Down Expand Up @@ -88,6 +95,7 @@ else
if [ "$TAOS_FIRST_EP" = "" ]; then
echo "run TDengine with single node."
taosd &
PIDS="$PIDS $!"
fi
while true; do
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

@gemini-code-assist gemini-code-assist Bot May 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

同样地，在单节点或非首节点初始化的 else 分支中，如果 taosd 启动失败，此处的 while true 循环也会陷入死循环。

建议在此处也保存 taosd 的 PID，并在循环中进行健康检查，以避免容器在启动失败时无限挂起。

Suggested change
if [ "$TAOS_FIRST_EP" = "" ]; then
echo "run TDengine with single node."
taosd &
PIDS="$PIDS $!"
fi
while true; do
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")
TAOSD_PID=""
if [ "$TAOS_FIRST_EP" = "" ]; then
echo "run TDengine with single node."
taosd &
TAOSD_PID=$!
PIDS="$PIDS $TAOSD_PID"
fi
while true; do
if [ -n "$TAOSD_PID" ] && ! kill -0 "$TAOSD_PID" 2>/dev/null; then
echo "taosd died during startup"
exit 1
fi
es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check | grep "^[0-9]*:")

@uk0 uk0 May 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recheck

Expand All @@ -101,27 +109,30 @@ else
done
fi

if [ "$DISABLE_ADAPTER" = "0" ]; then
which taosadapter >/dev/null && taosadapter &
if [ "$DISABLE_ADAPTER" = "0" ] && which taosadapter >/dev/null; then
taosadapter &
PIDS="$PIDS $!"
# wait for 6041 port ready
for _ in $(seq 1 20); do
curl -sf http://localhost:6041/metrics && break
sleep 0.5
done

@gemini-code-assist gemini-code-assist Bot May 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

如果 taosadapter 在启动时立即崩溃（例如由于端口冲突或配置错误），curl 探测会持续失败，导致脚本在 seq 1 20 循环中等待整整 10 秒（20 * 0.5s）才会继续。

建议在等待循环中加入进程存活检查（kill -0）。如果进程已经退出，应立即中断等待并报错退出，实现快速失败（Fail-Fast）。

Suggested change
taosadapter &
PIDS="$PIDS $!"
# wait for 6041 port ready
for _ in $(seq 1 20); do
curl -sf http://localhost:6041/metrics && break
sleep 0.5
done
taosadapter &
ADAPTER_PID=$!
PIDS="$PIDS $ADAPTER_PID"
# wait for 6041 port ready
for _ in $(seq 1 20); do
if ! kill -0 "$ADAPTER_PID" 2>/dev/null; then
echo "taosadapter died during startup"
exit 1
fi
curl -sf http://localhost:6041/metrics && break
sleep 0.5
done

@uk0 uk0 May 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recheck

fi

if [ "$DISABLE_KEEPER" = "0" ]; then
if [ "$DISABLE_KEEPER" = "0" ] && which taoskeeper >/dev/null; then
sleep 3
which taoskeeper >/dev/null && taoskeeper &
taoskeeper &
PIDS="$PIDS $!"
# wait for 6043 port ready
for _ in $(seq 1 20); do
curl -sf http://localhost:6043/metrics && break
sleep 0.5
done

@gemini-code-assist gemini-code-assist Bot May 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

同样地，对于 taoskeeper，如果启动即崩溃，等待循环会白白消耗 10 秒。建议加入进程存活检查以实现快速失败。

Suggested change
taoskeeper &
PIDS="$PIDS $!"
# wait for 6043 port ready
for _ in $(seq 1 20); do
curl -sf http://localhost:6043/metrics && break
sleep 0.5
done
taoskeeper &
KEEPER_PID=$!
PIDS="$PIDS $KEEPER_PID"
# wait for 6043 port ready
for _ in $(seq 1 20); do
if ! kill -0 "$KEEPER_PID" 2>/dev/null; then
echo "taoskeeper died during startup"
exit 1
fi
curl -sf http://localhost:6043/metrics && break
sleep 0.5
done

@uk0 uk0 May 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recheck

fi

if [ "$DISABLE_EXPLORER" = "0" ]; then
which taos-explorer >/dev/null && taos-explorer &
if [ "$DISABLE_EXPLORER" = "0" ] && which taos-explorer >/dev/null; then
taos-explorer &
PIDS="$PIDS $!"
# wait for 6060 port ready
for _ in $(seq 1 20); do
curl -sf http://localhost:6060/metrics && break
Expand Down Expand Up @@ -158,5 +169,31 @@ fi

sh -c "taos -p'$TAOS_ROOT_PASSWORD' -h $FIRST_EP_HOST -P $FIRST_EP_PORT -s 'create snode on dnode 1;'"

trap 'echo "Received stop signal, killing children"; pkill -P $$ || true; exit 0' SIGINT SIGTERM
wait
trap 'echo "Received stop signal, killing children"; pkill -P $$ 2>/dev/null || true; exit 0' SIGINT SIGTERM

# Watchdog: if any of taosd / taosadapter / taoskeeper / taos-explorer dies
# (e.g. taosd OOM-killed), tear the whole container down so Docker's --restart
# can bring it back. Before this, only taosd dying would silently leave
# taosadapter accepting requests it could not fulfill (#34179).
#
# Polling instead of `wait -n` keeps this compatible with bash versions that
# do not support `-n`; the 2s tick is negligible CPU.
if [ -z "$PIDS" ]; then
# cluster non-first node with every sidecar disabled — nothing to watch.
# Stay alive so the trap above can still catch docker stop signals.
echo "No background TDengine processes to watch; idling"
while true; do sleep 3600; done
fi

echo "All services started, watching child processes: $PIDS"
while true; do
for pid in $PIDS; do
if ! kill -0 "$pid" 2>/dev/null; then
echo "child process $pid exited; shutting container down so Docker can restart it"
pkill -P $$ 2>/dev/null || true
sleep 1
exit 1
fi
done
sleep 2
done
Loading