feat(knowledgeBase): 为日记批处理入库引入分段耗时(Phase Breakdown)可观测性日志 - #464
Closed
infinite-vector wants to merge 1 commit into
Closed
feat(knowledgeBase): 为日记批处理入库引入分段耗时(Phase Breakdown)可观测性日志#464infinite-vector wants to merge 1 commit into
infinite-vector wants to merge 1 commit into
Conversation
Owner
|
这个现在感觉没必要了吧? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景与意图
在排查日记并发批处理引发事件循环阻塞过程中,发现 _flushBatch 内部从 SQLite 事务提交、Rust 原子索引差分发布、Tag/日期元数据维护到矩阵重排判定,缺乏细粒度的分段可观测性指标。
当现场出现长时间挂起时,单纯依赖外层的看门狗仅能感知到 lag 发生,无法在不打桩的情况下快速界定到底是底层事务锁表、原生 Rust 邻居挂接、还是日期元数据正则扫盘所致。
改动内容
在 modules/knowledgeBase/ingestionPipeline.js 的 _flushBatch() 执行主干中植入零额外 I/O 开销的微秒级时间戳标尺:
SQLite: 统计 this.db.transaction() 执行纯物理写入与索引维护的时间;
applyChunkDelta: 统计 Rust 原生向量图差分挂接与旧节点删除的总耗时;
Tag&Date: 统计全局 Tag usearch 增量与日记日期元数据缓存维护耗时;
scheduleMatrix: 统计 1% 阈值与共现网络重排调度开销。
并在批次完成时打出结构化结算单:
[KnowledgeBase] ⏱️ Phase Breakdown: SQLite=xx ms, applyChunkDelta=xxx ms, Tag&Date=xxx ms, scheduleMatrix=xxx ms (Total=xxx ms)
验证与开销
使用原生 Date.now()(仅纳秒级 rdtsc 读取),只在批次结束汇总打印单行,不侵入任何内部循环热路径;
添加的日志log加都加了,干脆pr了