论文：VSAG: An Optimized Search Framework for Graph-based Approximate Nearest Neighbor Search（PVLDB 18(12), 5017-5030, 2025）
原文：https://www.vldb.org/pvldb/vol18/p5017-cheng.pdf
DOI：10.14778/3750601.3750624
Artifact / Code：论文中给出 https://github.com/antgroup/vsag（GitHub: antgroup/vsag）
VSAG 是一个面向图结构 ANNS（Approximate Nearest Neighbor Search）的生产级优化框架。它不是提出全新的图索引结构，而是针对“线上真实瓶颈”做了三类工程优化：内存访问、参数调优、距离计算，并在保持精度（召回）前提下，把吞吐做到了相对现有库（如 HNSWlib）的数倍提升。
图 ANNS（如 HNSW、VAMANA 等）在查询阶段通常是“从入口点出发做图遍历 + 维护候选集 + 多次距离计算”。论文指出三类常见瓶颈：
论文在引言里用一个具体实验把问题说得很硬：
这组数字非常典型：图 ANNS 的线上性能往往首先是硬件行为（缓存/带宽/随机访问）决定的，其次才是“算法步数”。
graph TD
D1Start[One query] --> D1Core[Graph ANNS search core]
D1Core --> D1MemA
D1Core --> D1DistA
D1Core --> D1TuneA
subgraph D1MemG[Memory access bottleneck]
D1MemA[Random vector neighbor access]
D1MemB[High L3 cache miss]
D1MemC[CPU stall waiting memory]
D1MemS1[L3 miss 67.42 percent]
D1MemS2[Time share 63.02 percent]
D1MemA --> D1MemB --> D1MemC
D1MemC --> D1MemS1
D1MemC --> D1MemS2
end
subgraph D1DistG[Distance compute bottleneck]
D1DistA[Many candidates high dimension]
D1DistS1[Time share 26.12 percent]
D1DistA --> D1DistS1
end
subgraph D1TuneG[Parameter tuning bottleneck]
D1TuneA[Sensitive parameters affect recall qps latency]
D1TuneS1[QPS gain 42.6 percent]
D1TuneS2[Tuning cost 60 hours plus]
D1TuneA --> D1TuneS1
D1TuneA --> D1TuneS2
end
%% Use unique class names per diagram (avoid cross-diagram collisions in some renderers)
classDef D1_start fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
classDef D1_core fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
classDef D1_box fill:#ffffff,stroke:#9e9e9e,stroke-width:1px
classDef D1_metric fill:#fff9c4,stroke:#f9a825,stroke-width:2px
class D1Start D1_start
class D1Core D1_core
class D1MemA D1_box
class D1MemB D1_box
class D1MemC D1_box
class D1DistA D1_box
class D1TuneA D1_box
class D1MemS1 D1_metric
class D1MemS2 D1_metric
class D1DistS1 D1_metric
class D1TuneS1 D1_metric
class D1TuneS2 D1_metric
论文把 VSAG 的优化归纳为三部分（摘要中给出）：
目标是减少 cache miss，让“图遍历 + 向量读取”尽可能连续、可预取。
直觉上可以理解为：
（论文原文用 “prefetching” 和 “cache-friendly vector organization” 概括。）
graph TD
D2Q[Query q] --> D2Init[Init entry point]
D2Init --> D2Expand[Expand node u]
subgraph D2LoopG[Traversal loop]
D2Expand --> D2Read[Read neighbor list]
D2Read --> D2PF[Prefetch]
D2PF --> D2Dist[Compute distance]
D2Dist --> D2Heap[Update candidate set]
D2Heap --> D2Check[Check stop condition]
D2Check --> D2More[Continue]
D2More --> D2Expand
end
D2Check --> D2Out[Output top k]
classDef D2_start fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
classDef D2_step fill:#ffffff,stroke:#9e9e9e,stroke-width:1px
classDef D2_mem fill:#fff3e0,stroke:#f57c00,stroke-width:2px
classDef D2_compute fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
classDef D2_ctrl fill:#fff9c4,stroke:#f9a825,stroke-width:2px
class D2Q D2_start
class D2Out D2_start
class D2Init D2_step
class D2Expand D2_step
class D2Read D2_step
class D2Heap D2_step
class D2PF D2_mem
class D2Dist D2_compute
class D2Check D2_ctrl
这张图对应一个很现实的工程目标：让“下一批要访问的向量”尽可能提前进入 cache，让 CPU 算距离时不至于完全空转等内存。
论文强调：生产环境里“为了调参反复 rebuild 索引”非常贵，因此 VSAG 的目标是自动选择性能更优的参数，并避免把调参成本放大到不可接受。
可以把它理解成把“参数选择”做成一条可重复的离线/半离线流水线：
graph TD
D3Start[Goal: higher QPS at same recall] --> D3Constraint[Constraints: recall target, latency SLO, memory budget]
D3Constraint --> D3Space[Parameter space: candidate configs]
subgraph D3Eval[Evaluation and selection loop]
D3Workload[Sample workload: queries and dataset slice]
D3Run[Run benchmark: build/search trials]
D3Metric[Compute metrics: recall, qps, p95, p99]
D3Select[Select best config: objective or Pareto]
D3Workload --> D3Run --> D3Metric --> D3Select
end
D3Space --> D3Workload
D3Select --> D3Out[Output config: tuned parameters]
D3Out --> D3Benefit[Benefit: fewer rebuild iterations]
classDef D3_start fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
classDef D3_step fill:#ffffff,stroke:#9e9e9e,stroke-width:1px
classDef D3_dec fill:#fff9c4,stroke:#f9a825,stroke-width:2px
class D3Start D3_start
class D3Out D3_start
class D3Constraint D3_step
class D3Space D3_step
class D3Workload D3_step
class D3Run D3_step
class D3Metric D3_step
class D3Benefit D3_step
class D3Select D3_dec
（这里用笔记化的抽象表示，具体策略以论文为准。）
很多 ANNS 系统在工程上真正难的不是实现 HNSW，而是：
论文在 Table 1 里把“Parameter tuning cost”当成一项指标，是很强的信号：调参成本已经是系统成本的一部分。
VSAG 的距离计算优化强调三点（摘要原话）：
可以把它理解成“先用更便宜的表示快速筛，再在必要时做更精确的计算”：
graph TD
D4Q[Query vector q] --> D4Cand[Candidates C]
D4Cand --> D4Coarse[Stage A: coarse scoring]
D4Coarse --> D4Rep[Low precision / SQ]
D4Rep --> D4ScoreA[Approx distance]
D4ScoreA --> D4Keep[Keep TopM or TopK]
D4Keep --> D4Refine[Stage B: refine optional]
D4Refine --> D4ScoreB[High precision distance]
D4ScoreB --> D4Out[TopK results]
%% Use the most compatible class syntax: no trailing semicolons, no comma-separated node lists
classDef D4_start fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
classDef D4_step fill:#ffffff,stroke:#9e9e9e,stroke-width:1px
classDef D4_coarse fill:#fff3e0,stroke:#f57c00,stroke-width:2px
classDef D4_refine fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
classDef D4_keep fill:#fff9c4,stroke:#f9a825,stroke-width:2px
class D4Q D4_start
class D4Out D4_start
class D4Cand D4_step
class D4Coarse D4_coarse
class D4Rep D4_coarse
class D4ScoreA D4_coarse
class D4Refine D4_refine
class D4ScoreB D4_refine
class D4Keep D4_keep
你可以把它理解成“分两段算距离”：
这样距离计算的重活被限制在更小的集合里，更容易把总成本压下去。
论文在摘要与表格中强调了“同精度下吞吐提升”和“瓶颈占比下降”。在给出的对比表（GIST1M）里（论文 Table 1）：
注：以上数字来自论文原文的摘要与表格（见原文 PDF）。
下面把论文 Table 1 的核心对比摘出来（便于复用和讨论）：
| Metric (GIST1M) | IVFPQFS | HNSW | VSAG |
|---|---|---|---|
| Memory Footprint | 3.8G | 4.0G | 4.5G |
| Recall@10 (QPS=2000) | 84.57% | 59.46% | 89.80% |
| QPS (Recall@10=90%) | 1195 | 511.9 | 2167.3 |
| Distance Computation Cost | 0.71ms | 1.62ms | 0.1ms |
| L3 Cache Miss Rate | 13.98% | 94.46% | 39.23% |
| Parameter Tuning Cost | >20h | >60h | 2.92h |
| Parameter Tuning | manual | manual | auto |
这张表里最“值钱”的三行是：
很多系统以为“换个更强的图结构”就够了，但生产中最大的敌人往往是：
VSAG 的价值在于把这些问题放进一套系统化的优化框架里。
当索引构建很贵时，调参必须尽量做到：
否则工程上根本落不了地。
VSAG 把量化与硬件优化放进距离计算主链路中，目标是降低每一次距离计算的真实成本（而不是只减少存储占用）。
https://www.vldb.org/pvldb/vol18/p5017-cheng.pdfhttps://github.com/antgroup/vsagIndexLib 的文件系统抽象通过统一的接口屏蔽底层存储差异，支持多种存储后端（本地文件系统、分布式文件系统、内存文件系统等）。从文件系统抽象到存储格式的完整机制如下：
flowchart TB
Start([文件系统抽象架构<br/>File System Abstraction Architecture]) --> Layer1[第一层：接口抽象层<br/>Layer 1: Interface Abstraction]
subgraph L1["第一层：接口抽象层 Layer 1: Interface Abstraction"]
direction TB
L1_1[IFileSystem<br/>文件系统接口<br/>统一文件系统操作入口]
L1_2[IDirectory<br/>目录接口<br/>目录和文件管理接口]
L1_3[Storage<br/>存储抽象接口<br/>底层存储封装接口]
end
Layer1 --> Layer2[第二层：文件操作层<br/>Layer 2: File Operations]
subgraph L2["第二层：文件操作层 Layer 2: File Operations"]
direction TB
L2_1[FileReader<br/>文件读取器<br/>提供文件读取功能]
L2_2[FileWriter<br/>文件写入器<br/>提供文件写入功能]
end
Layer2 --> Layer3[第三层：实现层<br/>Layer 3: Implementations]
subgraph L3["第三层：实现层 Layer 3: Implementations"]
direction TB
L3_1[本地文件系统<br/>Local File System<br/>PosixFileSystem实现]
L3_2[分布式文件系统<br/>Distributed File System<br/>HDFS Pangu实现]
L3_3[内存文件系统<br/>Memory File System<br/>MemFileSystem实现]
end
Layer3 --> End([统一存储访问<br/>Unified Storage Access])
Layer1 -.->|包含| L1
Layer2 -.->|包含| L2
Layer3 -.->|包含| L3
L1_1 -.->|创建| L2_1
L1_2 -.->|创建| L2_1
L1_3 -.->|创建| L2_1
L1_1 -.->|创建| L2_2
L1_2 -.->|创建| L2_2
L1_3 -.->|创建| L2_2
L2_1 -.->|基于| L3_1
L2_1 -.->|基于| L3_2
L2_1 -.->|基于| L3_3
L2_2 -.->|基于| L3_1
L2_2 -.->|基于| L3_2
L2_2 -.->|基于| L3_3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style Layer1 fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Layer2 fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Layer3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style L1 fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style L1_1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L1_2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L1_3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L2 fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style L2_1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style L2_2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style L3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style L3_1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L3_2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L3_3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
IndexLib 的文件系统抽象包括以下核心概念，通过统一的接口屏蔽底层存储差异，支持多种存储后端。让我们先通过类图来理解文件系统抽象的整体架构：
classDiagram
class IFileSystem {
<<interface>>
+ Init()
+ MountVersion()
+ MountDir()
+ MountFile()
+ CreateFileWriter()
+ CreateFileReader()
}
class IDirectory {
<<interface>>
+ CreateFileWriter()
+ CreateFileReader()
+ MakeDirectory()
+ GetDirectory()
+ RemoveFile()
+ RemoveDirectory()
+ Rename()
+ IsExist()
+ ListDir()
+ GetFileLength()
}
class FileReader {
<<interface>>
+ Open()
+ Close()
+ Read()
+ Prefetch()
+ ReadAsync()
+ GetLength()
+ GetLogicalPath()
+ GetPhysicalPath()
}
class FileWriter {
<<interface>>
+ Open()
+ Close()
+ Write()
+ ReserveFile()
+ Truncate()
+ GetLength()
+ GetLogicalPath()
+ GetPhysicalPath()
}
class Storage {
<<interface>>
+ CreateInputStorage()
+ CreateOutputStorage()
+ CreateFileReader()
+ CreateFileWriter()
+ Sync()
+ GetStorageType()
}
IFileSystem --> IDirectory : 创建
IFileSystem --> FileReader : 创建
IFileSystem --> FileWriter : 创建
IDirectory --> FileReader : 创建
IDirectory --> FileWriter : 创建
Storage --> FileReader : 创建
Storage --> FileWriter : 创建
文件系统抽象的核心组件：
文件系统抽象的核心组件包括 IFileSystem、IDirectory、FileReader、FileWriter，它们之间的关系如下：
flowchart TB
Start([文件系统抽象架构<br/>File System Abstraction Architecture]) --> InterfaceLayer[接口层<br/>Interface Layer]
subgraph InterfaceGroup["接口层 Interface Layer"]
direction TB
I1[IFileSystem<br/>文件系统接口<br/>统一文件系统操作入口]
I2[IDirectory<br/>目录接口<br/>目录和文件管理接口]
I3[Storage<br/>存储抽象接口<br/>底层存储封装接口]
end
InterfaceLayer --> OperationLayer[操作层<br/>Operation Layer]
subgraph OperationGroup["操作层 Operation Layer"]
direction TB
O1[FileReader<br/>文件读取器<br/>提供文件读取功能]
O2[FileWriter<br/>文件写入器<br/>提供文件写入功能]
end
OperationLayer --> End([统一文件操作<br/>Unified File Operations])
InterfaceLayer -.->|包含| InterfaceGroup
OperationLayer -.->|包含| OperationGroup
I1 -.->|创建| O1
I1 -.->|创建| O2
I2 -.->|创建| O1
I2 -.->|创建| O2
I3 -.->|创建| O1
I3 -.->|创建| O2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style InterfaceLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style OperationLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style InterfaceGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style I1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style OperationGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style O1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
文件系统抽象在 IndexLib 中起到关键作用，是存储管理的基础。下面通过流程图展示文件系统抽象的整体工作流程：
flowchart TB
Start([开始<br/>Start]) --> InitLayer[初始化层<br/>Initialization Layer]
subgraph InitGroup["初始化 Initialization"]
direction TB
I1[Init 文件系统<br/>Initialize File System<br/>设置文件系统选项]
I2[挂载版本<br/>Mount Version<br/>挂载指定版本]
I3[挂载目录<br/>Mount Directory<br/>挂载目录路径]
end
InitLayer --> WriteLayer[写入操作层<br/>Write Operation Layer]
subgraph WriteGroup["写入操作 Write Operation"]
direction TB
W1{需要创建写入器?<br/>Need Writer?}
W2[获取目录<br/>Get Directory<br/>获取目录对象]
W3[创建文件写入器<br/>Create File Writer<br/>创建写入器对象]
W4[写入文件<br/>Write File<br/>写入文件数据]
W5[关闭写入器<br/>Close Writer<br/>释放资源]
end
WriteLayer --> ReadLayer[读取操作层<br/>Read Operation Layer]
subgraph ReadGroup["读取操作 Read Operation"]
direction TB
R1{需要创建读取器?<br/>Need Reader?}
R2[获取目录<br/>Get Directory<br/>获取目录对象]
R3[创建文件读取器<br/>Create File Reader<br/>创建读取器对象]
R4[读取文件<br/>Read File<br/>读取文件数据]
R5[关闭读取器<br/>Close Reader<br/>释放资源]
end
ReadLayer --> End([结束<br/>End])
InitLayer -.->|包含| InitGroup
WriteLayer -.->|包含| WriteGroup
ReadLayer -.->|包含| ReadGroup
I1 --> I2
I2 --> I3
I3 --> W1
W1 -->|是| W2
W1 -->|否| R1
W2 --> W3
W3 --> W4
W4 --> W5
W5 --> R1
R1 -->|是| R2
R1 -->|否| End
R2 --> R3
R3 --> R4
R4 --> R5
R5 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style InitLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style WriteLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ReadLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style InitGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style I1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style WriteGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style W1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style W2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style W3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style W4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style W5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ReadGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style R1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R5 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
文件系统抽象的核心作用：
IFileSystem 是文件系统接口，定义在 file_system/IFileSystem.h 中。它提供了文件系统的基本操作，包括初始化、挂载、文件读写等。IFileSystem 的完整接口定义如下：
classDiagram
class IFileSystem {
<<interface>>
+ Init()
+ MountVersion()
+ MountDir()
+ MountFile()
+ CreateFileWriter()
+ CreateFileReader()
+ GetDirectory()
+ RemoveFile()
+ RemoveDirectory()
+ IsExist()
+ ListDir()
+ GetFileLength()
}
class FileSystemOptions {
+ string rootPath
+ bool enableCache
+ size_t cacheSize
+ FSStorageType storageType
}
class MountOption {
+ FSMountType mountType
+ bool readOnly
+ bool lazyLoad
}
class WriterOption {
+ bool atomicWrite
+ bool syncOnClose
+ size_t bufferSize
}
class ReaderOption {
+ bool useCache
+ bool prefetch
+ size_t bufferSize
}
IFileSystem --> FileSystemOptions : 使用
IFileSystem --> MountOption : 使用
IFileSystem --> WriterOption : 使用
IFileSystem --> ReaderOption : 使用
IFileSystem 的完整定义：
// file_system/IFileSystem.h
class IFileSystem : autil::NoMoveable
{
public:
// 初始化文件系统
virtual FSResult<void> Init(const FileSystemOptions& fileSystemOptions) = 0;
// 挂载版本：将物理路径映射到逻辑路径
virtual FSResult<void> MountVersion(
const std::string& physicalRoot, // 物理根路径
versionid_t versionId, // 版本ID
const std::string& logicalPath, // 逻辑路径
MountOption mountOption) = 0;
// 挂载目录：支持目录级别的挂载
virtual FSResult<void> MountDir(
const std::string& physicalRoot, // 物理根路径
const std::string& physicalPath, // 物理路径
const std::string& logicalPath, // 逻辑路径
MountOption mountOption) = 0;
// 挂载文件：支持文件级别的挂载
virtual FSResult<void> MountFile(
const std::string& physicalRoot, // 物理根路径
const std::string& physicalPath, // 物理路径
const std::string& logicalPath, // 逻辑路径
FSMountType mountType) = 0;
// 创建文件写入器
virtual FSResult<std::shared_ptr<FileWriter>> CreateFileWriter(
const std::string& rawPath, // 原始路径（逻辑路径或物理路径）
const WriterOption& writerOption) = 0;
// 创建文件读取器
virtual FSResult<std::shared_ptr<FileReader>> CreateFileReader(
const std::string& rawPath, // 原始路径（逻辑路径或物理路径）
const ReaderOption& readerOption) = 0;
// 获取目录
virtual FSResult<std::shared_ptr<IDirectory>> GetDirectory(
const std::string& logicalPath) = 0;
// 删除文件
virtual FSResult<void> RemoveFile(
const std::string& logicalPath,
const RemoveOption& removeOption) = 0;
// 删除目录
virtual FSResult<void> RemoveDirectory(
const std::string& logicalPath,
const RemoveOption& removeOption) = 0;
// 检查文件是否存在
virtual FSResult<bool> IsExist(const std::string& logicalPath) const = 0;
// 列出目录
virtual FSResult<void> ListDir(
const std::string& logicalPath,
const ListOption& listOption,
std::vector<std::string>& fileList) const = 0;
// 获取文件长度
virtual FSResult<size_t> GetFileLength(const std::string& logicalPath) const = 0;
// 同步文件系统
virtual FSResult<void> Sync(bool waitFinish = true) = 0;
// 获取文件系统类型
virtual FSStorageType GetStorageType() const = 0;
};
IFileSystem 的关键方法详解：
IFileSystem 接口：提供文件系统的基本操作：
flowchart TB
Start([IFileSystem 接口<br/>IFileSystem Interface]) --> MethodLayer[核心方法层<br/>Core Methods Layer]
subgraph MethodGroup["核心方法 Core Methods"]
direction TB
M1[Init<br/>初始化文件系统<br/>设置文件系统选项<br/>初始化底层存储]
M2[MountVersion/MountDir<br/>挂载版本和目录<br/>路径映射<br/>挂载管理]
M3[CreateFileWriter/Reader<br/>创建文件操作器<br/>创建写入器<br/>创建读取器]
end
MethodLayer --> ComponentLayer[相关组件层<br/>Related Components Layer]
subgraph ComponentGroup["相关组件 Related Components"]
direction TB
C1[FileSystemOptions<br/>文件系统选项<br/>配置参数<br/>存储类型]
C2[IDirectory<br/>目录接口<br/>目录操作<br/>文件管理]
end
ComponentLayer --> End([文件系统操作<br/>File System Operations])
MethodLayer -.->|包含| MethodGroup
ComponentLayer -.->|包含| ComponentGroup
M1 -.->|使用| C1
M2 -.->|创建| C2
M3 -.->|创建| C2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style MethodLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ComponentLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style MethodGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ComponentGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
文件系统抽象通过逻辑路径和物理路径管理文件，实现路径抽象和版本管理。路径映射的机制如下：
flowchart TB
Start([文件操作请求<br/>File Operation Request]) --> PathLayer[路径处理层<br/>Path Processing Layer]
subgraph PathGroup["路径处理 Path Processing"]
direction TB
P1{路径类型?<br/>Path Type?}
P2[解析逻辑路径<br/>Resolve Logical Path<br/>查找挂载点]
P3[直接访问<br/>Direct Access<br/>使用物理路径]
end
PathLayer --> MountLayer[挂载检查层<br/>Mount Check Layer]
subgraph MountGroup["挂载检查 Mount Check"]
direction TB
M1{检查挂载点<br/>Check Mount Point}
M2[获取物理路径<br/>Get Physical Path<br/>从挂载点获取]
M3[合并路径<br/>Merge Path<br/>组合物理路径]
M4[返回错误<br/>Return Error<br/>未找到挂载点]
end
MountLayer --> AccessLayer[文件访问层<br/>File Access Layer]
subgraph AccessGroup["文件访问 File Access"]
direction TB
A1[访问文件<br/>Access File<br/>执行文件操作]
end
AccessLayer --> End([结束<br/>End])
PathLayer -.->|包含| PathGroup
MountLayer -.->|包含| MountGroup
AccessLayer -.->|包含| AccessGroup
P1 -->|逻辑路径| P2
P1 -->|物理路径| P3
P2 --> M1
M1 -->|已挂载| M2
M1 -->|未挂载| M4
M2 --> M3
M3 --> A1
P3 --> A1
M4 --> End
A1 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style PathLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style MountLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style AccessLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style PathGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style MountGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M4 fill:#ef5350,stroke:#c62828,stroke-width:2px
style AccessGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style A1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
路径映射的实现：
// file_system/FileSystem.cpp
class FileSystem : public IFileSystem
{
private:
struct MountPoint {
std::string physicalPath; // 物理路径
std::string logicalPath; // 逻辑路径
FSMountType mountType; // 挂载类型
bool readOnly; // 是否只读
};
std::map<std::string, MountPoint> _mountPoints; // 挂载点映射
public:
FSResult<std::string> ResolvePath(const std::string& logicalPath) const {
// 1. 查找最长的匹配挂载点
std::string bestMatch;
size_t bestMatchLen = 0;
for (const auto& [logical, mount] : _mountPoints) {
if (logicalPath.find(logical) == 0) {
if (logical.length() > bestMatchLen) {
bestMatch = logical;
bestMatchLen = logical.length();
}
}
}
if (bestMatch.empty()) {
return FSResult<std::string>::Error("No mount point found");
}
// 2. 替换逻辑路径为物理路径
const auto& mount = _mountPoints.at(bestMatch);
std::string relativePath = logicalPath.substr(bestMatch.length());
std::string physicalPath = mount.physicalPath + relativePath;
return FSResult<std::string>::OK(physicalPath);
}
};
路径映射的关键概念：
/data/indexlib/version_1/segment_0/index/indexlib/version_1/segment_0/indexMountVersion("/data/indexlib", 1, "/indexlib/v1")MountDir("/data/indexlib/seg0", "/indexlib/seg0")MountFile("/data/indexlib/file", "/indexlib/file")逻辑路径与物理路径：从物理路径到逻辑路径的映射：
flowchart TB
Start([路径映射系统<br/>Path Mapping System]) --> ComponentLayer[组件层<br/>Component Layer]
subgraph ComponentGroup["路径映射组件 Path Mapping Components"]
direction TB
C1[IFileSystem<br/>文件系统接口<br/>提供路径操作接口]
C2[PathMapper<br/>路径映射器<br/>解析和转换路径]
C3[MountTable<br/>挂载表<br/>管理挂载点映射]
end
ComponentLayer --> PathLayer[路径类型层<br/>Path Type Layer]
subgraph PathGroup["路径类型 Path Types"]
direction TB
P1[逻辑路径<br/>Logical Path<br/>逻辑文件系统路径<br/>版本和Segment管理]
P2[物理路径<br/>Physical Path<br/>磁盘实际路径<br/>文件系统路径]
end
PathLayer --> End([路径映射完成<br/>Path Mapping Complete])
ComponentLayer -.->|包含| ComponentGroup
PathLayer -.->|包含| PathGroup
C1 -.->|使用| C2
C2 -.->|查询| C3
C3 -.->|映射到| P1
C3 -.->|映射到| P2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style ComponentLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style PathLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ComponentGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style PathGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style P1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
IndexLib 支持多种文件系统类型，包括本地文件系统、分布式文件系统、内存文件系统等。各种文件系统类型及其关系如下：
flowchart TB
Start([文件系统类型<br/>File System Types]) --> TypeLayer[类型层<br/>Type Layer]
subgraph TypeGroup["文件系统类型 File System Types"]
direction TB
T1[LocalFileSystem<br/>本地文件系统<br/>基于本地磁盘<br/>Posix文件系统]
T2[DistributedFileSystem<br/>分布式文件系统<br/>HDFS Pangu<br/>分布式存储]
T3[MemoryFileSystem<br/>内存文件系统<br/>基于内存<br/>临时存储]
end
TypeLayer --> InterfaceLayer[接口层<br/>Interface Layer]
subgraph InterfaceGroup["实现接口 Implementation Interfaces"]
direction TB
I1[IFileSystem<br/>文件系统接口<br/>统一接口定义<br/>标准操作]
I2[Storage<br/>存储抽象<br/>底层存储封装<br/>存储操作]
end
InterfaceLayer --> End([统一文件系统访问<br/>Unified File System Access])
TypeLayer -.->|包含| TypeGroup
InterfaceLayer -.->|包含| InterfaceGroup
T1 -.->|实现| I1
T2 -.->|实现| I1
T3 -.->|实现| I1
T1 -.->|使用| I2
T2 -.->|使用| I2
T3 -.->|使用| I2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style TypeLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style InterfaceLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style TypeGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style T1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style InterfaceGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style I1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style I2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
文件系统类型：
IDirectory 是目录接口，定义在 file_system/IDirectory.h 中：
// file_system/IDirectory.h
class IDirectory
{
public:
// 创建文件写入器
virtual FSResult<std::shared_ptr<FileWriter>> CreateFileWriter(
const std::string& filePath,
const WriterOption& writerOption) = 0;
// 创建文件读取器
virtual FSResult<std::shared_ptr<FileReader>> CreateFileReader(
const std::string& filePath,
const ReaderOption& readerOption) = 0;
// 创建目录
virtual FSResult<std::shared_ptr<IDirectory>> MakeDirectory(
const std::string& dirPath,
const DirectoryOption& directoryOption) = 0;
// 获取目录
virtual FSResult<std::shared_ptr<IDirectory>> GetDirectory(
const std::string& dirPath) = 0;
// 删除文件
virtual FSResult<void> RemoveFile(
const std::string& filePath,
const RemoveOption& removeOption) = 0;
// 删除目录
virtual FSResult<void> RemoveDirectory(
const std::string& dirPath,
const RemoveOption& removeOption) = 0;
// 重命名
virtual FSResult<void> Rename(
const std::string& srcPath,
const std::shared_ptr<IDirectory>& destDirectory,
const std::string& destPath) = 0;
// 检查文件是否存在
virtual FSResult<bool> IsExist(const std::string& path) const = 0;
// 列出目录
virtual FSResult<void> ListDir(
const std::string& path,
const ListOption& listOption,
std::vector<std::string>& fileList) const = 0;
// 获取文件长度
virtual FSResult<size_t> GetFileLength(const std::string& filePath) const = 0;
};
IDirectory 的关键方法：
IDirectory 接口：提供目录和文件的操作：
flowchart TB
Start([IDirectory 接口<br/>IDirectory Interface]) --> MethodLayer[方法层<br/>Methods Layer]
subgraph FileOpsGroup["文件操作 File Operations"]
direction TB
FO1[CreateFileWriter<br/>创建文件写入器<br/>创建写入器对象]
FO2[CreateFileReader<br/>创建文件读取器<br/>创建读取器对象]
FO3[RemoveFile<br/>删除文件<br/>删除指定文件]
FO4[GetFileLength<br/>获取文件长度<br/>获取文件大小]
FO5[IsExist<br/>检查文件是否存在<br/>检查路径存在性]
end
subgraph DirOpsGroup["目录操作 Directory Operations"]
direction TB
DO1[MakeDirectory<br/>创建目录<br/>创建新目录]
DO2[GetDirectory<br/>获取目录<br/>获取目录对象]
DO3[RemoveDirectory<br/>删除目录<br/>删除指定目录]
DO4[Rename<br/>重命名文件或目录<br/>重命名操作]
DO5[ListDir<br/>列出目录内容<br/>列出文件列表]
end
MethodLayer --> ResultLayer[结果层<br/>Result Layer]
subgraph ResultGroup["操作结果 Operation Results"]
direction TB
R1[文件操作结果<br/>File Operation Results<br/>FileWriter/FileReader对象]
R2[目录操作结果<br/>Directory Operation Results<br/>IDirectory对象/文件列表]
end
ResultLayer --> End([操作完成<br/>Operation Complete])
MethodLayer -.->|包含| FileOpsGroup
MethodLayer -.->|包含| DirOpsGroup
ResultLayer -.->|包含| ResultGroup
FileOpsGroup -.->|返回| R1
DirOpsGroup -.->|返回| R2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style MethodLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ResultLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style FileOpsGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FO1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FO2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FO3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FO4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FO5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style DirOpsGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style DO1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style DO2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style DO3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style DO4 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style DO5 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style ResultGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style R1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
目录操作的流程：
目录操作流程：从创建目录到文件操作的完整流程：
flowchart TB
Start([IDirectory 操作<br/>IDirectory Operations]) --> DirectoryLayer[目录操作层<br/>Directory Operations Layer]
subgraph DirectoryGroup["IDirectory 核心操作 Core Operations"]
direction TB
D1[GetDirectory<br/>获取子目录<br/>获取目录对象]
D2[CreateFileWriter<br/>创建文件写入器<br/>创建写入器对象]
D3[CreateFileReader<br/>创建文件读取器<br/>创建读取器对象]
D4[MakeDirectory<br/>创建目录<br/>创建新目录]
D5[RemoveFile<br/>删除文件<br/>删除指定文件]
D6[RemoveDirectory<br/>删除目录<br/>删除指定目录]
D7[Rename<br/>重命名文件或目录<br/>重命名操作]
D8[IsExist<br/>检查文件是否存在<br/>检查路径存在性]
D9[ListDir<br/>列出目录内容<br/>列出文件列表]
D10[GetFileLength<br/>获取文件长度<br/>获取文件大小]
end
DirectoryLayer --> FileOpsLayer[文件操作层<br/>File Operations Layer]
subgraph FileWriterGroup["FileWriter 操作 FileWriter Operations"]
direction TB
FW1[Write<br/>写入文件数据<br/>写入数据到文件]
FW2[ReserveFile<br/>预留文件空间<br/>预留文件大小]
FW3[Truncate<br/>截断文件<br/>调整文件大小]
end
subgraph FileReaderGroup["FileReader 操作 FileReader Operations"]
direction TB
FR1[Read<br/>读取文件数据<br/>同步读取数据]
FR2[Prefetch<br/>预取文件数据<br/>提前加载数据]
FR3[ReadAsync<br/>异步读取<br/>异步读取数据]
end
FileOpsLayer --> End([操作完成<br/>Operation Complete])
DirectoryLayer -.->|包含| DirectoryGroup
FileOpsLayer -.->|包含| FileWriterGroup
FileOpsLayer -.->|包含| FileReaderGroup
D2 -.->|创建| FileWriterGroup
D3 -.->|创建| FileReaderGroup
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style DirectoryLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style FileOpsLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style DirectoryGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style D1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D6 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D7 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D8 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D9 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D10 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style FileWriterGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FW1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FW2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FW3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FileReaderGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style FR1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style FR2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style FR3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
操作流程：
GetDirectory() 获取目录CreateFileWriter() 创建文件写入器FileWriter::Write() 写入文件CreateFileReader() 创建文件读取器FileReader::Read() 读取文件数据FileReader 是文件读取器，提供文件读取功能，支持同步和异步读取。让我们先通过序列图来理解 FileReader 的完整工作流程：
sequenceDiagram
participant Client
participant FileReader
participant Cache
participant Storage
Client->>FileReader: CreateFileReader(path, option)
FileReader->>FileReader: 解析路径
FileReader->>Storage: 打开文件
Storage-->>FileReader: 文件句柄
FileReader-->>Client: FileReader对象
Client->>FileReader: Read(buffer, length, offset)
FileReader->>Cache: 检查缓存
alt 缓存命中
Cache-->>FileReader: 返回缓存数据
else 缓存未命中
FileReader->>Storage: 读取文件
Storage-->>FileReader: 文件数据
FileReader->>Cache: 更新缓存
end
FileReader-->>Client: 返回读取长度
Client->>FileReader: Prefetch(length, offset)
FileReader->>Storage: 异步预取
Storage-->>FileReader: 预取完成
Client->>FileReader: Close()
FileReader->>Storage: 关闭文件
FileReader->>Cache: 清理缓存
FileReader 的完整定义：
// file_system/file/FileReader.h
class FileReader
{
public:
// 打开文件
virtual FSResult<void> Open() = 0;
// 关闭文件
virtual FSResult<void> Close() = 0;
// 读取文件：同步读取
virtual FSResult<size_t> Read(
void* buffer, // 缓冲区
size_t length, // 读取长度
size_t offset, // 偏移量
ReadOption option = ReadOption()) = 0;
// 预取文件：异步预取，不阻塞
virtual FSResult<size_t> Prefetch(
size_t length, // 预取长度
size_t offset, // 偏移量
ReadOption option) = 0;
// 异步读取：返回 Future
virtual future_lite::Future<FSResult<size_t>> ReadAsync(
void* buffer, // 缓冲区
size_t length, // 读取长度
size_t offset, // 偏移量
ReadOption option) = 0;
// 批量读取：读取多个不连续的区域
virtual FSResult<void> BatchRead(
const std::vector<ReadRequest>& requests,
ReadOption option) = 0;
// 获取文件长度
virtual size_t GetLength() const = 0;
// 获取逻辑路径
virtual const std::string& GetLogicalPath() const = 0;
// 获取物理路径
virtual const std::string& GetPhysicalPath() const = 0;
// 获取文件元数据
virtual FSResult<FileMeta> GetFileMeta() const = 0;
};
FileReader 的实现示例：
// file_system/file/LocalFileReader.cpp
class LocalFileReader : public FileReader
{
private:
std::string _logicalPath;
std::string _physicalPath;
int _fd;
size_t _fileLength;
std::shared_ptr<FileCache> _cache;
public:
FSResult<void> Open() override {
_fd = ::open(_physicalPath.c_str(), O_RDONLY);
if (_fd < 0) {
return FSResult<void>::Error("Failed to open file: " + _physicalPath);
}
// 获取文件长度
struct stat st;
if (fstat(_fd, &st) < 0) {
::close(_fd);
return FSResult<void>::Error("Failed to get file length");
}
_fileLength = st.st_size;
return FSResult<void>::OK();
}
FSResult<size_t> Read(void* buffer, size_t length, size_t offset,
ReadOption option) override {
// 1. 检查缓存
if (option.useCache && _cache) {
auto cached = _cache->Get(_physicalPath, offset, length);
if (cached) {
memcpy(buffer, cached->data(), length);
return FSResult<size_t>::OK(length);
}
}
// 2. 读取文件
ssize_t nread = pread(_fd, buffer, length, offset);
if (nread < 0) {
return FSResult<size_t>::Error("Failed to read file");
}
// 3. 更新缓存
if (option.useCache && _cache) {
_cache->Put(_physicalPath, offset, buffer, nread);
}
return FSResult<size_t>::OK(nread);
}
FSResult<size_t> Prefetch(size_t length, size_t offset, ReadOption option) override {
// 使用 posix_fadvise 预取
int ret = posix_fadvise(_fd, offset, length, POSIX_FADV_WILLNEED);
if (ret != 0) {
return FSResult<size_t>::Error("Failed to prefetch");
}
return FSResult<size_t>::OK(length);
}
future_lite::Future<FSResult<size_t>> ReadAsync(void* buffer, size_t length,
size_t offset, ReadOption option) override {
// 使用异步 IO（如 io_uring）实现
return future_lite::async([=]() {
return Read(buffer, length, offset, option);
});
}
FSResult<void> Close() override {
if (_fd >= 0) {
::close(_fd);
_fd = -1;
}
return FSResult<void>::OK();
}
};
FileReader 的关键特性：
Read() 方法提供同步读取，阻塞直到读取完成
ReadAsync() 方法提供异步读取，不阻塞
Prefetch() 方法提供预取功能，提前加载数据
posix_fadvise 或类似机制FileReader 接口：提供文件读取功能：
flowchart TB
Start([FileReader 核心功能<br/>FileReader Core Features]) --> FeatureLayer[功能层<br/>Features Layer]
subgraph ReadGroup["读取功能 Read Functions"]
direction TB
R1[Read<br/>读取文件数据<br/>同步读取数据]
R2[Prefetch<br/>预取文件数据<br/>提前加载数据]
R3[ReadAsync<br/>异步读取<br/>异步读取数据]
end
subgraph LifecycleGroup["生命周期管理 Lifecycle Management"]
direction TB
L1[Open<br/>打开文件<br/>打开文件进行读取]
L2[Close<br/>关闭文件<br/>关闭文件释放资源]
L3[GetLength<br/>获取文件长度<br/>获取文件大小]
end
subgraph PathGroup["路径管理 Path Management"]
direction TB
P1[GetLogicalPath<br/>获取逻辑路径<br/>获取逻辑文件路径]
P2[GetPhysicalPath<br/>获取物理路径<br/>获取物理文件路径]
end
FeatureLayer --> UsageLayer[使用场景层<br/>Usage Scenarios Layer]
subgraph UsageGroup["使用场景 Usage Scenarios"]
direction TB
U1[索引查询<br/>Index Query<br/>读取索引文件]
U2[数据加载<br/>Data Loading<br/>加载Segment数据]
U3[版本读取<br/>Version Read<br/>读取版本文件]
end
UsageLayer --> End([文件读取完成<br/>File Read Complete])
FeatureLayer -.->|包含| ReadGroup
FeatureLayer -.->|包含| LifecycleGroup
FeatureLayer -.->|包含| PathGroup
UsageLayer -.->|包含| UsageGroup
ReadGroup -.->|支持| UsageGroup
LifecycleGroup -.->|支持| UsageGroup
PathGroup -.->|支持| UsageGroup
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style FeatureLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style UsageLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ReadGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style R1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style R2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style R3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style LifecycleGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style L1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style L2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style L3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style PathGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style P1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style P2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style UsageGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style U1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style U2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style U3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
FileWriter 是文件写入器，提供文件写入功能，支持同步和异步写入。让我们先通过序列图来理解 FileWriter 的完整工作流程：
sequenceDiagram
participant Client
participant FileWriter
participant Buffer
participant Storage
Client->>FileWriter: CreateFileWriter(path, option)
FileWriter->>Storage: 创建文件
Storage-->>FileWriter: 文件句柄
FileWriter->>Buffer: 初始化缓冲区
FileWriter-->>Client: FileWriter对象
Client->>FileWriter: Write(buffer, length)
FileWriter->>Buffer: 写入缓冲区
alt 缓冲区满
Buffer->>Storage: 刷新到磁盘
Storage-->>Buffer: 刷新完成
end
FileWriter-->>Client: 返回写入长度
Client->>FileWriter: ReserveFile(size)
FileWriter->>Storage: 预留空间
Storage-->>FileWriter: 预留完成
Client->>FileWriter: Close()
FileWriter->>Buffer: 刷新缓冲区
Buffer->>Storage: 刷新到磁盘
Storage-->>FileWriter: 刷新完成
FileWriter->>Storage: 关闭文件
FileWriter-->>Client: 关闭完成
FileWriter 的完整定义：
// file_system/file/FileWriter.h
class FileWriter : public autil::NoCopyable
{
public:
// 打开文件
virtual FSResult<void> Open(
const std::string& logicalPath, // 逻辑路径
const std::string& physicalPath) = 0; // 物理路径
// 关闭文件
virtual FSResult<void> Close() = 0;
// 写入文件：同步写入
virtual FSResult<size_t> Write(
const void* buffer, // 缓冲区
size_t length) = 0; // 写入长度
// 异步写入：返回 Future
virtual future_lite::Future<FSResult<size_t>> WriteAsync(
const void* buffer,
size_t length) = 0;
// 预留文件空间：用于地址访问模式
virtual FSResult<void> ReserveFile(size_t reserveSize) = 0;
// 截断文件：调整文件大小
virtual FSResult<void> Truncate(size_t truncateSize) = 0;
// 刷新缓冲区：将缓冲区数据刷新到磁盘
virtual FSResult<void> Flush() = 0;
// 同步文件：确保数据写入磁盘
virtual FSResult<void> Sync() = 0;
// 获取文件长度
virtual size_t GetLength() const = 0;
// 获取逻辑路径
virtual const std::string& GetLogicalPath() const = 0;
// 获取物理路径
virtual const std::string& GetPhysicalPath() const = 0;
};
FileWriter 的实现示例：
// file_system/file/LocalFileWriter.cpp
class LocalFileWriter : public FileWriter
{
private:
std::string _logicalPath;
std::string _physicalPath;
int _fd;
size_t _fileLength;
std::vector<char> _buffer;
size_t _bufferSize;
bool _atomicWrite;
public:
FSResult<void> Open(const std::string& logicalPath,
const std::string& physicalPath) override {
_logicalPath = logicalPath;
_physicalPath = physicalPath;
// 原子写入：先写入临时文件
if (_atomicWrite) {
_physicalPath = physicalPath + ".tmp";
}
_fd = ::open(_physicalPath.c_str(), O_WRONLY | O_CREAT | O_TRUNC, 0644);
if (_fd < 0) {
return FSResult<void>::Error("Failed to open file: " + _physicalPath);
}
_fileLength = 0;
_buffer.clear();
_buffer.reserve(_bufferSize);
return FSResult<void>::OK();
}
FSResult<size_t> Write(const void* buffer, size_t length) override {
// 1. 写入缓冲区
const char* data = static_cast<const char*>(buffer);
_buffer.insert(_buffer.end(), data, data + length);
// 2. 如果缓冲区满，刷新到磁盘
if (_buffer.size() >= _bufferSize) {
auto status = Flush();
if (!status.IsOK()) {
return FSResult<size_t>::Error(status.GetError());
}
}
_fileLength += length;
return FSResult<size_t>::OK(length);
}
FSResult<void> Flush() override {
if (_buffer.empty()) {
return FSResult<void>::OK();
}
ssize_t nwrite = ::write(_fd, _buffer.data(), _buffer.size());
if (nwrite < 0) {
return FSResult<void>::Error("Failed to write file");
}
_buffer.clear();
return FSResult<void>::OK();
}
FSResult<void> Sync() override {
// 先刷新缓冲区
auto status = Flush();
if (!status.IsOK()) {
return status;
}
// 同步到磁盘
if (fsync(_fd) < 0) {
return FSResult<void>::Error("Failed to sync file");
}
return FSResult<void>::OK();
}
FSResult<void> Close() override {
// 1. 刷新缓冲区
auto status = Flush();
if (!status.IsOK()) {
return status;
}
// 2. 同步到磁盘
status = Sync();
if (!status.IsOK()) {
return status;
}
// 3. 关闭文件
if (_fd >= 0) {
::close(_fd);
_fd = -1;
}
// 4. 原子写入：重命名临时文件
if (_atomicWrite) {
std::string finalPath = _physicalPath.substr(0, _physicalPath.length() - 4);
if (rename(_physicalPath.c_str(), finalPath.c_str()) < 0) {
return FSResult<void>::Error("Failed to rename file");
}
}
return FSResult<void>::OK();
}
FSResult<void> ReserveFile(size_t reserveSize) override {
// 使用 fallocate 预留空间
if (fallocate(_fd, 0, 0, reserveSize) < 0) {
return FSResult<void>::Error("Failed to reserve file space");
}
return FSResult<void>::OK();
}
};
FileWriter 的关键特性：
fallocate 预留空间FileWriter 接口：提供文件写入功能：
flowchart TB
Start([FileWriter 写入功能<br/>FileWriter Write Features]) --> WriteLayer[写入类型层<br/>Write Types Layer]
subgraph WriteGroup["写入类型 Write Types"]
direction TB
W1[文件写入<br/>File Write<br/>基础写入操作]
W2[随机写入<br/>Random Write<br/>支持随机位置写入]
W3[顺序写入<br/>Sequential Write<br/>顺序写入优化]
end
WriteLayer --> SupportLayer[支持组件层<br/>Support Components Layer]
subgraph SupportGroup["支持组件 Support Components"]
direction TB
S1[缓冲区管理<br/>Buffer Management<br/>管理写入缓冲区]
S2[同步操作<br/>Sync Operation<br/>同步数据到磁盘]
end
SupportLayer --> End([写入功能完成<br/>Write Features Complete])
WriteLayer -.->|包含| WriteGroup
SupportLayer -.->|包含| SupportGroup
W1 -.->|使用| S1
W2 -.->|使用| S2
W3 -.->|使用| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style WriteLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style WriteGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style W1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style W2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style W3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
Storage 是存储抽象，定义在 file_system/Storage.h 中：
// file_system/Storage.h
class Storage
{
public:
// 创建输入存储
static std::shared_ptr<Storage> CreateInputStorage(
const std::shared_ptr<FileSystemOptions>& options,
const std::shared_ptr<util::BlockMemoryQuotaController>& memController,
const std::shared_ptr<EntryTable>& entryTable) = 0;
// 创建输出存储
static std::shared_ptr<Storage> CreateOutputStorage(
const std::string& outputRoot,
const std::shared_ptr<FileSystemOptions>& options,
const std::shared_ptr<util::BlockMemoryQuotaController>& memController) = 0;
// 创建文件读取器
virtual FSResult<std::shared_ptr<FileReader>> CreateFileReader(
const std::string& logicalFilePath,
const std::string& physicalFilePath,
const ReaderOption& readerOption) = 0;
// 创建文件写入器
virtual FSResult<std::shared_ptr<FileWriter>> CreateFileWriter(
const std::string& logicalFilePath,
const std::string& physicalFilePath,
const WriterOption& writerOption) = 0;
// 同步存储
virtual FSResult<std::future<bool>> Sync() = 0;
// 获取存储类型
virtual FSStorageType GetStorageType() const = 0;
};
Storage 的关键方法：
Storage 抽象：提供底层存储操作：
flowchart TB
Start([Storage 抽象<br/>Storage Abstraction]) --> MethodLayer[方法层<br/>Methods Layer]
subgraph MethodGroup["核心方法 Core Methods"]
direction TB
M1[CreateInputStorage<br/>创建输入存储<br/>用于读取操作]
M2[CreateOutputStorage<br/>创建输出存储<br/>用于写入操作]
M3[CreateFileReader<br/>创建文件读取器<br/>创建读取器对象]
M4[CreateFileWriter<br/>创建文件写入器<br/>创建写入器对象]
M5[Sync<br/>同步存储<br/>刷新数据到磁盘]
M6[GetStorageType<br/>获取存储类型<br/>返回存储类型]
end
MethodLayer --> End([存储操作完成<br/>Storage Operations Complete])
MethodLayer -.->|包含| MethodGroup
M1 -.->|创建| M3
M2 -.->|创建| M4
M1 -.->|使用| M5
M2 -.->|使用| M5
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style MethodLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style MethodGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M6 fill:#90caf9,stroke:#1976d2,stroke-width:2px
IndexLib 支持多种存储类型：
存储类型：本地存储、分布式存储等：
flowchart TB
Start([存储类型<br/>Storage Types]) --> TypeLayer[存储类型层<br/>Storage Types Layer]
subgraph TypeGroup["存储类型 Storage Types"]
direction TB
T1[本地存储<br/>Local Storage<br/>基于本地文件系统<br/>Posix文件系统]
T2[分布式存储<br/>Distributed Storage<br/>HDFS Pangu<br/>分布式文件系统]
T3[内存存储<br/>Memory Storage<br/>基于内存<br/>临时存储]
T4[混合存储<br/>Hybrid Storage<br/>多种存储组合<br/>灵活配置]
end
TypeLayer --> BackendLayer[存储后端层<br/>Storage Backend Layer]
subgraph BackendGroup["存储后端 Storage Backend"]
direction TB
B1[存储后端<br/>Storage Backend<br/>统一后端接口<br/>抽象存储操作]
end
BackendLayer --> End([统一存储访问<br/>Unified Storage Access])
TypeLayer -.->|包含| TypeGroup
BackendLayer -.->|包含| BackendGroup
T1 -.->|使用| B1
T2 -.->|使用| B1
T3 -.->|使用| B1
T4 -.->|使用| B1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style TypeLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style BackendLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style TypeGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style T1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style BackendGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style B1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
存储类型：
IndexLib 支持多种存储格式，包括 Package、Archive 和压缩格式，用于优化存储效率和访问性能。让我们先通过类图来理解存储格式的整体架构：
classDiagram
class StorageFormat {
<<interface>>
+ Pack()
+ Unpack()
+ GetFileInfo()
}
class PackageFormat {
+ Pack()
+ Unpack()
+ GetFileInfo()
- WriteIndex()
- ReadIndex()
}
class ArchiveFormat {
+ Pack()
+ Unpack()
+ AppendFile()
- WriteIndex()
- ReadIndex()
}
class Compressor {
<<interface>>
+ Compress()
+ Decompress()
+ GetCompressionRatio()
}
class LZ4Compressor {
+ Compress()
+ Decompress()
}
class ZstdCompressor {
+ Compress()
+ Decompress()
}
StorageFormat <|-- PackageFormat : 实现
StorageFormat <|-- ArchiveFormat : 实现
PackageFormat --> Compressor : 使用
ArchiveFormat --> Compressor : 使用
Compressor <|-- LZ4Compressor : 实现
Compressor <|-- ZstdCompressor : 实现
Package 格式是一种打包存储格式，将多个小文件打包成一个大文件，减少文件系统的小文件数量，提高 IO 效率。Package 格式的打包流程如下：
flowchart TB
Start([开始打包<br/>Start Packing]) --> InitLayer[初始化层<br/>Initialization Layer]
subgraph InitGroup["初始化 Initialization"]
direction TB
I1[读取文件列表<br/>Read File List<br/>获取待打包文件]
I2[创建 Package 文件<br/>Create Package File<br/>创建输出文件]
I3[写入 Package 头<br/>Write Package Header<br/>写入文件头信息]
end
InitLayer --> ProcessLayer[处理层<br/>Processing Layer]
subgraph ProcessGroup["文件处理 File Processing"]
direction TB
P1{遍历文件<br/>Loop Files}
P2[读取文件内容<br/>Read File Content<br/>读取文件数据]
P3{需要压缩?<br/>Need Compression?}
P4[压缩数据<br/>Compress Data<br/>压缩文件数据]
P5[写入文件数据<br/>Write File Data<br/>写入到Package]
P6[更新索引<br/>Update Index<br/>更新文件索引]
end
ProcessLayer --> FinalizeLayer[完成层<br/>Finalization Layer]
subgraph FinalizeGroup["完成处理 Finalization"]
direction TB
F1[写入索引<br/>Write Index<br/>写入文件索引]
F2[关闭 Package<br/>Close Package<br/>关闭文件]
end
FinalizeLayer --> End([打包完成<br/>Packing Complete])
InitLayer -.->|包含| InitGroup
ProcessLayer -.->|包含| ProcessGroup
FinalizeLayer -.->|包含| FinalizeGroup
I1 --> I2
I2 --> I3
I3 --> P1
P1 -->|下一个文件| P2
P1 -->|完成| F1
P2 --> P3
P3 -->|是| P4
P3 -->|否| P5
P4 --> P5
P5 --> P6
P6 --> P1
F1 --> F2
F2 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style InitLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ProcessLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FinalizeLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style InitGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style I1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ProcessGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style P1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P6 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FinalizeGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style F1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
Package 格式的结构：
// file_system/package/PackageFormat.h
struct PackageHeader {
uint32_t magic; // 魔数：0x504B4741 ("PKGA")
uint32_t version; // 版本号
uint32_t fileCount; // 文件数量
uint64_t indexOffset; // 索引偏移量
uint64_t indexSize; // 索引大小
uint32_t flags; // 标志位（压缩、加密等）
};
struct FileEntry {
std::string fileName; // 文件名
uint64_t offset; // 文件在 Package 中的偏移量
uint64_t size; // 文件大小（压缩前）
uint64_t compressedSize; // 压缩后大小
uint32_t compressionType; // 压缩类型
uint32_t crc32; // CRC32 校验
};
struct PackageIndex {
std::vector<FileEntry> entries; // 文件条目列表
std::map<std::string, size_t> nameToIndex; // 文件名到索引的映射
};
Package 格式的打包实现：
// file_system/package/PackageFormat.cpp
FSResult<void> PackageFormat::Pack(const std::vector<std::string>& files,
const std::string& outputPath) {
// 1. 创建 Package 文件
auto writer = CreateFileWriter(outputPath);
if (!writer.IsOK()) {
return FSResult<void>::Error("Failed to create package file");
}
// 2. 写入 Package 头（占位，稍后更新）
PackageHeader header = {};
header.magic = 0x504B4741; // "PKGA"
header.version = 1;
header.fileCount = files.size();
size_t headerSize = sizeof(PackageHeader);
size_t dataOffset = headerSize;
// 3. 写入文件数据
PackageIndex index;
for (const auto& filePath : files) {
// 读取文件
auto reader = CreateFileReader(filePath);
if (!reader.IsOK()) {
return FSResult<void>::Error("Failed to read file: " + filePath);
}
std::vector<char> data(reader->GetLength());
auto readResult = reader->Read(data.data(), data.size(), 0);
if (!readResult.IsOK()) {
return FSResult<void>::Error("Failed to read file data");
}
// 压缩文件（可选）
std::vector<char> compressed;
uint32_t compressionType = 0;
if (_options.compress) {
auto compressResult = _compressor->Compress(data, compressed);
if (compressResult.IsOK() && compressed.size() < data.size()) {
data = compressed;
compressionType = _compressor->GetType();
}
}
// 写入文件数据
FileEntry entry;
entry.fileName = GetFileName(filePath);
entry.offset = dataOffset;
entry.size = reader->GetLength();
entry.compressedSize = data.size();
entry.compressionType = compressionType;
entry.crc32 = CalculateCRC32(data);
auto writeResult = writer->Write(data.data(), data.size());
if (!writeResult.IsOK()) {
return FSResult<void>::Error("Failed to write file data");
}
dataOffset += data.size();
index.entries.push_back(entry);
index.nameToIndex[entry.fileName] = index.entries.size() - 1;
}
// 4. 写入索引
header.indexOffset = dataOffset;
std::string indexData = SerializeIndex(index);
header.indexSize = indexData.size();
auto writeResult = writer->Write(indexData.data(), indexData.size());
if (!writeResult.IsOK()) {
return FSResult<void>::Error("Failed to write index");
}
// 5. 更新 Package 头
writer->Seek(0);
writer->Write(&header, sizeof(header));
// 6. 关闭文件
writer->Close();
return FSResult<void>::OK();
}
Package 格式的特点：
Package 格式：将多个文件打包成一个文件：
flowchart TB
Start([Package 格式<br/>Package Format]) --> FeatureLayer[功能层<br/>Features Layer]
subgraph FeatureGroup["核心功能 Core Features"]
direction TB
F1[文件打包<br/>File Packaging<br/>将多个文件打包]
F2[压缩存储<br/>Compressed Storage<br/>支持文件压缩]
F3[索引管理<br/>Index Management<br/>管理文件索引]
end
FeatureLayer --> OperationLayer[操作层<br/>Operations Layer]
subgraph OperationGroup["文件操作 File Operations"]
direction TB
O1[文件读取<br/>File Read<br/>读取打包文件]
O2[文件写入<br/>File Write<br/>写入打包文件]
end
OperationLayer --> End([Package 操作完成<br/>Package Operations Complete])
FeatureLayer -.->|包含| FeatureGroup
OperationLayer -.->|包含| OperationGroup
F1 -.->|使用| O1
F2 -.->|使用| O2
F3 -.->|使用| O1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style FeatureLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style OperationLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FeatureGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style OperationGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style O1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
Archive 格式是一种归档存储格式，支持文件归档、压缩、索引和追加。Archive 格式的归档流程（包括创建归档和追加文件）如下：
flowchart TB
Start([Archive 操作<br/>Archive Operations]) --> OperationType{操作类型<br/>Operation Type}
OperationType -->|创建归档| CreateFlow[创建归档流程<br/>Create Archive Flow]
OperationType -->|追加文件| AppendFlow[追加文件流程<br/>Append File Flow]
subgraph CreateGroup["创建归档流程 Create Archive Flow"]
direction TB
C1[创建 Archive 文件<br/>Create Archive File<br/>创建归档文件]
C2[写入 Archive 头<br/>Write Archive Header<br/>写入文件头信息]
C3{遍历文件<br/>Loop Files}
C4[读取文件<br/>Read File<br/>读取文件内容]
C5[压缩文件<br/>Compress File<br/>压缩文件数据]
C6[写入文件数据<br/>Write File Data<br/>写入到Archive]
C7[更新索引<br/>Update Index<br/>更新文件索引]
C8[写入索引<br/>Write Index<br/>写入文件索引]
C9[关闭 Archive<br/>Close Archive<br/>关闭文件]
end
subgraph AppendGroup["追加文件流程 Append File Flow"]
direction TB
A1[打开 Archive<br/>Open Archive<br/>打开归档文件]
A2[读取索引<br/>Read Index<br/>读取文件索引]
A3[追加文件数据<br/>Append File Data<br/>追加新文件]
A4[更新索引<br/>Update Index<br/>更新文件索引]
A5[写入索引<br/>Write Index<br/>写入更新后的索引]
A6[关闭 Archive<br/>Close Archive<br/>关闭文件]
end
CreateFlow --> End1([创建完成<br/>Create Complete])
AppendFlow --> End2([追加完成<br/>Append Complete])
CreateFlow -.->|包含| CreateGroup
AppendFlow -.->|包含| AppendGroup
C1 --> C2
C2 --> C3
C3 -->|下一个文件| C4
C3 -->|完成| C8
C4 --> C5
C5 --> C6
C6 --> C7
C7 --> C3
C8 --> C9
C9 --> End1
A1 --> A2
A2 --> A3
A3 --> A4
A4 --> A5
A5 --> A6
A6 --> End2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End1 fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End2 fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style OperationType fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CreateFlow fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style AppendFlow fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style CreateGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C6 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C7 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C8 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C9 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style AppendGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style A1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A5 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A6 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
Archive 格式的结构：
// file_system/archive/ArchiveFormat.h
struct ArchiveHeader {
uint32_t magic; // 魔数：0x41524348 ("ARCH")
uint32_t version; // 版本号
uint64_t fileCount; // 文件数量
uint64_t indexOffset; // 索引偏移量
uint64_t indexSize; // 索引大小
uint32_t flags; // 标志位
};
struct ArchiveEntry {
std::string fileName; // 文件名
uint64_t offset; // 文件在 Archive 中的偏移量
uint64_t size; // 文件大小
uint64_t compressedSize; // 压缩后大小
uint32_t compressionType; // 压缩类型
uint64_t timestamp; // 时间戳
uint32_t crc32; // CRC32 校验
};
struct ArchiveIndex {
std::vector<ArchiveEntry> entries; // 文件条目列表
std::map<std::string, size_t> nameToIndex; // 文件名到索引的映射
std::map<uint64_t, std::vector<size_t>> timestampToIndex; // 时间戳到索引的映射
};
Archive 格式的特点：
Archive 格式：归档存储格式的特点和应用：
flowchart TB
Start([Archive 格式<br/>Archive Format]) --> FeatureLayer[功能层<br/>Features Layer]
subgraph FeatureGroup["核心功能 Core Features"]
direction TB
F1[文件归档<br/>File Archive<br/>归档文件存储]
F2[压缩存储<br/>Compressed Storage<br/>支持文件压缩]
F3[索引追加<br/>Index Append<br/>支持索引追加]
end
FeatureLayer --> OperationLayer[操作层<br/>Operations Layer]
subgraph OperationGroup["文件操作 File Operations"]
direction TB
O1[归档读取<br/>Archive Read<br/>读取归档文件]
O2[归档写入<br/>Archive Write<br/>写入归档文件]
end
OperationLayer --> End([Archive 操作完成<br/>Archive Operations Complete])
FeatureLayer -.->|包含| FeatureGroup
OperationLayer -.->|包含| OperationGroup
F1 -.->|使用| O1
F2 -.->|使用| O2
F3 -.->|使用| O1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style FeatureLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style OperationLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FeatureGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style OperationGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style O1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
IndexLib 支持多种压缩格式，包括 LZ4、Zstd、Snappy 和 Gzip。压缩格式的架构如下：
classDiagram
class Compressor {
<<interface>>
+ Compress()
+ Decompress()
+ GetType()
+ GetCompressionRatio()
+ GetCompressionSpeed()
+ GetDecompressionSpeed()
}
class LZ4Compressor {
+ Compress()
+ Decompress()
- lz4_compress_level int
}
class ZstdCompressor {
+ Compress()
+ Decompress()
- zstd_compression_level int
}
class SnappyCompressor {
+ Compress()
+ Decompress()
}
class GzipCompressor {
+ Compress()
+ Decompress()
- gzip_compression_level int
}
Compressor <|-- LZ4Compressor : 实现
Compressor <|-- ZstdCompressor : 实现
Compressor <|-- SnappyCompressor : 实现
Compressor <|-- GzipCompressor : 实现
压缩格式的对比：
| 压缩算法 | 压缩速度 | 压缩率 | 解压速度 | 适用场景 |
|---|---|---|---|---|
| LZ4 | 极快 | 中等 | 极快 | 实时写入、高频访问 |
| Zstd | 快 | 高 | 快 | 存储优化、离线处理 |
| Snappy | 极快 | 低 | 极快 | 实时写入、低延迟 |
| Gzip | 慢 | 高 | 中等 | 兼容性要求高 |
压缩格式的选择：
压缩格式：支持多种压缩算法：
flowchart TB
Start([压缩格式<br/>Compression Formats]) --> AlgorithmLayer[压缩算法层<br/>Compression Algorithms Layer]
subgraph AlgorithmGroup["压缩算法 Compression Algorithms"]
direction TB
A1[LZ4压缩<br/>LZ4 Compression<br/>快速压缩算法]
A2[Zstd压缩<br/>Zstd Compression<br/>高效压缩算法]
A3[Snappy压缩<br/>Snappy Compression<br/>快速压缩算法]
A4[Gzip压缩<br/>Gzip Compression<br/>通用压缩算法]
end
AlgorithmLayer --> ManagementLayer[管理层<br/>Management Layer]
subgraph ManagementGroup["压缩管理 Compression Management"]
direction TB
M1[压缩管理<br/>Compression Management<br/>统一管理接口]
end
ManagementLayer --> End([压缩功能完成<br/>Compression Features Complete])
AlgorithmLayer -.->|包含| AlgorithmGroup
ManagementLayer -.->|包含| ManagementGroup
A1 -.->|使用| M1
A2 -.->|使用| M1
A3 -.->|使用| M1
A4 -.->|使用| M1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style AlgorithmLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ManagementLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style AlgorithmGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style A1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ManagementGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
文件系统缓存通过缓存文件内容、元数据和预取数据来提高文件访问性能。缓存机制的整体架构如下：
flowchart TB
Start([文件系统缓存<br/>File System Cache]) --> CacheLayer[缓存类型层<br/>Cache Types Layer]
subgraph CacheGroup["缓存类型 Cache Types"]
direction TB
C1[文件缓存<br/>File Cache<br/>缓存文件内容]
C2[元数据缓存<br/>Metadata Cache<br/>缓存文件元数据]
C3[预取缓存<br/>Prefetch Cache<br/>预取文件数据]
end
CacheLayer --> StrategyLayer[策略层<br/>Strategy Layer]
subgraph StrategyGroup["缓存策略 Cache Strategies"]
direction TB
S1[LRU缓存<br/>LRU Cache<br/>最近最少使用]
S2[缓存管理<br/>Cache Management<br/>统一管理接口]
end
StrategyLayer --> End([缓存功能完成<br/>Cache Features Complete])
CacheLayer -.->|包含| CacheGroup
StrategyLayer -.->|包含| StrategyGroup
C1 -.->|使用| S1
C2 -.->|使用| S2
C3 -.->|使用| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style CacheLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style StrategyLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style CacheGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style StrategyGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
缓存机制：
文件系统支持多种缓存策略，包括 LRU（最近最少使用）、LFU（最不经常使用）和按需缓存等。各种缓存策略及其支持功能如下：
flowchart TB
Start([缓存策略<br/>Cache Strategies]) --> StrategyLayer[策略层<br/>Strategies Layer]
subgraph StrategyGroup["缓存策略 Cache Strategies"]
direction TB
S1[LRU策略<br/>LRU Strategy<br/>最近最少使用]
S2[LFU策略<br/>LFU Strategy<br/>最少使用频率]
S3[按需缓存<br/>On-Demand Cache<br/>按需加载缓存]
end
StrategyLayer --> SupportLayer[支持功能层<br/>Support Features Layer]
subgraph SupportGroup["支持功能 Support Features"]
direction TB
SF1[预取缓存<br/>Prefetch Cache<br/>提前加载数据]
SF2[缓存淘汰<br/>Cache Eviction<br/>淘汰过期缓存]
end
SupportLayer --> End([缓存策略完成<br/>Cache Strategies Complete])
StrategyLayer -.->|包含| StrategyGroup
SupportLayer -.->|包含| SupportGroup
S1 -.->|使用| SF1
S2 -.->|使用| SF2
S3 -.->|使用| SF1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style SF1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style SF2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
缓存策略：
文件系统通过多种 IO 优化策略来提高性能，包括批量 IO、异步 IO、预取等。IO 优化的整体架构如下：
flowchart TB
Start([IO 优化<br/>IO Optimization]) --> OptimizationLayer[优化策略层<br/>Optimization Strategies Layer]
subgraph OptimizationGroup["优化策略 Optimization Strategies"]
direction TB
O1[批量IO<br/>Batch IO<br/>批量读取和写入]
O2[异步IO<br/>Async IO<br/>异步读取和写入]
O3[预取<br/>Prefetch<br/>预取文件数据]
end
OptimizationLayer --> SupportLayer[支持功能层<br/>Support Features Layer]
subgraph SupportGroup["支持功能 Support Features"]
direction TB
S1[IO合并<br/>IO Merge<br/>合并多个IO操作]
S2[IO优化<br/>IO Optimization<br/>统一优化接口]
end
SupportLayer --> End([IO 优化完成<br/>IO Optimization Complete])
OptimizationLayer -.->|包含| OptimizationGroup
SupportLayer -.->|包含| SupportGroup
O1 -.->|使用| S1
O2 -.->|使用| S2
O3 -.->|使用| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style OptimizationLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style OptimizationGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style O1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style O2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style O3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
IO 优化策略：
文件系统通过压缩、打包、存储分层等策略来优化存储效率。存储优化的整体架构如下：
flowchart TB
Start([存储优化<br/>Storage Optimization]) --> StrategyLayer[优化策略层<br/>Optimization Strategies Layer]
subgraph StrategyGroup["优化策略 Optimization Strategies"]
direction TB
S1[文件压缩<br/>File Compression<br/>压缩文件数据]
S2[文件打包<br/>File Packaging<br/>打包多个文件]
S3[存储分层<br/>Storage Tiering<br/>分层存储管理]
end
StrategyLayer --> ManagementLayer[管理层<br/>Management Layer]
subgraph ManagementGroup["管理功能 Management Features"]
direction TB
M1[生命周期管理<br/>Lifecycle Management<br/>管理文件生命周期]
M2[存储优化<br/>Storage Optimization<br/>统一优化接口]
end
ManagementLayer --> End([存储优化完成<br/>Storage Optimization Complete])
StrategyLayer -.->|包含| StrategyGroup
ManagementLayer -.->|包含| ManagementGroup
S1 -.->|使用| M1
S2 -.->|使用| M2
S3 -.->|使用| M1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ManagementLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ManagementGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
存储优化策略：
文件系统通过统一接口设计来屏蔽底层存储差异，支持多种存储后端。统一接口设计的核心要点如下：
flowchart TB
Start([统一接口设计<br/>Unified Interface Design]) --> DesignLayer[设计层<br/>Design Layer]
subgraph DesignGroup["核心设计 Core Design"]
direction TB
D1[接口抽象<br/>Interface Abstraction<br/>统一接口定义]
D2[多后端支持<br/>Multi-Backend Support<br/>支持多种存储后端]
D3[透明访问<br/>Transparent Access<br/>透明访问机制]
end
DesignLayer --> SupportLayer[支持功能层<br/>Support Features Layer]
subgraph SupportGroup["支持功能 Support Features"]
direction TB
S1[灵活扩展<br/>Flexible Extension<br/>支持自定义扩展]
S2[存储适配<br/>Storage Adapter<br/>存储后端适配]
end
SupportLayer --> End([统一接口完成<br/>Unified Interface Complete])
DesignLayer -.->|包含| DesignGroup
SupportLayer -.->|包含| SupportGroup
D1 -.->|支持| S1
D2 -.->|使用| S2
D3 -.->|支持| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style DesignLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style DesignGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style D1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
逻辑路径的设计：
逻辑路径设计：通过逻辑路径管理文件和版本：
flowchart TB
Start([逻辑路径设计<br/>Logical Path Design]) --> ManagementLayer[管理层<br/>Management Layer]
subgraph ManagementGroup["管理功能 Management Features"]
direction TB
M1[路径映射<br/>Path Mapping<br/>物理路径到逻辑路径]
M2[版本管理<br/>Version Management<br/>版本路径管理]
M3[Segment管理<br/>Segment Management<br/>Segment路径管理]
end
ManagementLayer --> SupportLayer[支持功能层<br/>Support Features Layer]
subgraph SupportGroup["支持功能 Support Features"]
direction TB
S1[路径隔离<br/>Path Isolation<br/>路径相互隔离]
S2[路径解析<br/>Path Resolution<br/>解析逻辑路径]
end
SupportLayer --> End([逻辑路径设计完成<br/>Logical Path Design Complete])
ManagementLayer -.->|包含| ManagementGroup
SupportLayer -.->|包含| SupportGroup
M1 -.->|使用| S1
M2 -.->|使用| S2
M3 -.->|使用| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style ManagementLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ManagementGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
文件系统通过缓存、预取、批量 IO 等优化策略来提高性能。性能优化设计的核心机制如下：
flowchart TB
Start([性能优化设计<br/>Performance Optimization Design]) --> MechanismLayer[机制层<br/>Mechanisms Layer]
subgraph MechanismGroup["优化机制 Optimization Mechanisms"]
direction TB
M1[缓存机制<br/>Cache Mechanism<br/>文件内容缓存]
M2[预取机制<br/>Prefetch Mechanism<br/>预取文件数据]
M3[批量操作<br/>Batch Operation<br/>批量IO操作]
end
MechanismLayer --> SupportLayer[支持功能层<br/>Support Features Layer]
subgraph SupportGroup["支持功能 Support Features"]
direction TB
S1[异步操作<br/>Async Operation<br/>异步IO操作]
S2[性能调优<br/>Performance Tuning<br/>统一调优接口]
end
SupportLayer --> End([性能优化完成<br/>Performance Optimization Complete])
MechanismLayer -.->|包含| MechanismGroup
SupportLayer -.->|包含| SupportGroup
M1 -.->|使用| S1
M2 -.->|使用| S2
M3 -.->|使用| S1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style MechanismLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style MechanismGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
文件系统抽象与存储格式是 IndexLib 的核心功能，通过 IFileSystem、IDirectory、FileReader、FileWriter 等组件实现统一的文件系统访问接口。通过本文的深入解析，我们了解到：
关键组件：
关键特性：
关键设计：
理解文件系统抽象与存储格式，是掌握 IndexLib 存储管理机制的关键。文件系统抽象不仅提供了统一的文件访问接口，还通过逻辑路径、存储格式、缓存机制等特性，实现了高效、灵活、可扩展的存储管理方案。
通过本系列文章的深入解析，我们已经全面了解了 IndexLib 的架构、核心组件、构建流程、查询流程、版本管理、Segment 合并、内存管理、索引类型、Locator 与数据一致性、文件系统抽象等各个方面。希望这些文章能够帮助读者深入理解 IndexLib 的设计和实现，为实际应用和二次开发提供参考。
]]>Locator 与数据一致性概览：从 Locator 结构到数据一致性保证的完整机制：
flowchart TD
Start[Locator体系] --> CoreLayer[核心组件层]
subgraph LocatorGroup["Locator核心组件"]
direction TB
L1[Locator<br/>位置定位器]
L2[Progress<br/>进度信息]
L3[MultiProgress<br/>多进度信息]
L4[DocInfo<br/>文档信息]
L1 --> L2
L1 --> L3
L1 --> L4
L3 --> L2
end
subgraph CompareGroup["Locator比较组件"]
direction TB
C1[LocatorCompareResult<br/>比较结果枚举]
C2[IsFasterThan<br/>比较方法]
C3[LCR_SLOWER<br/>更慢]
C4[LCR_FULLY_FASTER<br/>完全更快]
C1 --> C2
C2 --> C3
C2 --> C4
end
CoreLayer --> LocatorGroup
CoreLayer --> CompareGroup
LocatorGroup --> Function[Locator功能]
CompareGroup --> Function
Function --> F1[位置定位<br/>精确定位数据处理位置]
Function --> F2[增量更新<br/>支持增量更新机制]
Function --> F3[一致性保证<br/>保证数据一致性]
Function --> F4[进度追踪<br/>追踪数据处理进度]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CoreLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style LocatorGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style L1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style L3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style L4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style CompareGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style Function fill:#f5f5f5,stroke:#757575,stroke-width:2px
style F1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F4 fill:#e0e0e0,stroke:#757575,stroke-width:1px
Locator 是增量更新的核心，定义在 framework/Locator.h 中。Locator 的设计目标是精确定位数据处理位置，支持增量更新和数据一致性保证。让我们先通过类图来理解 Locator 的整体架构：
classDiagram
class Locator {
- uint64_t _src
- Progress::Offset _minOffset
- MultiProgress _multiProgress
- string _userData
- bool _isLegacyLocator
+ IsFasterThan()
+ Update()
+ Serialize()
+ Deserialize()
+ GetSrc()
+ GetMinOffset()
+ GetMultiProgress()
+ GetUserData()
}
class LocatorCompareResult {
<<enumeration>>
LCR_INVALID
LCR_SLOWER
LCR_PARTIAL_FASTER
LCR_FULLY_FASTER
}
class DocInfo {
+ int64_t timestamp
+ uint32_t concurrentIdx
+ uint16_t hashId
+ uint8_t sourceIdx
}
class Progress {
+ uint32_t from
+ uint32_t to
+ Offset offset
}
class MultiProgress {
+ vector_ProgressVector _progresses
}
Locator --> LocatorCompareResult : 返回
Locator --> DocInfo : 包含
Locator --> MultiProgress : 包含
MultiProgress --> Progress : 包含
Locator 的完整定义：
// framework/Locator.h
class Locator final
{
public:
// Locator 比较结果
enum class LocatorCompareResult {
LCR_INVALID, // 无效：数据源不同，无法比较
LCR_SLOWER, // 比这个 locator 慢：数据未处理
LCR_PARTIAL_FASTER, // 部分 hash id 更快：需要部分处理
LCR_FULLY_FASTER // 完全比这个 locator 快（包括相等）：数据已处理
};
// 文档信息：记录文档在数据源中的位置
struct DocInfo {
int64_t timestamp; // 时间戳：记录数据的时间位置
uint32_t concurrentIdx; // 并发索引：处理时间戳相同的情况
uint16_t hashId; // Hash ID：用于分片处理
uint8_t sourceIdx; // 数据源索引：支持多数据源场景
// 比较两个 DocInfo
bool operator<(const DocInfo& other) const {
if (timestamp != other.timestamp) {
return timestamp < other.timestamp;
}
if (concurrentIdx != other.concurrentIdx) {
return concurrentIdx < other.concurrentIdx;
}
if (hashId != other.hashId) {
return hashId < other.hashId;
}
return sourceIdx < other.sourceIdx;
}
};
// 构造函数
Locator();
explicit Locator(uint64_t src);
Locator(uint64_t src, const MultiProgress& multiProgress);
Locator(const Locator& other);
Locator& operator=(const Locator& other);
// 比较方法：判断数据是否已处理
LocatorCompareResult IsFasterThan(const Locator& other,
bool ignoreLegacyDiffSrc = false) const;
// 更新方法：更新 Locator，只向前推进
void Update(const Locator& other);
// 序列化方法
std::string Serialize() const;
Status Deserialize(const std::string& str);
// 访问方法
uint64_t GetSrc() const { return _src; }
const Progress::Offset& GetMinOffset() const { return _minOffset; }
const MultiProgress& GetMultiProgress() const { return _multiProgress; }
const std::string& GetUserData() const { return _userData; }
bool IsLegacyLocator() const { return _isLegacyLocator; }
// 设置方法
void SetSrc(uint64_t src) { _src = src; }
void SetUserData(const std::string& userData) { _userData = userData; }
void SetMultiProgress(const MultiProgress& multiProgress);
// 工具方法
bool IsValid() const;
bool IsSameSrc(const Locator& other, bool ignoreLegacyDiffSrc = false) const;
std::string ToString() const;
private:
uint64_t _src; // 数据源标识
base::Progress::Offset _minOffset; // 最小偏移量
base::MultiProgress _multiProgress; // 多进度信息（每个 hashId 的进度）
std::string _userData; // 用户数据
bool _isLegacyLocator; // 是否遗留 Locator
// 内部方法
LocatorCompareResult CompareProgress(const ProgressVector& pv1,
const ProgressVector& pv2) const;
void UpdateMinOffset();
};
Locator 的关键字段：
Locator 的完整结构：包含所有关键字段和 DocInfo 结构：
flowchart TB
Locator["Locator 类<br/>class Locator<br/>增量更新的核心定位器"]
subgraph CoreFields["核心字段 Core Fields"]
direction LR
A["数据源标识<br/>_src: uint64_t<br/>区分不同数据源<br/>不同数据源无法比较"]
B["最小偏移量<br/>_minOffset: Progress::Offset<br/>std::pair<int64_t, uint32_t><br/>所有hashId的最小进度<br/>用于快速判断整体进度"]
C["多进度信息<br/>_multiProgress: MultiProgress<br/>std::vector<ProgressVector><br/>每个hashId的进度列表<br/>支持分片和并行处理"]
end
subgraph AuxFields["辅助字段 Auxiliary Fields"]
direction LR
D["用户数据<br/>_userData: std::string<br/>自定义业务信息<br/>支持业务扩展"]
E["遗留标识<br/>_isLegacyLocator: bool<br/>标识旧版本Locator<br/>保证向后兼容"]
end
subgraph InnerStruct["内部结构 Inner Structures"]
direction LR
F["DocInfo 结构<br/>struct DocInfo<br/>文档位置信息"]
G["比较结果枚举<br/>LocatorCompareResult<br/>LCR_INVALID/SLOWER/<br/>PARTIAL_FASTER/FULLY_FASTER"]
end
subgraph DocInfoFields["DocInfo 字段"]
direction LR
F1["timestamp: int64_t<br/>时间戳位置"]
F2["concurrentIdx: uint32_t<br/>并发索引"]
F3["hashId: uint16_t<br/>分片标识"]
F4["sourceIdx: uint8_t<br/>数据源索引"]
end
Locator --> CoreFields
Locator --> AuxFields
Locator --> InnerStruct
F --> DocInfoFields
B -.->|基于| C
C -.->|包含| F
style Locator fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style CoreFields fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style AuxFields fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style InnerStruct fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style DocInfoFields fill:#e1f5fe,stroke:#0277bd,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style G fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F1 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style F2 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style F3 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style F4 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
_src，不同数据源的 Locator 无法比较Progress 是进度信息，定义在 base/Progress.h 中：
// base/Progress.h
struct Progress {
using Offset = std::pair<int64_t, uint32_t>; // (timestamp, concurrentIdx)
static constexpr Offset INVALID_OFFSET = {-1, 0};
static constexpr Offset MIN_OFFSET = {0, 0};
Progress(uint32_t from, uint32_t to, const Offset& offset);
uint32_t from; // HashId 范围起始
uint32_t to; // HashId 范围结束
Offset offset; // 偏移量（timestamp, concurrentIdx）
};
typedef std::vector<Progress> ProgressVector; // 一个 hashId 范围的进度列表
typedef std::vector<ProgressVector> MultiProgress; // 多个 hashId 范围的进度列表
Progress 的关键字段：
Progress 的结构：包含 from、to、offset 等字段：
flowchart TB
Progress["Progress 结构<br/>记录单个hashId范围的进度信息"]
subgraph Fields["Progress 核心字段"]
direction LR
A["HashId范围<br/>from: uint32_t<br/>to: uint32_t<br/>定义分片范围"]
B["偏移量<br/>offset: Offset<br/>std::pair<int64_t, uint32_t><br/>timestamp + concurrentIdx"]
end
subgraph OffsetType["Offset 类型定义"]
direction LR
O1["INVALID_OFFSET<br/>{-1, 0}<br/>无效偏移量"]
O2["MIN_OFFSET<br/>{0, 0}<br/>最小偏移量"]
end
subgraph Collection["进度集合类型"]
direction LR
C["ProgressVector<br/>std::vector<Progress><br/>一个hashId范围的进度列表<br/>支持多个Progress对象"]
D["MultiProgress<br/>std::vector<ProgressVector><br/>多个hashId范围的进度列表<br/>支持并行处理和分片"]
end
Progress --> Fields
B -->|类型为| OffsetType
Fields -->|组成| C
C -->|组成| D
style Progress fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style Fields fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style OffsetType fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style Collection fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style O1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style O2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style C fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
DocInfo 是文档信息，记录文档在数据源中的位置：
DocInfo 的结构：包含 timestamp、concurrentIdx、hashId、sourceIdx 等字段：
flowchart TB
DocInfo["DocInfo 结构<br/>记录文档在数据源中的位置信息"]
subgraph Position["位置标识字段"]
direction LR
A["时间戳<br/>timestamp: int64_t<br/>记录数据的时间位置<br/>用于排序和定位"]
B["并发索引<br/>concurrentIdx: uint32_t<br/>处理时间戳相同的情况<br/>区分同一时刻的多个文档"]
end
subgraph Routing["路由相关字段"]
direction LR
C["Hash ID<br/>hashId: uint32_t<br/>用于分片处理<br/>决定文档所属分片"]
D["数据源索引<br/>sourceIdx: uint32_t<br/>支持多数据源<br/>标识数据来源"]
end
DocInfo --> Position
DocInfo --> Routing
A -.->|组成偏移量| B
style DocInfo fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style Position fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Routing fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
DocInfo 的关键字段：
IsFasterThan() 是 Locator 比较的核心方法，用于判断数据是否已处理。这是增量更新的基础，通过比较两个 Locator 来判断数据的新旧关系。让我们先通过流程图来理解比较的完整流程：
flowchart TD
Start([开始比较<br/>IsFasterThan]) --> CheckSrc{检查数据源<br/>IsSameSrc?}
CheckSrc -->|数据源不同| ReturnInvalid[返回 LCR_INVALID<br/>无法比较]
CheckSrc -->|数据源相同| CheckSize{比较 MultiProgress<br/>大小关系}
CheckSize -->|this.size > other.size<br/>覆盖更多hashId| ReturnPartial[返回 LCR_PARTIAL_FASTER<br/>部分更快]
CheckSize -->|this.size <= other.size| CheckEach[遍历每个 hashId<br/>逐一比较进度]
CheckEach --> CompareProgress[比较该 hashId 的进度<br/>CompareProgress]
CompareProgress --> CheckResult{比较结果判断}
CheckResult -->|LCR_FULLY_FASTER<br/>完全更快| CheckNext{还有更多<br/>hashId?}
CheckResult -->|LCR_SLOWER<br/>更慢| ReturnSlower[返回 LCR_SLOWER<br/>整体更慢]
CheckResult -->|LCR_PARTIAL_FASTER<br/>部分更快| ReturnPartial
CheckNext -->|是| CheckEach
CheckNext -->|否| ReturnFully[返回 LCR_FULLY_FASTER<br/>所有hashId都更快]
ReturnInvalid --> End([结束])
ReturnSlower --> End
ReturnPartial --> End
ReturnFully --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckSrc fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckNext fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ReturnInvalid fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ReturnSlower fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ReturnPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style ReturnFully fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckEach fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CompareProgress fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
IsFasterThan() 的完整实现：
// framework/Locator.cpp
LocatorCompareResult Locator::IsFasterThan(const Locator& other,
bool ignoreLegacyDiffSrc) const
{
// 1. 检查数据源是否相同
if (!IsSameSrc(other, ignoreLegacyDiffSrc)) {
return LCR_INVALID; // 数据源不同，无法比较
}
// 2. 快速路径：如果 MultiProgress 为空，特殊处理
if (_multiProgress.empty()) {
if (other._multiProgress.empty()) {
return LCR_FULLY_FASTER; // 都为空，认为相等
}
return LCR_SLOWER; // 当前为空，其他不为空，当前更慢
}
if (other._multiProgress.empty()) {
return LCR_FULLY_FASTER; // 当前不为空，其他为空，当前更快
}
// 3. 比较每个 hashId 的进度
bool hasPartialFaster = false;
bool hasSlower = false;
size_t minSize = std::min(_multiProgress.size(), other._multiProgress.size());
for (size_t i = 0; i < minSize; ++i) {
// 比较该 hashId 的进度
auto result = CompareProgress(_multiProgress[i], other._multiProgress[i]);
if (result == LCR_SLOWER) {
hasSlower = true;
// 如果有一个 hashId 更慢，且没有部分更快，直接返回更慢
if (!hasPartialFaster) {
return LCR_SLOWER;
}
} else if (result == LCR_PARTIAL_FASTER) {
hasPartialFaster = true;
// 如果有一个 hashId 部分更快，且没有更慢，继续检查
} else if (result == LCR_FULLY_FASTER) {
// 该 hashId 完全更快，继续检查下一个
continue;
} else {
// LCR_INVALID，不应该发生
return LCR_INVALID;
}
}
// 4. 处理大小不同的情况
if (_multiProgress.size() > other._multiProgress.size()) {
// 当前有更多的 hashId，部分更快
return LCR_PARTIAL_FASTER;
}
if (_multiProgress.size() < other._multiProgress.size()) {
// 当前有更少的 hashId，检查是否有更慢的
if (hasSlower) {
return LCR_SLOWER;
}
// 如果所有 hashId 都完全更快，但数量更少，返回部分更快
return LCR_PARTIAL_FASTER;
}
// 5. 大小相同，汇总结果
if (hasPartialFaster && !hasSlower) {
return LCR_PARTIAL_FASTER;
}
if (hasSlower) {
return LCR_SLOWER;
}
// 所有 hashId 都完全更快
return LCR_FULLY_FASTER;
}
比较算法的性能优化：
LCR_INVALID，避免遍历 ProgressLCR_SLOWERIsFasterThan() 方法：比较两个 Locator 的实现逻辑：
flowchart TB
subgraph Steps["比较步骤 Comparison Steps"]
direction LR
A["数据源检查<br/>IsSameSrc()<br/>检查 _src 是否相同<br/>不同则返回 LCR_INVALID"]
B["快速路径检查<br/>Empty Check<br/>检查 MultiProgress 是否为空<br/>快速判断避免遍历"]
C["多进度比较<br/>MultiProgress Compare<br/>遍历每个 hashId<br/>调用 CompareProgress()"]
end
subgraph Optimization["性能优化策略 Performance Optimization"]
direction LR
D["快速路径优化<br/>Fast Path<br/>数据源不同直接返回<br/>空Progress快速判断"]
E["短路优化<br/>Short Circuit<br/>发现更慢立即返回<br/>减少不必要的比较"]
F["缓存优化<br/>Cache<br/>缓存比较结果<br/>避免重复计算"]
end
subgraph Results["比较结果类型 Compare Results"]
direction LR
G["LCR_INVALID<br/>数据源不同<br/>无法比较"]
H["LCR_SLOWER<br/>整体更慢<br/>数据未处理"]
I["LCR_PARTIAL_FASTER<br/>部分更快<br/>需要部分处理"]
J["LCR_FULLY_FASTER<br/>完全更快<br/>数据已处理"]
end
Steps -->|产生| Results
Steps -->|采用| Optimization
A -->|第一步| B
B -->|第二步| C
C -->|第三步| Results
style Steps fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Optimization fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Results fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style H fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style I fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style J fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
CompareProgress() 是比较单个 hashId 进度的核心方法：
// framework/Locator.cpp
LocatorCompareResult Locator::CompareProgress(const ProgressVector& pv1,
const ProgressVector& pv2) const
{
// 1. 快速路径：如果 ProgressVector 为空
if (pv1.empty()) {
if (pv2.empty()) {
return LCR_FULLY_FASTER; // 都为空，认为相等
}
return LCR_SLOWER; // pv1 为空，pv2 不为空，pv1 更慢
}
if (pv2.empty()) {
return LCR_FULLY_FASTER; // pv1 不为空，pv2 为空，pv1 更快
}
// 2. 比较每个 Progress
bool hasPartialFaster = false;
bool hasSlower = false;
// 合并两个 ProgressVector，按 from 排序
std::vector<std::pair<const Progress*, const Progress*>> pairs;
// ... 合并逻辑 ...
for (const auto& pair : pairs) {
const Progress* p1 = pair.first;
const Progress* p2 = pair.second;
if (!p1) {
// p1 没有该范围的 Progress，p2 有，p1 更慢
hasSlower = true;
continue;
}
if (!p2) {
// p1 有该范围的 Progress，p2 没有，p1 部分更快
hasPartialFaster = true;
continue;
}
// 比较 offset
if (p1->offset < p2->offset) {
hasSlower = true;
if (!hasPartialFaster) {
return LCR_SLOWER;
}
} else if (p1->offset > p2->offset) {
hasPartialFaster = true;
} else {
// 相等，继续检查下一个
continue;
}
}
// 3. 汇总结果
if (hasPartialFaster && !hasSlower) {
return LCR_PARTIAL_FASTER;
}
if (hasSlower) {
return LCR_SLOWER;
}
return LCR_FULLY_FASTER;
}
Locator 比较结果的语义：
Locator 比较结果的语义：不同结果的含义和应用场景：
flowchart TB
subgraph Results["Locator 比较结果类型"]
direction LR
A["LCR_INVALID<br/>无效比较<br/>数据源不同"]
B["LCR_SLOWER<br/>更慢<br/>数据未处理"]
C["LCR_PARTIAL_FASTER<br/>部分更快<br/>部分数据已处理"]
D["LCR_FULLY_FASTER<br/>完全更快<br/>数据已处理"]
end
subgraph Meaning["结果含义 Result Meaning"]
direction LR
E["无法比较<br/>数据源不同<br/>跳过处理"]
F["需要处理<br/>有hashId更慢<br/>更新Locator"]
G["部分处理<br/>部分hashId更快<br/>分别处理每个hashId"]
H["跳过处理<br/>所有hashId更快或相等<br/>数据已处理"]
end
subgraph Application["应用场景 Application Scenarios"]
direction LR
I["增量更新<br/>判断数据是否已处理<br/>决定是否需要更新"]
J["数据一致性<br/>保证数据处理的顺序<br/>避免重复处理"]
K["性能优化<br/>跳过已处理数据<br/>减少不必要的处理"]
end
A -->|含义| E
B -->|含义| F
C -->|含义| G
D -->|含义| H
E -->|应用于| I
F -->|应用于| I
G -->|应用于| J
H -->|应用于| K
style Results fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Meaning fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Application fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style B fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style C fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style D fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style H fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style I fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style J fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style K fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
比较结果详解：
stateDiagram-v2
[*] --> LCR_INVALID: 数据源不同
[*] --> LCR_SLOWER: 有hashId更慢
[*] --> LCR_PARTIAL_FASTER: 部分hashId更快
[*] --> LCR_FULLY_FASTER: 所有hashId更快或相等
LCR_INVALID: 无法比较，跳过处理
LCR_SLOWER: 数据未处理，需要处理
LCR_PARTIAL_FASTER: 部分数据已处理，需要部分处理
LCR_FULLY_FASTER: 数据已处理，跳过处理
LCR_INVALID --> [*]
LCR_SLOWER --> [*]
LCR_PARTIAL_FASTER --> [*]
LCR_FULLY_FASTER --> [*]
多进度比较的实现：
多进度比较：比较 MultiProgress 中每个 hashId 的进度：
flowchart TB
Start([开始多进度比较]) --> Init[初始化状态<br/>设置标志位]
Init --> Iterate[遍历每个 hashId<br/>遍历所有hashId]
Iterate --> Compare[调用 CompareProgress<br/>比较该 hashId 的进度]
Compare --> CheckResult{比较结果判断}
CheckResult -->|LCR_SLOWER| CheckPartial{是否有部分更快}
CheckResult -->|LCR_PARTIAL_FASTER| SetPartial[设置 hasPartialFaster]
CheckResult -->|LCR_FULLY_FASTER| CheckNext{还有更多hashId}
CheckPartial -->|否| ReturnSlower[返回 LCR_SLOWER]
CheckPartial -->|是| SetSlower[设置 hasSlower]
SetPartial --> CheckNext
SetSlower --> CheckNext
CheckNext -->|是| Iterate
CheckNext -->|否| CheckSize{比较大小}
CheckSize -->|当前size大于其他size| ReturnPartial[返回 LCR_PARTIAL_FASTER]
CheckSize -->|当前size小于其他size| CheckSlower2{是否有更慢的}
CheckSize -->|当前size等于其他size| Aggregate[汇总结果]
CheckSlower2 -->|是| ReturnSlower
CheckSlower2 -->|否| ReturnPartial
Aggregate -->|有部分更快且无更慢| ReturnPartial
Aggregate -->|有更慢| ReturnSlower
Aggregate -->|无部分更快且无更慢| ReturnFully[返回 LCR_FULLY_FASTER]
ReturnSlower --> End([结束])
ReturnPartial --> End
ReturnFully --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Init fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Iterate fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckPartial fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckNext fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckSlower2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style SetPartial fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SetSlower fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Aggregate fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReturnSlower fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ReturnPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style ReturnFully fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
多进度比较的序列图：
sequenceDiagram
participant Client
participant Locator1
participant Locator2
participant CompareProgress
Client->>Locator1: IsFasterThan(Locator2)
Locator1->>Locator1: IsSameSrc(Locator2)
alt 数据源不同
Locator1-->>Client: LCR_INVALID
else 数据源相同
loop 遍历每个 hashId
Locator1->>CompareProgress: CompareProgress(pv1[i], pv2[i])
CompareProgress->>CompareProgress: 比较 ProgressVector
CompareProgress-->>Locator1: 比较结果
alt 有更慢的 hashId
Locator1-->>Client: LCR_SLOWER
else 有部分更快的 hashId
Locator1-->>Client: LCR_PARTIAL_FASTER
end
end
Locator1-->>Client: LCR_FULLY_FASTER
end
比较流程详解：
CompareProgress() 方法Update() 方法用于更新 Locator，保证 Locator 只向前推进，不会回退。这是数据一致性保证的关键。让我们先通过流程图来理解更新的完整流程：
flowchart TD
Start([开始更新]) --> CheckFaster{检查新 Locator<br/>是否完全更快?}
CheckFaster -->|否| Return[返回，不更新]
CheckFaster -->|是| CheckSrc{检查数据源<br/>是否相同?}
CheckSrc -->|不同| Return
CheckSrc -->|相同| UpdateMulti[更新 MultiProgress]
UpdateMulti --> MergeProgress[合并 ProgressVector]
MergeProgress --> UpdateMin[更新 MinOffset]
UpdateMin --> UpdateUserData{需要更新<br/>UserData?}
UpdateUserData -->|是| SetUserData[设置 UserData]
UpdateUserData -->|否| End([结束])
SetUserData --> End
Return --> End
Update() 的完整实现：
// framework/Locator.cpp
void Locator::Update(const Locator& other)
{
// 1. 检查数据源是否相同
if (!IsSameSrc(other)) {
// 数据源不同，不更新
return;
}
// 2. 检查新 Locator 是否完全更快
auto result = other.IsFasterThan(*this);
if (result != LCR_FULLY_FASTER) {
// 新 Locator 不是完全更快，不更新
// 这保证了 Locator 只向前推进，不会回退
return;
}
// 3. 更新 MultiProgress
// 合并两个 MultiProgress，保留更大的进度
if (other._multiProgress.size() > _multiProgress.size()) {
_multiProgress = other._multiProgress;
} else {
// 逐个 hashId 合并，保留更大的进度
for (size_t i = 0; i < other._multiProgress.size(); ++i) {
if (i >= _multiProgress.size()) {
_multiProgress.push_back(other._multiProgress[i]);
} else {
// 合并 ProgressVector
MergeProgressVector(_multiProgress[i], other._multiProgress[i]);
}
}
}
// 4. 更新 MinOffset
UpdateMinOffset();
// 5. 更新 UserData（如果新 Locator 有 UserData）
if (!other._userData.empty()) {
_userData = other._userData;
}
}
MergeProgressVector() 的实现：
// framework/Locator.cpp
void Locator::MergeProgressVector(ProgressVector& pv1,
const ProgressVector& pv2)
{
// 合并两个 ProgressVector，保留更大的进度
// 1. 按 from 排序
std::sort(pv1.begin(), pv1.end(),
[](const Progress& a, const Progress& b) {
return a.from < b.from;
});
// 2. 合并重叠的 Progress
ProgressVector merged;
for (const auto& p : pv1) {
bool merged = false;
for (auto& m : merged) {
if (m.from <= p.to && m.to >= p.from) {
// 有重叠，合并
m.from = std::min(m.from, p.from);
m.to = std::max(m.to, p.to);
if (p.offset > m.offset) {
m.offset = p.offset; // 保留更大的进度
}
merged = true;
break;
}
}
if (!merged) {
merged.push_back(p);
}
}
// 3. 与 pv2 合并
for (const auto& p : pv2) {
bool merged = false;
for (auto& m : merged) {
if (m.from <= p.to && m.to >= p.from) {
m.from = std::min(m.from, p.from);
m.to = std::max(m.to, p.to);
if (p.offset > m.offset) {
m.offset = p.offset;
}
merged = true;
break;
}
}
if (!merged) {
merged.push_back(p);
}
}
pv1 = merged;
}
UpdateMinOffset() 的实现：
// framework/Locator.cpp
void Locator::UpdateMinOffset()
{
if (_multiProgress.empty()) {
_minOffset = Progress::INVALID_OFFSET;
return;
}
// 找到所有 Progress 中最小的 offset
Progress::Offset minOffset = Progress::MAX_OFFSET;
for (const auto& pv : _multiProgress) {
for (const auto& p : pv) {
if (p.offset < minOffset) {
minOffset = p.offset;
}
}
}
_minOffset = minOffset;
}
更新机制的关键设计：
_minOffset，用于快速判断整体进度Update() 方法：更新 Locator 的实现逻辑：
flowchart TB
Start([开始更新<br/>Update方法]) --> CheckSrc{检查数据源<br/>IsSameSrc}
CheckSrc -->|数据源不同| Return1[返回，不更新]
CheckSrc -->|数据源相同| CheckFaster{检查新Locator<br/>是否完全更快<br/>IsFasterThan}
CheckFaster -->|不是完全更快| Return2[返回，不更新<br/>保证只向前推进]
CheckFaster -->|完全更快| CheckSize{比较MultiProgress<br/>大小关系}
CheckSize -->|other.size大于当前size| Replace[直接替换<br/>_multiProgress = other._multiProgress]
CheckSize -->|other.size小于等于当前size| Merge[逐个hashId合并<br/>MergeProgressVector]
Replace --> UpdateMin[更新MinOffset<br/>UpdateMinOffset方法]
Merge --> UpdateMin
UpdateMin --> CheckUserData{新Locator是否有<br/>UserData}
CheckUserData -->|有UserData| SetUserData[设置UserData<br/>_userData = other._userData]
CheckUserData -->|无UserData| End([结束])
SetUserData --> End
Return1 --> End
Return2 --> End
subgraph MergeDetail["MergeProgressVector 详细流程"]
direction TB
M1[按from排序<br/>std::sort]
M2[合并重叠的Progress<br/>保留更大的offset]
M3[与pv2合并<br/>处理重叠和大小]
end
Merge -.->|调用| MergeDetail
M1 --> M2
M2 --> M3
subgraph MinOffsetDetail["UpdateMinOffset 详细流程"]
direction TB
O1{MultiProgress<br/>是否为空}
O2[设置为INVALID_OFFSET]
O3[遍历所有Progress<br/>找到最小offset]
end
UpdateMin -.->|调用| MinOffsetDetail
O1 -->|是| O2
O1 -->|否| O3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckSrc fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckFaster fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckUserData fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Replace fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Merge fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateMin fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SetUserData fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Return1 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Return2 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style MergeDetail fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style MinOffsetDetail fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style M1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style M2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style M3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style O1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style O2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style O3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
Locator 的更新时机：
Locator 的更新时机：在数据处理完成后更新 Locator：
flowchart TB
subgraph Trigger["更新触发时机 Update Triggers"]
direction LR
A["数据处理完成<br/>TabletWriter::Build<br/>处理完一批文档后<br/>更新MemSegment的Locator"]
B["Segment构建完成<br/>MemSegment::Seal<br/>Segment构建完成后<br/>更新Locator"]
C["版本提交时<br/>VersionCommitter::Commit<br/>版本提交时<br/>设置Version的Locator"]
D["增量更新时<br/>IncrementalUpdate<br/>处理完新数据后<br/>更新Locator记录处理位置"]
end
subgraph Action["更新操作 Update Actions"]
direction LR
E["调用Update方法<br/>Locator::Update<br/>检查是否完全更快<br/>合并MultiProgress"]
F["更新MinOffset<br/>UpdateMinOffset<br/>重新计算最小偏移量"]
G["持久化Locator<br/>Version::SetLocator<br/>保存到Version中"]
end
subgraph Purpose["更新目的 Update Purpose"]
direction LR
H["反映最新位置<br/>保证Locator反映<br/>最新的数据处理位置"]
I["保证一致性<br/>保证数据处理的<br/>顺序和一致性"]
J["支持增量更新<br/>记录处理位置<br/>支持下次增量更新"]
end
A -->|触发| E
B -->|触发| E
C -->|触发| G
D -->|触发| E
E -->|包含| F
E -->|实现| H
G -->|实现| I
E -->|实现| J
style Trigger fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Action fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Purpose fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#90caf9,stroke:#1976d2,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style H fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style I fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style J fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
更新时机的序列图：
sequenceDiagram
participant DataSource
participant TabletWriter
participant MemSegment
participant Locator
participant Version
DataSource->>TabletWriter: 写入数据
TabletWriter->>MemSegment: Build(doc)
MemSegment->>MemSegment: 处理文档
MemSegment->>Locator: 更新 Locator
Locator->>Locator: Update(newLocator)
MemSegment->>MemSegment: Flush()
MemSegment->>Locator: 获取 Locator
MemSegment->>Version: 提交版本
Version->>Version: SetLocator(locator)
Note over Version: 版本提交时，Locator 被持久化
更新时机详解：
TabletWriter::Build() 中，每处理完一批文档，更新 MemSegment 的 LocatorMemSegment::Seal() 中，Segment 构建完成后，更新 LocatorVersionCommitter::Commit() 中，版本提交时，将 TabletWriter 的 Locator 设置到 Version 中Serialize() 方法用于序列化 Locator，将 Locator 持久化到磁盘或网络传输。序列化格式需要支持版本兼容和向后兼容。让我们先通过流程图来理解序列化的完整流程：
flowchart TB
Start([开始序列化<br/>Serialize方法]) --> InitBuffer[构建序列化缓冲区<br/>autil::DataBuffer]
InitBuffer --> WriteHeader[写入头部信息]
subgraph HeaderGroup["头部信息 Header"]
direction LR
H1[Magic Number<br/>0x4C4F4341 LOCA<br/>uint32_t 验证标识]
H2[Version<br/>版本号 2<br/>uint32_t 兼容性]
end
WriteHeader --> WriteBasic[写入基础字段]
subgraph BasicGroup["基础字段 Basic Fields"]
direction TB
B1[Src<br/>数据源标识<br/>uint64_t]
B2[MinOffset<br/>timestamp int64_t<br/>concurrentIdx uint32_t]
end
WriteBasic --> WriteMulti[写入 MultiProgress]
subgraph MultiGroup["MultiProgress 序列化"]
direction TB
M1[写入 hashId 数量<br/>uint32_t]
M2[循环遍历每个 hashId]
M3[写入 ProgressVector]
end
subgraph PVGroup["ProgressVector 序列化"]
direction TB
P1[写入 Progress 数量<br/>uint32_t]
P2[循环遍历每个 Progress]
P3[写入 Progress 数据<br/>from uint32_t<br/>to uint32_t<br/>offset timestamp + concurrentIdx]
end
WriteMulti --> WriteUser[写入 UserData]
M2 -->|对每个hashId| M3
M3 -->|调用| PVGroup
P2 -->|对每个Progress| P3
subgraph UserGroup["UserData 序列化"]
direction TB
U1[写入数据长度<br/>uint32_t]
U2{数据是否为空}
U3[写入数据内容<br/>writeBytes]
end
WriteUser --> WriteLegacy[写入 Legacy 标志<br/>_isLegacyLocator<br/>uint8_t]
WriteLegacy --> ToString[转换为字符串<br/>buffer.toString]
ToString --> CheckSize{数据大小<br/>是否大于1KB}
CheckSize -->|是| Compress[压缩数据<br/>Compress方法]
CheckSize -->|否| End([结束<br/>返回序列化结果])
Compress --> End
WriteHeader -.->|包含| HeaderGroup
WriteBasic -.->|包含| BasicGroup
WriteMulti -.->|包含| MultiGroup
WriteUser -.->|包含| UserGroup
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style InitBuffer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteHeader fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteBasic fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteUser fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteLegacy fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ToString fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Compress fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style HeaderGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style H1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style H2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style BasicGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style B1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style MultiGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style PVGroup fill:#e1f5fe,stroke:#0277bd,stroke-width:3px
style P1 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style P2 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style P3 fill:#81d4fa,stroke:#0277bd,stroke-width:2px
style UserGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style U1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
Serialize() 的完整实现：
// framework/Locator.cpp
std::string Locator::Serialize() const
{
// 1. 构建序列化缓冲区
autil::DataBuffer buffer;
// 2. 写入 Magic Number（用于验证）
const uint32_t MAGIC_NUMBER = 0x4C4F4341; // "LOCA"
buffer.write(MAGIC_NUMBER);
// 3. 写入 Version（用于兼容性）
const uint32_t VERSION = 2; // 当前版本
buffer.write(VERSION);
// 4. 写入 Src
buffer.write(_src);
// 5. 写入 MinOffset
buffer.write(_minOffset.first); // timestamp
buffer.write(_minOffset.second); // concurrentIdx
// 6. 写入 MultiProgress
buffer.write(static_cast<uint32_t>(_multiProgress.size()));
for (const auto& pv : _multiProgress) {
buffer.write(static_cast<uint32_t>(pv.size()));
for (const auto& p : pv) {
buffer.write(p.from);
buffer.write(p.to);
buffer.write(p.offset.first); // timestamp
buffer.write(p.offset.second); // concurrentIdx
}
}
// 7. 写入 UserData
buffer.write(static_cast<uint32_t>(_userData.size()));
if (!_userData.empty()) {
buffer.writeBytes(_userData.data(), _userData.size());
}
// 8. 写入 Legacy 标志
buffer.write(static_cast<uint8_t>(_isLegacyLocator ? 1 : 0));
// 9. 转换为字符串（可选：压缩）
std::string result = buffer.toString();
// 可选：压缩序列化数据
if (result.size() > 1024) { // 大于 1KB 时压缩
result = Compress(result);
}
return result;
}
序列化格式详解：
0x4C4F4341（”LOCA”），用于验证数据格式是否正确Locator 的序列化：将 Locator 序列化为字符串：
flowchart TB
Start([开始序列化<br/>Serialize方法]) --> InitBuffer[初始化缓冲区<br/>autil::DataBuffer]
InitBuffer --> WriteHeader[写入头部信息]
subgraph Header["头部信息 Header 8字节"]
direction TB
H1["Magic Number<br/>0x4C4F4341 LOCA<br/>uint32_t 4字节<br/>格式验证标识"]
H2["Version<br/>版本号 2<br/>uint32_t 4字节<br/>兼容性控制"]
end
WriteHeader --> WriteBasic[写入基础字段]
subgraph Basic["基础字段 Basic Fields 20字节"]
direction TB
B1["Src<br/>数据源标识<br/>uint64_t 8字节"]
B2["MinOffset<br/>最小偏移量<br/>timestamp int64_t 8字节<br/>concurrentIdx uint32_t 4字节"]
end
WriteBasic --> WriteMulti[写入MultiProgress]
subgraph Multi["MultiProgress 可变长度"]
direction TB
M1["hashId数量<br/>uint32_t 4字节"]
M2["ProgressVector数组<br/>每个hashId一个<br/>嵌套结构"]
M3["Progress数据<br/>from uint32_t 4字节<br/>to uint32_t 4字节<br/>offset 12字节"]
end
WriteMulti --> WriteUser[写入UserData]
subgraph User["UserData 可变长度"]
direction TB
U1["数据长度<br/>uint32_t 4字节"]
U2["数据内容<br/>可选字段<br/>writeBytes"]
end
WriteUser --> WriteLegacy[写入Legacy标志<br/>_isLegacyLocator<br/>uint8_t 1字节]
WriteLegacy --> ToString[转换为字符串<br/>buffer.toString]
ToString --> CheckSize{数据大小<br/>是否大于1KB}
CheckSize -->|是| Compress[压缩数据<br/>Compress方法<br/>LZ4或Snappy]
CheckSize -->|否| End([结束<br/>返回序列化结果])
Compress --> End
WriteHeader -.->|包含| Header
WriteBasic -.->|包含| Basic
WriteMulti -.->|包含| Multi
WriteUser -.->|包含| User
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style InitBuffer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteHeader fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteBasic fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteUser fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WriteLegacy fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ToString fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Compress fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Header fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style H1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style H2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Basic fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style B1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Multi fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style User fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style U1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
Deserialize() 方法用于反序列化 Locator，从字符串恢复 Locator 对象。需要支持版本兼容和向后兼容。让我们先通过流程图来理解反序列化的完整流程：
flowchart TB
Start([开始反序列化<br/>Deserialize方法]) --> CheckEmpty{字符串<br/>是否为空}
CheckEmpty -->|是| Error1[返回错误<br/>Empty string]
CheckEmpty -->|否| CheckCompress{数据是否压缩<br/>IsCompressed}
CheckCompress -->|是| Decompress[解压数据<br/>Decompress方法]
CheckCompress -->|否| InitBuffer[构建缓冲区<br/>autil::DataBuffer]
Decompress -->|成功| InitBuffer
Decompress -->|失败| Error2[返回错误<br/>Decompress failed]
InitBuffer --> ReadMagic[读取Magic Number<br/>uint32_t]
ReadMagic --> CheckReadMagic{读取<br/>是否成功}
CheckReadMagic -->|失败| Error3[返回错误<br/>Failed to read magic]
CheckReadMagic -->|成功| CheckMagic{验证Magic Number<br/>是否为0x4C4F4341}
CheckMagic -->|失败| Error4[返回错误<br/>Invalid magic number]
CheckMagic -->|成功| ReadVersion[读取Version<br/>uint32_t]
ReadVersion --> CheckReadVersion{读取<br/>是否成功}
CheckReadVersion -->|失败| Error5[返回错误<br/>Failed to read version]
CheckReadVersion -->|成功| CheckVersion{检查版本<br/>Version 1或2}
CheckVersion -->|不支持| Error6[返回错误<br/>Unsupported version]
CheckVersion -->|V1| DeserializeV1[调用DeserializeV1<br/>V1格式解析]
CheckVersion -->|V2| DeserializeV2[调用DeserializeV2<br/>V2格式解析]
DeserializeV1 --> ReadFields[读取基础字段]
DeserializeV2 --> ReadFields
subgraph Fields["基础字段读取"]
direction TB
F1[读取Src<br/>uint64_t 8字节]
F2[读取MinOffset<br/>timestamp int64_t 8字节<br/>concurrentIdx uint32_t 4字节]
end
ReadFields --> ReadMulti[读取MultiProgress]
subgraph Multi["MultiProgress读取"]
direction TB
M1[读取hashId数量<br/>uint32_t]
M2[遍历每个hashId<br/>for each hashId]
M3[读取ProgressVector大小<br/>uint32_t]
M4[遍历每个Progress<br/>for each Progress]
M5[读取Progress数据<br/>from to offset]
end
ReadMulti --> ReadUser[读取UserData]
subgraph User["UserData读取"]
direction TB
U1[读取数据长度<br/>uint32_t]
U2{长度是否<br/>大于0}
U3[读取数据内容<br/>readBytes]
end
ReadUser --> ReadLegacy[读取Legacy标志<br/>_isLegacyLocator<br/>uint8_t]
ReadLegacy --> Validate[验证数据<br/>IsValid检查]
Validate -->|失败| Error7[返回错误<br/>Invalid locator]
Validate -->|成功| End([结束<br/>返回Status::OK])
Error1 --> End
Error2 --> End
Error3 --> End
Error4 --> End
Error5 --> End
Error6 --> End
Error7 --> End
ReadFields -.->|包含| Fields
ReadMulti -.->|包含| Multi
ReadUser -.->|包含| User
M2 -->|循环| M3
M3 -->|循环| M4
M4 -->|循环| M5
M5 -->|继续| M2
U2 -->|是| U3
U2 -->|否| ReadLegacy
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckEmpty fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckCompress fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Decompress fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style InitBuffer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadMagic fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckReadMagic fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckMagic fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ReadVersion fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckReadVersion fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckVersion fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DeserializeV1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style DeserializeV2 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadFields fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadUser fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadLegacy fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Validate fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Error1 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error2 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error3 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error4 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error5 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error6 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error7 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Fields fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Multi fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style User fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style U1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style U3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
Deserialize() 的完整实现：
// framework/Locator.cpp
Status Locator::Deserialize(const std::string& str)
{
if (str.empty()) {
return Status::InvalidArgs("Empty string");
}
// 1. 尝试解压（如果压缩了）
std::string data = str;
if (IsCompressed(str)) {
auto status = Decompress(str, data);
if (!status.IsOK()) {
return status;
}
}
// 2. 构建反序列化缓冲区
autil::DataBuffer buffer(data.data(), data.size());
// 3. 读取并验证 Magic Number
uint32_t magic;
if (!buffer.read(magic)) {
return Status::InvalidArgs("Failed to read magic number");
}
if (magic != 0x4C4F4341) {
return Status::InvalidArgs("Invalid magic number");
}
// 4. 读取 Version
uint32_t version;
if (!buffer.read(version)) {
return Status::InvalidArgs("Failed to read version");
}
// 5. 根据版本选择解析方式
if (version == 1) {
return DeserializeV1(buffer);
} else if (version == 2) {
return DeserializeV2(buffer);
} else {
return Status::InvalidArgs("Unsupported version: " + std::to_string(version));
}
}
Status Locator::DeserializeV2(autil::DataBuffer& buffer)
{
// 1. 读取 Src
if (!buffer.read(_src)) {
return Status::InvalidArgs("Failed to read src");
}
// 2. 读取 MinOffset
int64_t timestamp;
uint32_t concurrentIdx;
if (!buffer.read(timestamp) || !buffer.read(concurrentIdx)) {
return Status::InvalidArgs("Failed to read min offset");
}
_minOffset = std::make_pair(timestamp, concurrentIdx);
// 3. 读取 MultiProgress
uint32_t multiProgressSize;
if (!buffer.read(multiProgressSize)) {
return Status::InvalidArgs("Failed to read multi progress size");
}
_multiProgress.clear();
_multiProgress.reserve(multiProgressSize);
for (uint32_t i = 0; i < multiProgressSize; ++i) {
uint32_t pvSize;
if (!buffer.read(pvSize)) {
return Status::InvalidArgs("Failed to read progress vector size");
}
ProgressVector pv;
pv.reserve(pvSize);
for (uint32_t j = 0; j < pvSize; ++j) {
uint32_t from, to;
int64_t ts;
uint32_t idx;
if (!buffer.read(from) || !buffer.read(to) ||
!buffer.read(ts) || !buffer.read(idx)) {
return Status::InvalidArgs("Failed to read progress");
}
pv.emplace_back(from, to, std::make_pair(ts, idx));
}
_multiProgress.push_back(std::move(pv));
}
// 4. 读取 UserData
uint32_t userDataSize;
if (!buffer.read(userDataSize)) {
return Status::InvalidArgs("Failed to read user data size");
}
if (userDataSize > 0) {
_userData.resize(userDataSize);
if (!buffer.readBytes(_userData.data(), userDataSize)) {
return Status::InvalidArgs("Failed to read user data");
}
} else {
_userData.clear();
}
// 5. 读取 Legacy 标志
uint8_t legacyFlag;
if (!buffer.read(legacyFlag)) {
return Status::InvalidArgs("Failed to read legacy flag");
}
_isLegacyLocator = (legacyFlag != 0);
// 6. 验证数据
if (!IsValid()) {
return Status::InvalidArgs("Invalid locator after deserialization");
}
return Status::OK();
}
反序列化的关键设计：
Locator 的反序列化：从字符串反序列化为 Locator：
flowchart TB
Start([开始反序列化<br/>Deserialize方法]) --> CheckEmpty{字符串<br/>是否为空}
CheckEmpty -->|是| Error1[返回错误<br/>Empty string]
CheckEmpty -->|否| CheckCompress{检查是否压缩<br/>IsCompressed}
CheckCompress -->|是| Decompress[解压数据<br/>Decompress方法]
CheckCompress -->|否| InitBuffer[构建缓冲区<br/>autil::DataBuffer]
Decompress -->|成功| InitBuffer
Decompress -->|失败| Error2[返回错误<br/>Decompress failed]
InitBuffer --> ReadMagic[读取Magic Number<br/>uint32_t]
ReadMagic --> CheckReadMagic{读取<br/>是否成功}
CheckReadMagic -->|失败| Error3[返回错误<br/>Failed to read magic]
CheckReadMagic -->|成功| CheckMagic{验证Magic Number<br/>是否为0x4C4F4341}
CheckMagic -->|失败| Error4[返回错误<br/>Invalid magic number]
CheckMagic -->|成功| ReadVersion[读取Version<br/>uint32_t]
ReadVersion --> CheckReadVersion{读取<br/>是否成功}
CheckReadVersion -->|失败| Error5[返回错误<br/>Failed to read version]
CheckReadVersion -->|成功| CheckVersion{检查版本<br/>Version 1或2}
CheckVersion -->|不支持| Error6[返回错误<br/>Unsupported version]
CheckVersion -->|V1| DeserializeV1[调用DeserializeV1<br/>V1格式解析]
CheckVersion -->|V2| DeserializeV2[调用DeserializeV2<br/>V2格式解析]
DeserializeV1 --> ReadFields[读取基础字段]
DeserializeV2 --> ReadFields
subgraph Fields["基础字段读取"]
direction TB
F1[读取Src<br/>uint64_t 8字节]
F2[读取MinOffset<br/>timestamp int64_t 8字节<br/>concurrentIdx uint32_t 4字节]
end
ReadFields --> ReadMulti[读取MultiProgress]
subgraph Multi["MultiProgress读取"]
direction TB
M1[读取hashId数量<br/>uint32_t]
M2[遍历每个hashId<br/>for i in multiProgressSize]
M3[读取ProgressVector大小<br/>uint32_t]
M4[遍历每个Progress<br/>for j in pvSize]
M5[读取Progress数据<br/>from uint32_t 4字节<br/>to uint32_t 4字节<br/>offset 12字节]
end
ReadMulti --> ReadUser[读取UserData]
subgraph User["UserData读取"]
direction TB
U1[读取数据长度<br/>uint32_t]
U2{长度是否<br/>大于0}
U3[读取数据内容<br/>readBytes]
end
ReadUser --> ReadLegacy[读取Legacy标志<br/>_isLegacyLocator<br/>uint8_t]
ReadLegacy --> Validate[验证数据<br/>IsValid检查]
Validate -->|失败| Error7[返回错误<br/>Invalid locator]
Validate -->|成功| End([结束<br/>返回Status::OK])
Error1 --> End
Error2 --> End
Error3 --> End
Error4 --> End
Error5 --> End
Error6 --> End
Error7 --> End
ReadFields -.->|包含| Fields
ReadMulti -.->|包含| Multi
ReadUser -.->|包含| User
M1 --> M2
M2 --> M3
M3 --> M4
M4 --> M5
M5 -->|继续循环| M2
M2 -->|循环结束| ReadUser
U1 --> U2
U2 -->|是| U3
U2 -->|否| ReadLegacy
U3 --> ReadLegacy
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckEmpty fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckCompress fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Decompress fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style InitBuffer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadMagic fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckReadMagic fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckMagic fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ReadVersion fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckReadVersion fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckVersion fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DeserializeV1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style DeserializeV2 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadFields fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadUser fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadLegacy fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Validate fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Error1 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error2 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error3 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error4 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error5 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error6 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Error7 fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Fields fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Multi fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style M1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style User fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style U1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style U3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
数据一致性是 IndexLib 的核心保证，通过 Locator 实现数据不重复、不丢失，支持多数据源场景。让我们先通过流程图来理解数据一致性保证的完整机制：
flowchart TB
Start([数据到达<br/>Document Arrival]) --> GetLocator[获取文档Locator<br/>doc.GetLocator]
GetLocator --> Compare[调用IsFasterThan<br/>比较docLocator和currentLocator<br/>IsFasterThan方法]
Compare --> CheckResult{比较结果判断<br/>LocatorCompareResult}
CheckResult -->|LCR_FULLY_FASTER<br/>数据已处理| Skip[跳过处理<br/>Skip Processing<br/>数据已完全处理<br/>避免重复处理]
CheckResult -->|LCR_SLOWER<br/>数据未处理| Process[处理新数据<br/>Process New Data<br/>Build文档<br/>构建索引]
CheckResult -->|LCR_PARTIAL_FASTER<br/>部分已处理| ProcessPartial[部分处理<br/>Partial Processing<br/>处理未处理部分<br/>部分构建索引]
CheckResult -->|LCR_INVALID<br/>数据源不同| CheckSrc{检查数据源<br/>IsSameSrc}
CheckSrc -->|数据源不同<br/>不同数据源| ProcessMulti[多数据源处理<br/>Multi-Source Processing<br/>根据数据源选择Locator<br/>独立处理]
CheckSrc -->|数据源相同<br/>但比较无效| Error[错误处理<br/>Error Handling<br/>数据源相同但比较无效<br/>返回错误]
Process --> UpdateLocator[更新Locator<br/>Update Locator<br/>调用Update方法<br/>合并MultiProgress]
ProcessPartial --> UpdateLocator
ProcessMulti --> UpdateLocator
UpdateLocator --> Commit[提交版本<br/>Commit Version<br/>VersionCommitter::Commit<br/>创建新版本]
Commit --> Persist[持久化Locator<br/>Persist Locator<br/>序列化Locator<br/>写入Version文件]
Persist --> End([结束<br/>End])
Skip --> End
Error --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style GetLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckSrc fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Skip fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style Process fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ProcessPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style ProcessMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Error fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style UpdateLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Commit fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Persist fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
通过 Locator 保证数据不重复，这是增量更新的基础。让我们通过序列图来理解数据不重复保证的完整流程：
sequenceDiagram
participant DataSource
participant TabletWriter
participant Locator
participant MemSegment
DataSource->>TabletWriter: 写入数据(doc)
TabletWriter->>Locator: IsFasterThan(doc.locator)
Locator->>Locator: 比较 MultiProgress
alt 数据已处理 (LCR_FULLY_FASTER)
Locator-->>TabletWriter: LCR_FULLY_FASTER
TabletWriter->>TabletWriter: 跳过该文档
else 数据未处理 (LCR_SLOWER)
Locator-->>TabletWriter: LCR_SLOWER
TabletWriter->>MemSegment: Build(doc)
MemSegment->>MemSegment: 处理文档
MemSegment->>Locator: Update(newLocator)
else 部分已处理 (LCR_PARTIAL_FASTER)
Locator-->>TabletWriter: LCR_PARTIAL_FASTER
TabletWriter->>TabletWriter: 部分处理该文档
TabletWriter->>MemSegment: Build(doc, partial)
end
数据不重复保证的实现：
// framework/TabletWriter.cpp
Status TabletWriter::Build(const Document& doc)
{
// 1. 获取文档的 Locator
Locator docLocator = doc.GetLocator();
// 2. 检查数据是否已处理
auto result = docLocator.IsFasterThan(_currentLocator);
if (result == Locator::LCR_FULLY_FASTER) {
// 数据已处理，跳过
return Status::OK();
}
if (result == Locator::LCR_INVALID) {
// 数据源不同，需要特殊处理
return HandleDifferentSource(doc);
}
// 3. 处理新数据
if (result == Locator::LCR_SLOWER) {
// 数据未处理，正常处理
return ProcessDocument(doc);
}
if (result == Locator::LCR_PARTIAL_FASTER) {
// 部分数据已处理，需要部分处理
return ProcessPartialDocument(doc);
}
return Status::OK();
}
保证机制详解：
IsFasterThan() 判断数据是否已处理
LCR_FULLY_FASTER，说明数据已处理，跳过LCR_SLOWER，说明数据未处理，需要处理LCR_PARTIAL_FASTER，说明部分数据已处理，需要部分处理数据不重复保证：通过 Locator 比较避免重复处理数据：
flowchart TB
subgraph Mechanism["数据不重复保证机制"]
direction LR
A["Locator比较<br/>IsFasterThan方法<br/>判断数据是否已处理<br/>返回比较结果"]
B["重复检测<br/>DuplicateDetection<br/>LCR_FULLY_FASTER时<br/>检测到已处理数据"]
C["跳过处理<br/>SkipProcessing<br/>跳过已处理数据<br/>避免重复计算"]
end
subgraph Result["比较结果处理"]
direction LR
D["LCR_FULLY_FASTER<br/>数据已处理<br/>直接跳过"]
E["LCR_SLOWER<br/>数据未处理<br/>正常处理"]
F["LCR_PARTIAL_FASTER<br/>部分已处理<br/>部分处理"]
G["LCR_INVALID<br/>数据源不同<br/>特殊处理"]
end
subgraph Benefit["保证效果"]
direction LR
H["数据一致性<br/>保证数据不重复<br/>避免重复处理"]
I["性能优化<br/>减少不必要计算<br/>提高处理效率"]
end
Mechanism -->|产生| Result
Result -->|实现| Benefit
A -->|返回| D
A -->|返回| E
A -->|返回| F
A -->|返回| G
style Mechanism fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Result fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Benefit fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style E fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style F fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style G fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style H fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style I fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
通过 Locator 保证数据不丢失，这是数据可靠性的基础。让我们通过序列图来理解数据不丢失保证的完整流程：
sequenceDiagram
participant DataSource
participant TabletWriter
participant MemSegment
participant Locator
participant Version
participant Disk
DataSource->>TabletWriter: 写入数据
TabletWriter->>MemSegment: Build(doc)
MemSegment->>MemSegment: 处理文档
MemSegment->>Locator: Update(newLocator)
Locator->>Locator: 更新 MultiProgress
MemSegment->>MemSegment: Flush()
MemSegment->>Locator: 获取 Locator
MemSegment->>Version: 提交版本
Version->>Version: SetLocator(locator)
Version->>Disk: 持久化 Version
Note over Disk: Locator 被持久化到磁盘
alt 故障恢复
Disk->>Version: 加载 Version
Version->>Version: 获取 Locator
Version->>TabletWriter: 设置 Locator
TabletWriter->>DataSource: 从 Locator 位置继续处理
end
数据不丢失保证的实现：
// framework/VersionCommitter.cpp
Status VersionCommitter::Commit(const TabletData& tabletData,
const Schema& schema,
const CommitOptions& options)
{
// 1. 获取 TabletWriter 的 Locator
Locator currentLocator = tabletData.GetLocator();
// 2. 创建新版本
Version newVersion = CreateNewVersion(tabletData);
// 3. 设置 Locator
newVersion.SetLocator(currentLocator);
// 4. 持久化版本
auto status = WriteVersion(newVersion);
if (!status.IsOK()) {
return status;
}
// 5. 持久化 Locator（在 Version 中）
// Locator 会被序列化并写入版本文件
return Status::OK();
}
保证机制详解：
数据不丢失保证：通过 Locator 记录处理位置，保证数据不丢失：
flowchart TB
Start([开始数据处理<br/>Data Processing]) --> Process[处理数据<br/>Process Data<br/>TabletWriter::Build<br/>处理文档]
Process --> UpdateLocator[更新Locator<br/>Update Locator<br/>Locator::Update<br/>记录处理位置]
subgraph Recording["位置记录机制 Position Recording"]
direction TB
R1[记录每个hashId进度<br/>MultiProgress更新<br/>记录timestamp和concurrentIdx]
R2[更新MinOffset<br/>快速判断整体进度<br/>最小偏移量维护]
R3[合并Progress<br/>MergeProgressVector<br/>保留更大进度]
end
UpdateLocator --> Commit[提交版本<br/>Commit Version<br/>VersionCommitter::Commit<br/>创建新版本]
Commit --> Persist[持久化Locator<br/>Persist Locator<br/>序列化Locator<br/>写入Version文件]
Persist --> NormalEnd([正常结束<br/>Normal End])
subgraph Recovery["故障恢复流程 Fault Recovery"]
direction TB
F1[检测故障<br/>Fault Detection<br/>系统故障或重启]
F2[加载版本<br/>Load Version<br/>加载故障前版本<br/>Version::GetLocator]
F3[恢复Locator<br/>Recover Locator<br/>反序列化Locator<br/>恢复处理位置]
F4[从位置继续<br/>Continue from Position<br/>IsFasterThan判断<br/>只处理未处理数据]
F5[增量处理<br/>Incremental Processing<br/>避免重复处理<br/>保证数据不丢失]
end
F1 -->|触发| F2
F2 --> F3
F3 --> F4
F4 --> F5
F5 --> Process
subgraph Guarantee["保证效果 Guarantee Effects"]
direction LR
G1[数据完整性<br/>Data Integrity<br/>保证数据不丢失<br/>支持故障恢复]
G2[版本一致性<br/>Version Consistency<br/>每个版本有Locator<br/>保证版本数据一致]
G3[增量更新<br/>Incremental Update<br/>只处理新数据<br/>提高处理效率]
end
NormalEnd -->|实现| G1
F5 -->|实现| G1
Persist -->|实现| G2
UpdateLocator -->|实现| G3
UpdateLocator -.->|包含| Recording
R1 --> R2
R2 --> R3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style NormalEnd fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Process fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Commit fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Persist fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Recording fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style R1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style R2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style R3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Recovery fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style F1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Guarantee fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style G1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style G2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style G3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
多数据源场景下的数据一致性，通过 _src 和 sourceIdx 区分数据源，每个数据源有独立的 Locator。让我们通过类图来理解多数据源一致性的架构：
classDiagram
class Version {
- versionid_t _versionId
- map_uint64_t_Locator _locators
+ GetLocator()
+ SetLocator()
+ GetAllLocators()
}
class Locator {
- uint64_t _src
- MultiProgress _multiProgress
+ IsFasterThan()
+ Update()
}
class TabletWriter {
- map_uint64_t_Locator _locators
+ Build()
+ GetLocator()
}
class Document {
+ uint64_t _src
+ DocInfo _docInfo
+ GetLocator()
}
Version --> Locator : 包含多个
TabletWriter --> Locator : 管理多个
Document --> Locator : 包含
多数据源一致性的实现：
// framework/Version.h
class Version
{
private:
std::map<uint64_t, Locator> _locators; // 每个数据源的 Locator
public:
Locator GetLocator(uint64_t src) const {
auto it = _locators.find(src);
if (it != _locators.end()) {
return it->second;
}
return Locator(src); // 返回空的 Locator
}
void SetLocator(uint64_t src, const Locator& locator) {
_locators[src] = locator;
}
const std::map<uint64_t, Locator>& GetAllLocators() const {
return _locators;
}
};
保证机制详解：
_src 和 sourceIdx 区分数据源
_srcsourceIdx 标识数据来源_src 选择对应的 Locator多数据源一致性：通过 sourceIdx 区分数据源，保证多数据源场景的数据一致性：
flowchart TB
subgraph Identification["数据源标识 Source Identification"]
direction LR
A["数据源标识<br/>_src: uint64_t<br/>每个数据源唯一标识<br/>区分不同数据源"]
B["文档来源<br/>sourceIdx: uint8_t<br/>DocInfo中的字段<br/>标识数据来源"]
end
subgraph Management["多数据源管理 Multi-Source Management"]
direction LR
C["独立Locator<br/>Independent Locator<br/>每个数据源一个Locator<br/>Version中维护map"]
D["独立处理<br/>Independent Processing<br/>根据_src选择Locator<br/>并行处理不同数据源"]
E["统一管理<br/>Unified Management<br/>Version统一管理<br/>所有Locator持久化"]
end
subgraph Guarantee["一致性保证 Consistency Guarantee"]
direction LR
F["隔离机制<br/>Isolation Mechanism<br/>不同数据源互不干扰<br/>保证数据一致性"]
G["并行支持<br/>Parallel Support<br/>支持并行处理<br/>提高处理效率"]
end
Identification -->|支持| Management
Management -->|实现| Guarantee
A -->|用于| C
B -->|用于| C
C -->|支持| D
D -->|通过| E
E -->|实现| F
F -->|支持| G
style Identification fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Management fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Guarantee fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style G fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
Locator 支持分片处理：
分片处理支持：通过 hashId 支持分片处理：
flowchart TB
subgraph Sharding["分片处理机制 Sharding Mechanism"]
direction LR
A["HashId分片<br/>HashId Sharding<br/>通过hashId分片<br/>Progress的from/to定义范围"]
B["独立进度<br/>Independent Progress<br/>每个hashId范围独立进度<br/>MultiProgress追踪"]
C["并行处理<br/>Parallel Processing<br/>不同hashId并行处理<br/>提高处理效率"]
end
subgraph Management["分片管理 Shard Management"]
direction LR
D["范围定义<br/>Range Definition<br/>from/to定义hashId范围<br/>支持灵活分片"]
E["进度追踪<br/>Progress Tracking<br/>MultiProgress追踪每个hashId<br/>支持分片恢复"]
F["负载均衡<br/>Load Balancing<br/>根据hashId分布<br/>均衡处理负载"]
end
subgraph Benefit["分片优势 Sharding Benefits"]
direction LR
G["并行能力<br/>Parallel Capability<br/>支持并行处理<br/>提高吞吐量"]
H["灵活扩展<br/>Flexible Scaling<br/>支持动态分片<br/>适应不同场景"]
end
Sharding -->|包含| Management
Management -->|带来| Benefit
A -->|实现| D
B -->|实现| E
C -->|实现| F
style Sharding fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Management fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Benefit fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style H fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
分片机制：
Locator 支持并发控制：
并发控制：通过 concurrentIdx 处理时间戳相同的情况：
flowchart TB
Start([并发数据处理<br/>Concurrent Data Processing]) --> GetOffset[获取Offset<br/>Get Offset<br/>timestamp + concurrentIdx<br/>Progress::Offset]
subgraph Positioning["两级定位机制 Two-Level Positioning"]
direction TB
P1[第一级：时间戳定位<br/>Timestamp Positioning<br/>timestamp: int64_t 8字节<br/>记录数据时间位置]
P2[第二级：并发索引定位<br/>Concurrent Index Positioning<br/>concurrentIdx: uint32_t 4字节<br/>处理时间戳相同的情况]
P3[组合定位<br/>Combined Positioning<br/>std::pair timestamp, concurrentIdx<br/>保证唯一性和顺序性]
end
GetOffset --> Compare[比较Offset<br/>Compare Offset<br/>比较timestamp和concurrentIdx<br/>IsFasterThan方法]
subgraph Conflict["冲突解决机制 Conflict Resolution"]
direction TB
C1{时间戳<br/>是否相同}
C2[使用concurrentIdx区分<br/>Use concurrentIdx to Distinguish<br/>concurrentIdx小的优先<br/>保证顺序性]
C3[时间戳不同<br/>Timestamp Different<br/>时间戳大的优先<br/>直接比较]
end
Compare --> C1
C1 -->|相同| C2
C1 -->|不同| C3
subgraph Safety["并发安全保证 Thread Safety"]
direction LR
S1[原子操作<br/>Atomic Operations<br/>比较操作原子性<br/>保证一致性]
S2[无锁设计<br/>Lock-Free Design<br/>减少锁竞争<br/>提高并发性能]
S3[读写分离<br/>Read-Write Separation<br/>读操作无锁<br/>写操作同步]
end
C2 --> Update[更新Locator<br/>Update Locator<br/>原子性更新<br/>保证线程安全]
C3 --> Update
Update --> Order[顺序保证<br/>Order Guarantee<br/>保证数据处理顺序<br/>避免乱序问题]
Order --> End([结束<br/>End])
subgraph Benefit["并发优势 Concurrency Benefits"]
direction LR
B1[高并发支持<br/>High Concurrency<br/>支持高并发场景<br/>提高处理能力]
B2[顺序性保证<br/>Order Guarantee<br/>保证数据顺序<br/>避免乱序问题]
B3[性能优化<br/>Performance Optimization<br/>减少锁竞争<br/>提高处理效率]
end
End -->|实现| B1
Order -->|实现| B2
Safety -->|实现| B3
GetOffset -.->|包含| Positioning
Update -.->|使用| Safety
P1 --> P2
P2 --> P3
S1 --> S2
S2 --> S3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style GetOffset fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Update fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Order fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Positioning fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Conflict fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Safety fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style S1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Benefit fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
并发机制：
Locator 支持用户数据：
用户数据支持：通过 _userData 存储自定义信息：
flowchart TB
Start([设置用户数据<br/>Set UserData]) --> SetData[设置数据<br/>SetUserData方法<br/>_userData = data<br/>std::string类型]
subgraph Storage["数据存储 Data Storage"]
direction TB
S1[数据字段<br/>_userData: std::string<br/>存储自定义信息<br/>支持任意字符串数据]
S2[可选字段<br/>Optional Field<br/>可以为空<br/>灵活使用]
S3[内存存储<br/>Memory Storage<br/>存储在Locator对象中<br/>随Locator生命周期]
end
SetData --> Serialize[序列化UserData<br/>Serialize UserData<br/>序列化到Locator<br/>持久化存储]
subgraph Serialization["序列化支持 Serialization Support"]
direction TB
Ser1[写入数据长度<br/>Write Data Length<br/>uint32_t 4字节<br/>数据大小]
Ser2[写入数据内容<br/>Write Data Content<br/>writeBytes方法<br/>实际数据]
Ser3[持久化存储<br/>Persistent Storage<br/>序列化到Version文件<br/>支持故障恢复]
end
Serialize --> Query[查询UserData<br/>Query UserData<br/>GetUserData方法<br/>获取用户数据]
subgraph Application["应用场景 Application Scenarios"]
direction LR
A1[业务扩展<br/>Business Extension<br/>存储业务自定义信息<br/>支持业务需求]
A2[元数据存储<br/>Metadata Storage<br/>存储处理元数据<br/>支持追踪和调试]
A3[配置信息<br/>Configuration Info<br/>存储配置参数<br/>支持动态配置]
end
Query --> Use[使用UserData<br/>Use UserData<br/>业务逻辑处理<br/>信息追踪]
subgraph Benefit["扩展优势 Extension Benefits"]
direction LR
B1[业务定制<br/>Business Customization<br/>支持业务定制需求<br/>提高灵活性]
B2[信息追踪<br/>Information Tracking<br/>存储处理信息<br/>支持问题排查]
B3[灵活扩展<br/>Flexible Extension<br/>支持任意数据格式<br/>适应不同场景]
end
Use --> End([结束<br/>End])
SetData -.->|包含| Storage
Serialize -.->|包含| Serialization
Use -.->|用于| Application
End -->|实现| Benefit
S1 --> S2
S2 --> S3
Ser1 --> Ser2
Ser2 --> Ser3
A1 --> A2
A2 --> A3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style SetData fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Serialize fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Query fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Use fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Storage fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Serialization fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Ser1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Ser2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Ser3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Application fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style A2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style A3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Benefit fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
用户数据机制：
_userData 存储自定义信息GetUserData() 获取用户数据在实时写入场景中，Locator 的应用：
实时写入场景中的 Locator：通过 Locator 判断数据是否已处理：
flowchart TB
Start([实时数据到达<br/>Real-Time Data Arrival]) --> Receive[接收数据<br/>Receive Data<br/>TabletWriter::Build<br/>获取文档和Locator]
Receive --> GetLocator[获取文档Locator<br/>Get Document Locator<br/>doc.GetLocator<br/>提取文档位置信息]
GetLocator --> Compare[比较Locator<br/>Compare Locator<br/>IsFasterThan方法<br/>docLocator vs currentLocator]
Compare --> CheckResult{比较结果判断<br/>LocatorCompareResult}
CheckResult -->|LCR_FULLY_FASTER<br/>数据已完全处理| Skip[跳过处理<br/>Skip Processing<br/>返回Status::OK<br/>避免重复处理]
CheckResult -->|LCR_SLOWER<br/>数据未处理| Process[处理新数据<br/>Process New Data<br/>MemSegment::Build<br/>构建索引]
CheckResult -->|LCR_PARTIAL_FASTER<br/>部分已处理| ProcessPartial[部分处理<br/>Partial Processing<br/>处理未处理部分<br/>部分构建索引]
CheckResult -->|LCR_INVALID<br/>数据源不同| HandleMulti[多数据源处理<br/>Multi-Source Processing<br/>根据数据源选择Locator<br/>独立处理]
Process --> UpdateLocator[更新Locator<br/>Update Locator<br/>Locator::Update方法<br/>合并MultiProgress]
ProcessPartial --> UpdateLocator
HandleMulti --> UpdateLocator
subgraph UpdateDetail["更新详细步骤 Update Details"]
direction TB
U1[合并MultiProgress<br/>Merge MultiProgress<br/>保留更大进度<br/>更新每个hashId]
U2[更新MinOffset<br/>Update MinOffset<br/>重新计算最小偏移量<br/>快速判断整体进度]
U3[更新UserData<br/>Update UserData<br/>如果新Locator有UserData<br/>则更新]
end
UpdateLocator --> Commit[提交版本<br/>Commit Version<br/>VersionCommitter::Commit<br/>定期提交]
subgraph CommitDetail["提交详细步骤 Commit Details"]
direction TB
C1[创建新版本<br/>Create New Version<br/>包含所有Segment<br/>设置版本信息]
C2[设置Locator<br/>Set Locator<br/>Version::SetLocator<br/>保存当前Locator]
C3[持久化版本<br/>Persist Version<br/>WriteVersion方法<br/>序列化Locator]
end
Commit --> End([结束<br/>End])
Skip --> End
UpdateLocator -.->|包含| UpdateDetail
Commit -.->|包含| CommitDetail
U1 --> U2
U2 --> U3
C1 --> C2
C2 --> C3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Receive fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Skip fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style Process fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ProcessPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style HandleMulti fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Commit fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateDetail fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style U1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style U2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style U3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CommitDetail fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
应用流程：
IsFasterThan() 判断数据是否已处理在批量更新场景中，Locator 的应用：
批量更新场景中的 Locator：批量处理数据，避免重复处理：
flowchart TB
Start([批量更新开始<br/>Batch Update Start]) --> ReadBatch[批量读取数据<br/>Batch Read Data<br/>从数据源批量读取<br/>获取文档列表]
ReadBatch --> Iterate[遍历文档<br/>Iterate Documents<br/>for each document<br/>逐个检查]
Iterate --> GetDocLocator[获取文档Locator<br/>Get Document Locator<br/>doc.GetLocator<br/>提取文档位置信息]
GetDocLocator --> Compare[比较Locator<br/>Compare Locator<br/>IsFasterThan方法<br/>判断是否已处理]
Compare --> CheckResult{比较结果判断<br/>LocatorCompareResult}
CheckResult -->|LCR_FULLY_FASTER<br/>数据已处理| Filter[过滤已处理数据<br/>Filter Processed Data<br/>跳过该文档<br/>不加入处理列表]
CheckResult -->|LCR_SLOWER<br/>数据未处理| AddToProcess[加入处理列表<br/>Add to Process List<br/>保留未处理数据<br/>等待批量处理]
CheckResult -->|LCR_PARTIAL_FASTER<br/>部分已处理| AddPartial[加入部分处理列表<br/>Add to Partial List<br/>保留未处理部分<br/>等待部分处理]
Filter --> CheckMore{还有更多<br/>文档?}
AddToProcess --> CheckMore
AddPartial --> CheckMore
CheckMore -->|是| Iterate
CheckMore -->|否| BatchProcess[批量处理<br/>Batch Processing<br/>批量构建索引<br/>MemSegment::Build]
subgraph ProcessDetail["批量处理详细步骤"]
direction TB
P1[批量构建索引<br/>Batch Build Index<br/>并行处理多个文档<br/>提高处理效率]
P2[批量更新Locator<br/>Batch Update Locator<br/>合并所有文档的Locator<br/>更新MultiProgress]
P3[更新MinOffset<br/>Update MinOffset<br/>重新计算最小偏移量<br/>快速判断整体进度]
end
BatchProcess --> Commit[提交版本<br/>Commit Version<br/>VersionCommitter::Commit<br/>批量处理完成后提交]
subgraph CommitDetail["提交详细步骤"]
direction TB
C1[创建新版本<br/>Create New Version<br/>包含所有Segment<br/>设置版本信息]
C2[设置Locator<br/>Set Locator<br/>Version::SetLocator<br/>保存当前Locator]
C3[持久化版本<br/>Persist Version<br/>WriteVersion方法<br/>序列化Locator]
end
Commit --> End([结束<br/>End])
subgraph Benefit["批量优势 Batch Benefits"]
direction LR
B1[效率优化<br/>Efficiency Optimization<br/>批量处理提高效率<br/>减少单次开销]
B2[一致性保证<br/>Consistency Guarantee<br/>保证数据一致性<br/>避免重复处理]
B3[资源优化<br/>Resource Optimization<br/>批量操作减少IO<br/>提高吞吐量]
end
End -->|实现| B1
BatchProcess -->|实现| B2
Commit -->|实现| B3
BatchProcess -.->|包含| ProcessDetail
Commit -.->|包含| CommitDetail
P1 --> P2
P2 --> P3
C1 --> C2
C2 --> C3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style ReadBatch fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Iterate fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetDocLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckMore fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Filter fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style AddToProcess fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style AddPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style BatchProcess fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Commit fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ProcessDetail fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CommitDetail fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Benefit fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
应用流程：
IsFasterThan() 判断哪些数据已处理在故障恢复场景中，Locator 的应用：
故障恢复场景中的 Locator：通过 Locator 判断需要重新处理的数据：
flowchart TB
Start([系统故障<br/>System Fault]) --> Detect[故障检测<br/>Fault Detection<br/>检测到系统故障<br/>需要恢复]
Detect --> LoadVersion[加载版本<br/>Load Version<br/>加载故障前版本<br/>Version::Load]
subgraph LoadDetail["加载详细步骤"]
direction TB
L1[读取版本文件<br/>Read Version File<br/>从磁盘读取版本信息<br/>获取版本号]
L2[反序列化Locator<br/>Deserialize Locator<br/>恢复Locator对象<br/>获取处理位置]
L3[设置当前Locator<br/>Set Current Locator<br/>TabletWriter::SetLocator<br/>恢复处理位置]
end
LoadVersion --> ReadData[读取数据源<br/>Read Data Source<br/>从数据源读取数据<br/>获取文档列表]
ReadData --> Iterate[遍历文档<br/>Iterate Documents<br/>for each document<br/>逐个检查]
Iterate --> GetDocLocator[获取文档Locator<br/>Get Document Locator<br/>doc.GetLocator<br/>提取文档位置信息]
GetDocLocator --> Compare[比较Locator<br/>Compare Locator<br/>IsFasterThan方法<br/>判断是否已处理]
Compare --> CheckResult{比较结果判断<br/>LocatorCompareResult}
CheckResult -->|LCR_FULLY_FASTER<br/>数据已处理| Skip[跳过处理<br/>Skip Processing<br/>数据已处理<br/>避免重复处理]
CheckResult -->|LCR_SLOWER<br/>数据未处理| Reprocess[重新处理<br/>Reprocess Data<br/>MemSegment::Build<br/>构建索引]
CheckResult -->|LCR_PARTIAL_FASTER<br/>部分已处理| ReprocessPartial[部分重新处理<br/>Partial Reprocess<br/>处理未处理部分<br/>部分构建索引]
Skip --> CheckMore{还有更多<br/>文档?}
Reprocess --> UpdateLocator[更新Locator<br/>Update Locator<br/>Locator::Update方法<br/>合并MultiProgress]
ReprocessPartial --> UpdateLocator
CheckMore -->|是| Iterate
CheckMore -->|否| UpdateLocator
subgraph UpdateDetail["更新详细步骤"]
direction TB
U1[合并MultiProgress<br/>Merge MultiProgress<br/>保留更大进度<br/>更新每个hashId]
U2[更新MinOffset<br/>Update MinOffset<br/>重新计算最小偏移量<br/>快速判断整体进度]
U3[更新UserData<br/>Update UserData<br/>如果新Locator有UserData<br/>则更新]
end
UpdateLocator --> Commit[提交版本<br/>Commit Version<br/>VersionCommitter::Commit<br/>恢复完成后提交]
subgraph CommitDetail["提交详细步骤"]
direction TB
C1[创建新版本<br/>Create New Version<br/>包含所有Segment<br/>设置版本信息]
C2[设置Locator<br/>Set Locator<br/>Version::SetLocator<br/>保存当前Locator]
C3[持久化版本<br/>Persist Version<br/>WriteVersion方法<br/>序列化Locator]
end
Commit --> Verify[验证数据完整性<br/>Verify Data Integrity<br/>检查数据一致性<br/>保证数据不丢失]
Verify --> End([恢复完成<br/>Recovery Complete])
LoadVersion -.->|包含| LoadDetail
UpdateLocator -.->|包含| UpdateDetail
Commit -.->|包含| CommitDetail
L1 --> L2
L2 --> L3
U1 --> U2
U2 --> U3
C1 --> C2
C2 --> C3
style Start fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Detect fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style LoadVersion fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadData fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Iterate fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetDocLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckResult fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckMore fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Skip fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style Reprocess fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style ReprocessPartial fill:#ffe0b2,stroke:#f57c00,stroke-width:2px
style UpdateLocator fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Commit fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Verify fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style LoadDetail fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style L1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style UpdateDetail fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style U1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style U2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style U3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CommitDetail fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
应用流程：
IsFasterThan() 判断哪些数据已处理Locator 的性能直接影响增量更新的效率，需要从比较、更新、序列化等多个方面进行优化。让我们先通过流程图来理解性能优化的整体策略：
flowchart TB
Start([性能优化策略<br/>Performance Optimization Strategy]) --> CompareOpt[比较优化<br/>Comparison Optimization]
Start --> UpdateOpt[更新优化<br/>Update Optimization]
Start --> SerializeOpt[序列化优化<br/>Serialization Optimization]
Start --> MemoryOpt[内存优化<br/>Memory Optimization]
subgraph Compare["比较优化策略"]
direction TB
C1[快速路径优化<br/>Fast Path Optimization<br/>数据源不同直接返回<br/>空Progress快速判断]
C2[结果缓存<br/>Result Cache<br/>LRU缓存策略<br/>避免重复计算]
C3[并行比较<br/>Parallel Comparison<br/>支持并行比较<br/>提高并发性能]
C4[短路优化<br/>Short Circuit Optimization<br/>发现更慢立即返回<br/>减少比较次数]
end
subgraph Update["更新优化策略"]
direction TB
U1[原子更新<br/>Atomic Update<br/>原子性更新操作<br/>保证一致性]
U2[进度合并<br/>Progress Merge<br/>合并MultiProgress<br/>保留更大进度]
U3[批量更新<br/>Batch Update<br/>批量更新Locator<br/>提高更新效率]
end
subgraph Serialize["序列化优化策略"]
direction TB
S1[紧凑格式<br/>Compact Format<br/>VarInt编码<br/>减少存储空间]
S2[压缩支持<br/>Compression Support<br/>LZ4或Snappy算法<br/>大于1KB时压缩]
S3[批量序列化<br/>Batch Serialization<br/>批量序列化多个Locator<br/>对象池复用缓冲区]
end
subgraph Memory["内存优化策略"]
direction TB
M1[对象池<br/>Object Pool<br/>复用Locator对象<br/>减少内存分配]
M2[对象复用<br/>Object Reuse<br/>重置状态复用<br/>减少构造析构开销]
M3[内存预分配<br/>Memory Pre-allocation<br/>预分配MultiProgress容量<br/>减少动态分配]
end
CompareOpt -->|包含| Compare
UpdateOpt -->|包含| Update
SerializeOpt -->|包含| Serialize
MemoryOpt -->|包含| Memory
Compare --> End([优化完成<br/>Optimization Complete])
Update --> End
Serialize --> End
Memory --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CompareOpt fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateOpt fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SerializeOpt fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style MemoryOpt fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Update fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style U1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style U3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Serialize fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Memory fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style M1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style M2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style M3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
Locator 比较是增量更新的核心操作，需要优化比较算法，提高比较效率。让我们通过流程图来理解比较优化的策略：
flowchart TB
Start([开始比较<br/>IsFasterThan方法]) --> CheckCache{检查缓存<br/>Check Cache<br/>LocatorCompareCache::Get}
CheckCache -->|命中| ReturnCache[返回缓存结果<br/>Return Cached Result<br/>直接返回结果<br/>避免重复计算]
CheckCache -->|未命中| FastPathNode[快速路径优化<br/>Fast Path Optimization]
subgraph FastPathGroup["快速路径优化 Fast Path"]
direction TB
F1[检查数据源<br/>Check Data Source<br/>IsSameSrc方法<br/>比较_src字段]
F2{数据源<br/>是否相同}
F3[返回LCR_INVALID<br/>Return LCR_INVALID<br/>数据源不同<br/>直接返回]
F4[检查是否为空<br/>Check Empty<br/>MultiProgress是否为空]
F5{是否都为空}
F6[返回LCR_FULLY_FASTER<br/>Return LCR_FULLY_FASTER<br/>都为空时<br/>直接返回]
F7[比较大小<br/>Compare Size<br/>比较MultiProgress大小]
F8{大小关系}
F9[返回LCR_PARTIAL_FASTER<br/>Return LCR_PARTIAL_FASTER<br/>当前size大于other.size<br/>覆盖更多hashId]
end
FastPathNode --> CompareEach[逐个比较<br/>Compare Each<br/>遍历每个hashId<br/>调用CompareProgress]
subgraph CompareDetail["逐个比较详细步骤"]
direction TB
D1[遍历hashId<br/>Iterate hashId<br/>遍历minSize个hashId<br/>比较multiProgress]
D2[调用CompareProgress<br/>Call CompareProgress<br/>比较该hashId的进度<br/>返回比较结果]
D3[短路优化检查<br/>Short Circuit Check<br/>检查是否有更慢的<br/>且无部分更快]
D4{短路条件<br/>满足?}
D5[立即返回LCR_SLOWER<br/>Return LCR_SLOWER Immediately<br/>减少比较次数<br/>提高效率]
end
CompareEach --> UpdateCache[更新缓存<br/>Update Cache<br/>LocatorCompareCache::Put<br/>保存比较结果]
UpdateCache --> End([结束<br/>End])
ReturnCache --> End
FastPathNode -.->|包含| FastPathGroup
F1 --> F2
F2 -->|不同| F3
F2 -->|相同| F4
F4 --> F5
F5 -->|都为空| F6
F5 -->|不同| F7
F7 --> F8
F8 -->|当前size大于| F9
F8 -->|相等| CompareEach
F3 --> End
F6 --> End
F9 --> End
CompareEach -.->|包含| CompareDetail
D1 --> D2
D2 --> D3
D3 --> D4
D4 -->|满足| D5
D4 -->|不满足| D1
D5 --> UpdateCache
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CheckCache fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ReturnCache fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style FastPathNode fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CompareEach fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UpdateCache fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style FastPathGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style F4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F5 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F6 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style F7 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F8 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F9 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style CompareDetail fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style D1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style D4 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style D5 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
比较性能优化的实现：
// framework/Locator.cpp
class LocatorCompareCache
{
private:
struct CacheKey {
uint64_t src1, src2;
size_t hash1, hash2;
bool operator==(const CacheKey& other) const {
return src1 == other.src1 && src2 == other.src2 &&
hash1 == other.hash1 && hash2 == other.hash2;
}
};
struct CacheValue {
LocatorCompareResult result;
std::chrono::steady_clock::time_point timestamp;
};
std::unordered_map<CacheKey, CacheValue> _cache;
static constexpr size_t MAX_CACHE_SIZE = 1000;
static constexpr auto CACHE_TTL = std::chrono::minutes(5);
public:
std::optional<LocatorCompareResult> Get(const Locator& l1, const Locator& l2) {
CacheKey key = MakeKey(l1, l2);
auto it = _cache.find(key);
if (it != _cache.end()) {
auto now = std::chrono::steady_clock::now();
if (now - it->second.timestamp < CACHE_TTL) {
return it->second.result;
}
_cache.erase(it);
}
return std::nullopt;
}
void Put(const Locator& l1, const Locator& l2, LocatorCompareResult result) {
if (_cache.size() >= MAX_CACHE_SIZE) {
// 清理过期项
CleanExpired();
}
CacheKey key = MakeKey(l1, l2);
_cache[key] = {result, std::chrono::steady_clock::now()};
}
};
优化策略详解：
LCR_INVALID，避免遍历 ProgressLCR_SLOWERLocator 比较的性能优化：优化比较算法，提高比较效率：
flowchart TB
Start([比较性能优化<br/>Comparison Performance Optimization]) --> StrategyLayer[优化策略层<br/>Optimization Strategy Layer]
subgraph StrategyGroup["优化策略 Optimization Strategies"]
direction TB
S1[快速路径优化<br/>Fast Path Optimization<br/>数据源不同直接返回<br/>空Progress快速判断<br/>大小不同快速判断]
S2[短路优化<br/>Short Circuit Optimization<br/>发现更慢立即返回<br/>减少比较次数<br/>提高比较效率]
S3[缓存优化<br/>Cache Optimization<br/>比较结果缓存<br/>LRU缓存策略<br/>避免重复计算]
end
StrategyLayer --> TechniqueLayer[优化技术层<br/>Optimization Technique Layer]
subgraph TechniqueGroup["优化技术 Optimization Techniques"]
direction TB
T1[最小偏移量优化<br/>MinOffset Optimization<br/>使用MinOffset快速判断<br/>减少遍历次数<br/>快速判断整体进度]
T2[位运算优化<br/>Bitwise Optimization<br/>使用位运算优化比较<br/>减少比较开销<br/>提高比较速度]
T3[并行比较<br/>Parallel Comparison<br/>支持并行比较<br/>提高并发性能<br/>充分利用多核]
end
TechniqueLayer --> BenefitLayer[优化效果层<br/>Optimization Benefit Layer]
subgraph BenefitGroup["优化效果 Optimization Benefits"]
direction TB
B1[性能提升<br/>Performance Improvement<br/>减少比较时间<br/>提高处理效率<br/>降低延迟]
B2[资源优化<br/>Resource Optimization<br/>减少CPU使用<br/>降低系统负载<br/>提高吞吐量]
end
BenefitLayer --> End([优化完成<br/>Optimization Complete])
StrategyLayer -.->|包含| StrategyGroup
TechniqueLayer -.->|包含| TechniqueGroup
BenefitLayer -.->|包含| BenefitGroup
S1 -->|应用| T1
S2 -->|应用| T2
S3 -->|应用| T3
T1 -->|实现| B1
T2 -->|实现| B1
T3 -->|实现| B2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style TechniqueLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style BenefitLayer fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style TechniqueGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style T1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style T2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style T3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style BenefitGroup fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
Locator 序列化的性能优化，包括格式优化、压缩支持、批量序列化等。让我们通过流程图来理解序列化优化的策略：
flowchart TB
Start([开始序列化<br/>Serialize方法]) --> Estimate[估算序列化大小<br/>EstimateSize方法<br/>计算预估大小]
Estimate --> CheckSize{检查大小<br/>是否大于1KB}
CheckSize -->|小于1KB| SerializeDirect[直接序列化<br/>SerializeDirect方法<br/>小数据直接序列化]
CheckSize -->|大于1KB| SerializeCompressed[压缩序列化<br/>SerializeCompressed方法<br/>大数据压缩后序列化]
subgraph Direct["直接序列化流程"]
direction TB
D1[写入头部信息<br/>Magic Number + Version<br/>8字节]
D2[写入基础字段<br/>Src + MinOffset<br/>20字节]
D3[写入MultiProgress<br/>嵌套结构<br/>可变长度]
D4[写入UserData<br/>可选字段<br/>可变长度]
D5[写入Legacy标志<br/>1字节]
D6[转换为字符串<br/>buffer.toString]
end
subgraph Compressed["压缩序列化流程"]
direction TB
C1[先序列化<br/>SerializeDirect<br/>获取原始数据]
C2[压缩数据<br/>Compress方法<br/>LZ4或Snappy算法]
C3[添加压缩标志<br/>写入压缩标志<br/>uint8_t 1字节]
C4[写入压缩数据大小<br/>uint32_t 4字节]
C5[写入压缩数据内容<br/>writeBytes]
C6[转换为字符串<br/>buffer.toString]
end
SerializeDirect --> CompactNode[紧凑格式优化<br/>Compact Format Optimization]
subgraph CompactGroup["紧凑格式优化"]
direction TB
CF1[VarInt编码<br/>Variable Integer Encoding<br/>变长编码整数<br/>减少存储空间]
CF2[合并相邻Progress<br/>Merge Adjacent Progress<br/>减少存储空间<br/>优化MultiProgress]
CF3[位图压缩<br/>Bitmap Compression<br/>压缩MultiProgress<br/>减少内存占用]
end
CompactNode --> End([结束<br/>返回序列化结果])
SerializeCompressed --> End
SerializeDirect -.->|包含| Direct
SerializeCompressed -.->|包含| Compressed
CompactNode -.->|包含| CompactGroup
D1 --> D2
D2 --> D3
D3 --> D4
D4 --> D5
D5 --> D6
D6 --> CompactNode
C1 --> C2
C2 --> C3
C3 --> C4
C4 --> C5
C5 --> C6
C6 --> End
CF1 --> CF2
CF2 --> CF3
CF3 --> CompactNode
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Estimate fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CheckSize fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style SerializeDirect fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SerializeCompressed fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CompactNode fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Direct fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style D1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D6 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Compressed fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C5 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C6 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CompactGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style CF1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CF2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CF3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
序列化性能优化的实现：
// framework/Locator.cpp
std::string Locator::Serialize() const
{
// 1. 估算序列化大小
size_t estimatedSize = EstimateSize();
// 2. 选择序列化策略
if (estimatedSize < 1024) {
// 小数据，直接序列化
return SerializeDirect();
} else {
// 大数据，压缩后序列化
return SerializeCompressed();
}
}
std::string Locator::SerializeCompressed() const
{
// 1. 先序列化
std::string data = SerializeDirect();
// 2. 压缩
std::string compressed = Compress(data);
// 3. 添加压缩标志
autil::DataBuffer buffer;
buffer.write(static_cast<uint8_t>(1)); // 压缩标志
buffer.write(static_cast<uint32_t>(compressed.size()));
buffer.writeBytes(compressed.data(), compressed.size());
return buffer.toString();
}
优化策略详解：
Locator 序列化的性能优化：优化序列化格式，提高序列化效率：
flowchart TB
Start([序列化性能优化<br/>Serialization Performance Optimization]) --> FormatLayer[格式优化层<br/>Format Optimization Layer]
subgraph FormatGroup["格式优化 Format Optimization"]
direction TB
F1[紧凑格式<br/>Compact Format<br/>使用VarInt编码<br/>合并相邻Progress<br/>位图压缩MultiProgress]
F2[版本兼容<br/>Version Compatibility<br/>支持多版本格式<br/>向后兼容<br/>平滑升级]
F3[批量序列化<br/>Batch Serialization<br/>批量序列化多个Locator<br/>对象池复用缓冲区<br/>减少开销]
end
FormatLayer --> CompressionLayer[压缩优化层<br/>Compression Optimization Layer]
subgraph CompressionGroup["压缩优化 Compression Optimization"]
direction TB
C1[智能压缩<br/>Smart Compression<br/>大于1KB时压缩<br/>LZ4或Snappy算法<br/>压缩标志存储]
C2[压缩策略<br/>Compression Strategy<br/>估算序列化大小<br/>选择压缩策略<br/>平衡性能和空间]
end
CompressionLayer --> BenefitLayer[优化效果层<br/>Optimization Benefit Layer]
subgraph BenefitGroup["优化效果 Optimization Benefits"]
direction TB
B1[空间优化<br/>Space Optimization<br/>减少存储空间<br/>降低网络传输<br/>节省带宽]
B2[性能提升<br/>Performance Improvement<br/>提高序列化效率<br/>减少序列化时间<br/>降低延迟]
end
BenefitLayer --> End([优化完成<br/>Optimization Complete])
FormatLayer -.->|包含| FormatGroup
CompressionLayer -.->|包含| CompressionGroup
BenefitLayer -.->|包含| BenefitGroup
F1 -->|支持| F2
F2 -->|支持| F3
F3 -->|结合| C1
C1 -->|使用| C2
C2 -->|实现| B1
FormatGroup -->|实现| B2
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style FormatLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CompressionLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style BenefitLayer fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style FormatGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style F1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style F3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CompressionGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style BenefitGroup fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
Locator 的内存优化，包括对象池、对象复用等。让我们通过类图来理解内存优化的架构：
classDiagram
class LocatorPool {
- queue~Locator*~ _pool
- mutex _mutex
- size_t MAX_POOL_SIZE = 100
+ Locator* Get()
+ void Put(Locator*)
+ void Clear()
+ size_t Size()
}
class Locator {
- uint64_t _src
- MultiProgress _multiProgress
- string _userData
- Progress::Offset _minOffset
+ void Reset()
+ void Reuse()
+ bool IsValid()
+ LocatorCompareResult IsFasterThan()
+ void Update()
}
class ProgressPool {
- queue~ProgressVector*~ _pool
- size_t MAX_POOL_SIZE = 200
+ ProgressVector* Get()
+ void Put(ProgressVector*)
+ void Clear()
}
class ProgressVector {
- vector~Progress~ _progresses
+ void Reserve(size_t)
+ void Clear()
+ size_t Size()
}
class MemoryOptimization {
<<interface>>
+ 对象池管理
+ 对象复用
+ 内存预分配
}
LocatorPool --> Locator : 管理对象池<br/>复用Locator对象<br/>减少内存分配
ProgressPool --> ProgressVector : 管理对象池<br/>复用ProgressVector<br/>减少内存分配
Locator --> ProgressVector : 包含<br/>MultiProgress包含<br/>ProgressVector数组
LocatorPool ..> MemoryOptimization : 实现
ProgressPool ..> MemoryOptimization : 实现
Locator ..> MemoryOptimization : 支持
内存优化的实现：
// framework/LocatorPool.h
class LocatorPool
{
private:
std::queue<Locator*> _pool;
std::mutex _mutex;
static constexpr size_t MAX_POOL_SIZE = 100;
public:
Locator* Get() {
std::lock_guard<std::mutex> lock(_mutex);
if (!_pool.empty()) {
Locator* locator = _pool.front();
_pool.pop();
locator->Reset(); // 重置状态
return locator;
}
return new Locator();
}
void Put(Locator* locator) {
if (!locator) return;
std::lock_guard<std::mutex> lock(_mutex);
if (_pool.size() < MAX_POOL_SIZE) {
locator->Reset();
_pool.push(locator);
} else {
delete locator;
}
}
};
内存优化策略：
Locator 的设计遵循简单、高效、可靠、可扩展的原则，是 IndexLib 数据一致性保证的基础。让我们先通过类图来理解 Locator 的整体设计：
classDiagram
class DesignPrinciples {
<<设计原则>>
+简单性 Simplicity
+高效性 Efficiency
+可靠性 Reliability
+扩展性 Extensibility
}
class Locator {
- uint64_t _src
- MultiProgress _multiProgress
- Progress::Offset _minOffset
- string _userData
- bool _isLegacyLocator
+ LocatorCompareResult IsFasterThan()
+ void Update()
+ string Serialize()
+ Status Deserialize()
+ bool IsValid()
+ bool IsSameSrc()
+ void Reset()
}
class Compatibility {
<<兼容性支持>>
+遗留Locator支持 Legacy Support
+版本兼容 Version Compatibility
+平滑升级 Smooth Upgrade
+向后兼容 Backward Compatibility
+多版本格式支持
}
class ThreadSafety {
<<线程安全>>
+原子操作 Atomic Operations
+无锁设计 Lock-Free Design
+读写分离 Read-Write Separation
+并发控制 Concurrency Control
+线程安全保证
}
class Performance {
<<性能优化>>
+快速路径优化 Fast Path
+缓存优化 Cache Optimization
+短路优化 Short Circuit
+批量操作 Batch Operations
}
class DataConsistency {
<<数据一致性>>
+只向前推进 Forward Only
+原子性更新 Atomic Update
+数据不重复 No Duplication
+数据不丢失 No Loss
}
DesignPrinciples --> Locator : 指导设计<br/>Guides Design
Locator --> Compatibility : 支持<br/>Supports
Locator --> ThreadSafety : 保证<br/>Guarantees
Locator --> Performance : 优化<br/>Optimizes
Locator --> DataConsistency : 实现<br/>Implements
DesignPrinciples --> Compatibility : 要求<br/>Requires
DesignPrinciples --> ThreadSafety : 要求<br/>Requires
DesignPrinciples --> Performance : 要求<br/>Requires
DesignPrinciples --> DataConsistency : 要求<br/>Requires
Locator 的设计遵循以下核心原则，确保简单、高效、可靠、可扩展：
Locator 的设计原则：简单、高效、可靠的设计原则：
flowchart TB
Start([Locator设计原则<br/>Locator Design Principles]) --> PrinciplesLayer[核心设计原则层<br/>Core Design Principles Layer]
subgraph PrinciplesGroup["核心设计原则 Core Design Principles"]
direction TB
P1[简单性<br/>Simplicity<br/>清晰的接口设计<br/>直观的语义<br/>最小化依赖]
P2[高效性<br/>Efficiency<br/>快速路径优化<br/>短路优化<br/>缓存优化]
P3[可靠性<br/>Reliability<br/>只向前推进<br/>原子性更新<br/>持久化支持]
P4[扩展性<br/>Extensibility<br/>支持自定义扩展<br/>灵活的数据结构<br/>版本兼容]
end
PrinciplesLayer --> SupportLayer[支持特性层<br/>Support Features Layer]
subgraph SupportGroup["支持特性 Support Features"]
direction TB
S1[可扩展性<br/>Extensibility<br/>支持自定义扩展<br/>灵活的数据结构<br/>版本兼容]
S2[易用性<br/>Usability<br/>简单的API接口<br/>清晰的文档<br/>良好的错误处理]
S3[兼容性<br/>Compatibility<br/>遗留Locator支持<br/>版本兼容<br/>平滑升级]
end
SupportLayer --> BenefitLayer[设计优势层<br/>Design Benefits Layer]
subgraph BenefitGroup["设计优势 Design Benefits"]
direction TB
B1[易于维护<br/>Easy Maintenance<br/>代码清晰易懂<br/>易于调试和优化<br/>降低维护成本]
B2[高性能<br/>High Performance<br/>优化的算法实现<br/>高效的资源使用<br/>提高处理效率]
B3[高可靠性<br/>High Reliability<br/>数据一致性保证<br/>故障恢复支持<br/>稳定运行]
end
BenefitLayer --> End([设计完成<br/>Design Complete])
PrinciplesLayer -.->|包含| PrinciplesGroup
SupportLayer -.->|包含| SupportGroup
BenefitLayer -.->|包含| BenefitGroup
P1 -->|实现| S1
P1 -->|实现| S2
P2 -->|实现| S2
P3 -->|保证| S3
P4 -->|支持| S1
S1 -->|带来| B1
S2 -->|带来| B1
S2 -->|带来| B2
S3 -->|带来| B3
P2 -->|直接带来| B2
P3 -->|直接带来| B3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style PrinciplesLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style BenefitLayer fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style PrinciplesGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style BenefitGroup fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
设计原则详解：
IsFasterThan() 和 Update() 接口清晰，易于使用_src 和 sourceIdx 支持多数据源_userData 支持业务扩展Locator 的兼容性设计，支持遗留 Locator 和版本兼容，保证平滑升级。让我们通过流程图来理解兼容性设计的机制：
flowchart TB
Start([开始加载 Locator<br/>Start Loading Locator]) --> ParseLayer[解析处理层<br/>Parse Processing Layer]
subgraph ParseGroup["解析处理 Parse Processing"]
direction TB
P1[读取字符串<br/>Read String<br/>从存储中读取序列化数据]
P2[解压缩<br/>Decompress<br/>如果数据被压缩则解压]
P3[读取 Magic Number<br/>Read Magic Number<br/>验证数据格式]
P4{Magic Number<br/>是否有效<br/>Is Valid}
end
ParseLayer --> VersionLayer[版本处理层<br/>Version Processing Layer]
subgraph VersionGroup["版本处理 Version Processing"]
direction TB
V1[读取版本号<br/>Read Version<br/>从数据中读取版本信息]
V2{版本类型<br/>Version Type}
V3[反序列化 V1<br/>Deserialize V1<br/>读取基本字段<br/>src minOffset]
V4[反序列化 V2<br/>Deserialize V2<br/>读取完整字段<br/>MultiProgress UserData]
V5[未知版本<br/>Unknown Version<br/>不支持该版本]
end
VersionLayer --> LegacyLayer[遗留格式处理层<br/>Legacy Format Processing Layer]
subgraph LegacyGroup["遗留格式处理 Legacy Format Processing"]
direction TB
L1{检查 Legacy 标志<br/>Check Legacy Flag<br/>_isLegacyLocator}
L2[转换为新格式<br/>Convert to New Format<br/>迁移到 MultiProgress<br/>设置默认值]
L3[保持新格式<br/>Keep New Format<br/>已经是新格式]
end
LegacyLayer --> ValidateLayer[验证处理层<br/>Validation Processing Layer]
subgraph ValidateGroup["验证处理 Validation Processing"]
direction TB
Val1[验证数据完整性<br/>Validate Data Integrity<br/>检查必需字段]
Val2[验证数据有效性<br/>Validate Data Validity<br/>检查数据范围]
Val3{验证结果<br/>Validation Result}
Val4[验证成功<br/>Validation Success<br/>数据有效]
Val5[验证失败<br/>Validation Failed<br/>数据无效]
end
ValidateLayer --> EndLayer[完成处理层<br/>Completion Processing Layer]
subgraph EndGroup["完成处理 Completion Processing"]
direction TB
E1[返回 Locator 对象<br/>Return Locator Object<br/>反序列化完成]
E2[返回错误<br/>Return Error<br/>处理失败]
end
EndLayer --> End([结束<br/>End])
ParseLayer -.->|包含| ParseGroup
VersionLayer -.->|包含| VersionGroup
LegacyLayer -.->|包含| LegacyGroup
ValidateLayer -.->|包含| ValidateGroup
EndLayer -.->|包含| EndGroup
P1 --> P2
P2 --> P3
P3 --> P4
P4 -->|有效| V1
P4 -->|无效| V5
V1 --> V2
V2 -->|版本1| V3
V2 -->|版本2| V4
V2 -->|其他| V5
V3 --> L1
V4 --> L1
V5 --> E2
L1 -->|是| L2
L1 -->|否| L3
L2 --> Val1
L3 --> Val1
Val1 --> Val2
Val2 --> Val3
Val3 -->|成功| Val4
Val3 -->|失败| Val5
Val4 --> E1
Val5 --> E2
E1 --> End
E2 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style ParseLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style VersionLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style LegacyLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ValidateLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style EndLayer fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style ParseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style VersionGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style V1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V5 fill:#ef5350,stroke:#c62828,stroke-width:2px
style LegacyGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style L1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style L2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style L3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ValidateGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style Val1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Val2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Val3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Val4 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style Val5 fill:#ef5350,stroke:#c62828,stroke-width:2px
style EndGroup fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style E1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style E2 fill:#ef5350,stroke:#c62828,stroke-width:2px
兼容性机制详解：
_isLegacyLocator 标识
Locator 的兼容性设计：支持遗留 Locator 和版本兼容：
flowchart TB
Start([兼容性设计<br/>Compatibility Design]) --> CoreLayer[核心兼容机制层<br/>Core Compatibility Mechanisms Layer]
subgraph CoreGroup["核心兼容机制 Core Compatibility Mechanisms"]
direction LR
C1[遗留Locator支持<br/>Legacy Locator Support<br/>_isLegacyLocator标识<br/>自动格式转换<br/>透明迁移]
C2[版本兼容<br/>Version Compatibility<br/>版本号识别<br/>V1/V2支持<br/>版本升级]
C3[向后兼容<br/>Backward Compatibility<br/>旧版本读取新数据<br/>字段跳过机制<br/>数据保护]
end
CoreLayer --> SupportLayer[支持功能层<br/>Support Functions Layer]
subgraph SupportGroup["支持功能 Support Functions"]
direction LR
S1[兼容性检查<br/>Compatibility Check<br/>版本验证<br/>格式验证<br/>数据完整性检查]
S2[平滑升级<br/>Smooth Upgrade<br/>渐进式迁移<br/>零停机升级<br/>数据一致性保证]
S3[格式转换<br/>Format Conversion<br/>Legacy到新格式<br/>字段映射<br/>默认值设置]
end
SupportLayer --> FeatureLayer[特性支持层<br/>Feature Support Layer]
subgraph FeatureGroup["特性支持 Feature Support"]
direction LR
F1[多版本共存<br/>Multi-Version Coexistence<br/>同时支持V1和V2<br/>版本识别<br/>自动适配]
F2[数据迁移<br/>Data Migration<br/>批量转换<br/>增量迁移<br/>回滚支持]
F3[错误处理<br/>Error Handling<br/>版本错误处理<br/>格式错误处理<br/>降级策略]
end
FeatureLayer --> End([兼容性保证<br/>Compatibility Guarantee])
CoreLayer -.->|包含| CoreGroup
SupportLayer -.->|包含| SupportGroup
FeatureLayer -.->|包含| FeatureGroup
C1 -->|需要| S1
C1 -->|使用| S3
C2 -->|需要| S1
C2 -->|支持| S2
C3 -->|需要| S1
C3 -->|支持| S2
S1 -->|支持| F1
S2 -->|实现| F2
S3 -->|支持| F2
S1 -->|处理| F3
S2 -->|处理| F3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CoreLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SupportLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style FeatureLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style CoreGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SupportGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FeatureGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style F1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
Locator 的线程安全设计，支持并发访问，保证线程安全。让我们通过序列图来理解线程安全设计的机制：
sequenceDiagram
participant T1 as 读线程1<br/>Read Thread1
participant T2 as 读线程2<br/>Read Thread2
participant T3 as 写线程1<br/>Write Thread1
participant T4 as 写线程2<br/>Write Thread2
participant L as Locator对象<br/>Locator Object
participant RL as 读写锁<br/>ReadWriteLock
participant AM as 原子操作<br/>Atomic Operations
par 并发读操作 Concurrent Read Operations
T1->>L: IsFasterThan(other1)
activate L
L->>RL: 尝试获取读锁<br/>Try Acquire Read Lock
RL-->>L: 获取成功<br/>Acquire Success
Note over L: 无锁设计<br/>Lock-Free Design<br/>只读访问内部状态
L->>AM: 原子读取字段<br/>Atomic Read Fields<br/>_src minOffset
AM-->>L: 返回字段值<br/>Return Field Values
L->>L: 比较逻辑<br/>Comparison Logic<br/>CompareProgress
L-->>T1: 返回比较结果<br/>Return Comparison Result
RL-->>L: 释放读锁<br/>Release Read Lock
deactivate L
T2->>L: IsFasterThan(other2)
activate L
L->>RL: 尝试获取读锁<br/>Try Acquire Read Lock
RL-->>L: 获取成功<br/>Acquire Success
Note over L: 支持并发读取<br/>Support Concurrent Read<br/>多个读线程可同时访问
L->>AM: 原子读取字段<br/>Atomic Read Fields
AM-->>L: 返回字段值<br/>Return Field Values
L->>L: 比较逻辑<br/>Comparison Logic
L-->>T2: 返回比较结果<br/>Return Comparison Result
RL-->>L: 释放读锁<br/>Release Read Lock
deactivate L
end
par 并发写操作 Concurrent Write Operations
T3->>L: Update(newLocator1)
activate L
L->>RL: 尝试获取写锁<br/>Try Acquire Write Lock
Note over RL: 写操作需要互斥<br/>Write Requires Mutual Exclusion<br/>等待读操作完成
RL-->>L: 获取成功<br/>Acquire Success
L->>L: 检查数据源<br/>Check Data Source<br/>_src 匹配检查
L->>L: 调用IsFasterThan<br/>Call IsFasterThan<br/>判断是否需要更新
alt 需要更新 Need Update
L->>AM: 原子更新字段<br/>Atomic Update Fields<br/>minOffset MultiProgress
AM-->>L: 更新成功<br/>Update Success
L->>L: 合并ProgressVector<br/>Merge ProgressVector<br/>MergeProgressVector
L->>L: 更新UserData<br/>Update UserData<br/>_userData 更新
else 不需要更新 No Update
Note over L: 跳过更新<br/>Skip Update<br/>数据已处理
end
RL-->>L: 释放写锁<br/>Release Write Lock
L-->>T3: 更新完成<br/>Update Complete
deactivate L
T4->>L: Update(newLocator2)
activate L
L->>RL: 尝试获取写锁<br/>Try Acquire Write Lock
Note over RL: 等待前一个写操作完成<br/>Wait for Previous Write<br/>保证写操作串行化
RL-->>L: 获取成功<br/>Acquire Success
L->>L: 检查数据源<br/>Check Data Source
L->>L: 调用IsFasterThan<br/>Call IsFasterThan
alt 需要更新 Need Update
L->>AM: 原子更新字段<br/>Atomic Update Fields
AM-->>L: 更新成功<br/>Update Success
L->>L: 合并ProgressVector<br/>Merge ProgressVector
L->>L: 更新UserData<br/>Update UserData
else 不需要更新 No Update
Note over L: 跳过更新<br/>Skip Update
end
RL-->>L: 释放写锁<br/>Release Write Lock
L-->>T4: 更新完成<br/>Update Complete
deactivate L
end
Note over T1,T4: 线程安全保证<br/>Thread Safety Guarantee<br/>读操作并发执行<br/>写操作互斥执行<br/>读写操作互斥
线程安全机制详解：
IsFasterThan() 是只读操作，不需要锁Update() 是写操作，需要锁保护std::atomic 保证基本类型的原子性std::shared_mutex 实现读写分离concurrentIdx 处理时间戳相同的情况线程安全实现的示例：
// framework/Locator.cpp
class Locator
{
private:
mutable std::shared_mutex _mutex; // 读写锁
uint64_t _src;
MultiProgress _multiProgress;
public:
// 读操作：使用共享锁
LocatorCompareResult IsFasterThan(const Locator& other) const {
std::shared_lock<std::shared_mutex> lock(_mutex);
// 只读操作，不需要互斥锁
return IsFasterThanImpl(other);
}
// 写操作：使用独占锁
void Update(const Locator& other) {
std::unique_lock<std::shared_mutex> lock(_mutex);
// 写操作，需要互斥锁
UpdateImpl(other);
}
};
Locator 的线程安全设计：支持并发访问，保证线程安全：
flowchart TB
Start([线程安全设计<br/>Thread Safety Design]) --> CoreLayer[核心机制层<br/>Core Mechanisms Layer]
subgraph CoreGroup["核心机制 Core Mechanisms"]
direction LR
C1[并发访问控制<br/>Concurrent Access Control<br/>多线程同时访问<br/>读写分离设计<br/>性能优化]
C2[线程安全保障<br/>Thread Safety Guarantee<br/>数据一致性保证<br/>无竞争条件<br/>可见性保证]
C3[锁机制设计<br/>Lock Mechanism Design<br/>读写锁实现<br/>细粒度锁控制<br/>死锁避免]
end
CoreLayer --> ImplementationLayer[实现机制层<br/>Implementation Mechanisms Layer]
subgraph ImplGroup["实现机制 Implementation Mechanisms"]
direction LR
I1[原子操作<br/>Atomic Operations<br/>std::atomic字段<br/>无锁读取<br/>原子更新]
I2[同步机制<br/>Synchronization Mechanisms<br/>std::shared_mutex<br/>读写锁分离<br/>条件变量]
I3[无锁设计<br/>Lock-Free Design<br/>读操作无锁<br/>CAS操作<br/>内存屏障]
end
ImplementationLayer --> FeatureLayer[特性支持层<br/>Feature Support Layer]
subgraph FeatureGroup["特性支持 Feature Support"]
direction LR
F1[读写分离<br/>Read-Write Separation<br/>多读单写模式<br/>读操作并发<br/>写操作互斥]
F2[细粒度控制<br/>Fine-Grained Control<br/>最小锁范围<br/>减少锁竞争<br/>提高并发度]
F3[性能优化<br/>Performance Optimization<br/>无锁读操作<br/>快速路径<br/>缓存友好]
end
FeatureLayer --> BenefitLayer[设计优势层<br/>Design Benefits Layer]
subgraph BenefitGroup["设计优势 Design Benefits"]
direction LR
B1[高并发性能<br/>High Concurrency Performance<br/>支持多线程并发读取<br/>减少锁竞争<br/>提高吞吐量]
B2[数据一致性<br/>Data Consistency<br/>保证数据正确性<br/>无竞争条件<br/>可见性保证]
B3[易于使用<br/>Easy to Use<br/>透明的线程安全<br/>简单的API接口<br/>无需手动同步]
end
BenefitLayer --> End([线程安全保证<br/>Thread Safety Guarantee])
CoreLayer -.->|包含| CoreGroup
ImplementationLayer -.->|包含| ImplGroup
FeatureLayer -.->|包含| FeatureGroup
BenefitLayer -.->|包含| BenefitGroup
C1 -->|使用| I1
C1 -->|使用| I2
C1 -->|使用| I3
C2 -->|依赖| I1
C2 -->|依赖| I2
C3 -->|实现| I2
C3 -->|支持| I3
I1 -->|支持| F1
I2 -->|实现| F1
I2 -->|实现| F2
I3 -->|支持| F1
I3 -->|支持| F3
F1 -->|带来| B1
F2 -->|带来| B1
F3 -->|带来| B1
F1 -->|保证| B2
F2 -->|保证| B2
C2 -->|直接带来| B2
C1 -->|直接带来| B3
C2 -->|直接带来| B3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CoreLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ImplementationLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style FeatureLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style BenefitLayer fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style CoreGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ImplGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style I1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style I2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style I3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FeatureGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style F1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style BenefitGroup fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style B1 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
style B3 fill:#a5d6a7,stroke:#388e3c,stroke-width:2px
Locator 与数据一致性是 IndexLib 的核心功能，通过 Locator 实现增量更新和数据一致性保证。通过本文的深入解析，我们了解到：
关键要点：
IsFasterThan() 判断数据是否已处理，支持多种比较结果Update() 更新 Locator，保证只向前推进理解 Locator 与数据一致性，是掌握 IndexLib 数据管理机制的关键。在下一篇文章中，我们将深入介绍文件系统抽象与存储格式的实现细节。
]]>索引类型概览：Normal、KV、KKV 三种索引类型的特点和应用场景：
flowchart TD
Start[索引类型体系] --> TypeLayer[索引类型层]
subgraph NormalGroup["NormalTable：标准表"]
direction TB
N1[支持全文检索<br/>倒排索引查询]
N2[支持属性查询<br/>正排索引查询]
N3[支持主键查询<br/>PrimaryKeyIndex]
N4[支持摘要查询<br/>SummaryReader]
N1 --> N2
N2 --> N3
N3 --> N4
end
subgraph KVGroup["KVTable：键值表"]
direction TB
K1[主键查询<br/>PrimaryKeyIndex]
K2[单值存储<br/>一个主键对应一个值]
K3[简单场景<br/>键值存储场景]
K1 --> K2
K2 --> K3
end
subgraph KKVGroup["KKVTable：键键值表"]
direction TB
KK1[主键+排序键查询<br/>PrimaryKey + SortKey]
KK2[多值存储<br/>一个主键对应多个值]
KK3[复杂场景<br/>多值存储场景]
KK1 --> KK2
KK2 --> KK3
end
TypeLayer --> NormalGroup
TypeLayer --> KVGroup
TypeLayer --> KKVGroup
NormalGroup --> Usage[使用场景]
KVGroup --> Usage
KKVGroup --> Usage
Usage --> U1[全文搜索场景<br/>NormalTable]
Usage --> U2[简单键值场景<br/>KVTable]
Usage --> U3[多值存储场景<br/>KKVTable]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style TypeLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style NormalGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style N1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style N2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style N3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style N4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style KVGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style K1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style K2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style K3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style KKVGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style KK1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style KK2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style KK3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Usage fill:#f5f5f5,stroke:#757575,stroke-width:2px
style U1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
IndexLib 支持三种主要的索引类型：
让我们先通过图来理解三种索引类型的区别：
索引类型对比：Normal、KV、KKV 的数据模型和查询方式：
flowchart TD
Start[索引类型对比] --> NormalTable[NormalTable<br/>标准表]
Start --> KVTable[KVTable<br/>键值表]
Start --> KKVTable[KKVTable<br/>键键值表]
subgraph NormalFeatures["NormalTable特性"]
direction TB
NF1[全文检索<br/>倒排索引]
NF2[属性查询<br/>正排索引]
NF3[主键查询<br/>PrimaryKeyIndex]
NF4[摘要查询<br/>SummaryReader]
NF1 --> NF2
NF2 --> NF3
NF3 --> NF4
end
subgraph KVFeatures["KVTable特性"]
direction TB
KF1[主键查询<br/>PrimaryKeyIndex]
KF2[单值存储<br/>一个主键对应一个值]
KF3[简单场景<br/>键值存储]
KF1 --> KF2
KF2 --> KF3
end
subgraph KKVFeatures["KKVTable特性"]
direction TB
KKF1[主键+排序键查询<br/>PrimaryKey + SortKey]
KKF2[多值存储<br/>一个主键对应多个值]
KKF3[复杂场景<br/>多值存储]
KKF1 --> KKF2
KKF2 --> KKF3
end
NormalTable --> NormalFeatures
KVTable --> KVFeatures
KKVTable --> KKVFeatures
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style NormalTable fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style NormalFeatures fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style NF1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style NF2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style NF3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style NF4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style KVTable fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style KVFeatures fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style KF1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style KF2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style KF3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style KKVTable fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style KKVFeatures fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style KKF1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style KKF2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style KKF3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
不同索引类型适用于不同的场景。让我们通过类图来理解三种索引类型的架构关系：
classDiagram
class ITabletFactory {
<<interface>>
+ CreateTabletWriter()
+ CreateTabletReader()
+ CreateMemSegment()
+ CreateDiskSegment()
}
class NormalTableFactory {
+ CreateTabletWriter()
+ CreateTabletReader()
}
class KVTableFactory {
+ CreateTabletWriter()
+ CreateTabletReader()
}
class KKVTableFactory {
+ CreateTabletWriter()
+ CreateTabletReader()
}
class NormalTabletReader {
- MultiFieldIndexReader _multiFieldIndexReader
- AttributeReader _attributeReader
- PrimaryKeyReader _primaryKeyReader
+ Search()
}
class KVTabletReader {
- KVIndexReader _kvIndexReader
- PackAttributeFormatter _formatter
+ Search()
}
class KKVTabletReader {
- KKVReader _kkvReader
- KKVIterator _iterator
+ Search()
}
ITabletFactory <|-- NormalTableFactory : 实现
ITabletFactory <|-- KVTableFactory : 实现
ITabletFactory <|-- KKVTableFactory : 实现
NormalTableFactory --> NormalTabletReader : 创建
KVTableFactory --> KVTabletReader : 创建
KKVTableFactory --> KKVTabletReader : 创建
索引类型的选择：
不同索引类型适用于不同的场景，选择合适的索引类型可以显著提高系统性能和降低复杂度：
NormalTable 是标准表，支持完整的索引功能：
NormalTable 的特点：支持全文检索、倒排索引、正排索引等：
flowchart TD
NormalTable[NormalTable<br/>标准表] --> Main
subgraph Main["主要组件"]
direction LR
A[倒排索引<br/>InvertedIndex<br/>全文检索]
B[正排索引<br/>AttributeIndex<br/>属性查询]
C[主键索引<br/>PrimaryKeyIndex<br/>主键查询]
end
NormalTable --> A
NormalTable --> B
NormalTable --> C
A --> Sub1
B --> Sub2
C --> Sub1
subgraph Sub["子组件"]
direction LR
Sub1[摘要索引<br/>SummaryIndex<br/>文档摘要]
Sub2[删除位图<br/>DeletionMap<br/>删除标记]
end
Sub --> Sub1
Sub --> Sub2
style NormalTable fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Sub fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Sub1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style Sub2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
主要特点：
NormalTable 的架构：
NormalTable 的架构：NormalTabletReader、NormalTabletWriter 等组件：
flowchart TD
NormalTable[NormalTable<br/>标准表架构] --> Main
subgraph Main["主要组件"]
direction LR
A[NormalTabletReader<br/>读取器<br/>查询入口]
B[NormalTabletWriter<br/>写入器<br/>数据写入]
C[IndexReader<br/>索引读取器<br/>索引查询]
end
NormalTable --> A
NormalTable --> B
NormalTable --> C
A --> Sub1
B --> Sub2
C --> Sub1
subgraph Sub["子组件"]
direction LR
Sub1[AttributeReader<br/>属性读取器<br/>属性查询]
Sub2[SummaryReader<br/>摘要读取器<br/>文档摘要]
end
Sub --> Sub1
Sub --> Sub2
style NormalTable fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Sub fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Sub1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style Sub2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
核心组件：
NormalTable 的架构采用组合模式，将不同的索引 Reader 组合在一起。让我们通过类图来理解详细的架构：
classDiagram
class NormalTabletReader {
- MultiFieldIndexReader _multiFieldIndexReader
- AttributeReader _attributeReader
- PrimaryKeyReader _primaryKeyReader
- SummaryReader _summaryReader
- DeletionMapReader _deletionMapReader
+ Search()
+ GetMultiFieldIndexReader()
+ GetAttributeReader()
}
class MultiFieldIndexReader {
- map_string_InvertedIndexReader _indexReaders
+ Search()
}
class AttributeReader {
- map_string_AttributeIndexReader _readers
+ ReadAttribute()
}
class PrimaryKeyReader {
- PrimaryKeyIndexReader _reader
+ Lookup()
}
NormalTabletReader --> MultiFieldIndexReader : 包含
NormalTabletReader --> AttributeReader : 包含
NormalTabletReader --> PrimaryKeyReader : 包含
核心组件详解：
NormalTable 的查询方式：
NormalTable 的查询：全文检索、属性查询、主键查询等：
flowchart TD
NormalTable[NormalTable<br/>查询方式] --> Main
subgraph Main["主要查询方式"]
direction LR
A[全文检索<br/>FullTextSearch<br/>倒排索引查询]
B[属性查询<br/>AttributeQuery<br/>正排索引查询]
C[主键查询<br/>PrimaryKeyQuery<br/>主键索引查询]
end
NormalTable --> A
NormalTable --> B
NormalTable --> C
A --> Sub1
B --> Sub2
C --> Sub1
subgraph Sub["扩展查询方式"]
direction LR
Sub1[范围查询<br/>RangeQuery<br/>范围条件查询]
Sub2[聚合查询<br/>AggregationQuery<br/>数据聚合统计]
end
Sub --> Sub1
Sub --> Sub2
style NormalTable fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Sub fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Sub1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style Sub2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
查询方式：
NormalTable 支持多种查询方式，可以灵活组合使用。让我们通过序列图来理解完整的查询流程：
sequenceDiagram
participant Client
participant NormalReader as NormalTabletReader
participant InvertedReader as MultiFieldIndexReader
participant AttributeReader as AttributeReader
participant PrimaryKeyReader as PrimaryKeyReader
participant ResultMerger as ResultMerger
Client->>NormalReader: Search(jsonQuery)
NormalReader->>NormalReader: 解析查询类型
alt 全文检索查询
NormalReader->>InvertedReader: Search(termQuery)
InvertedReader-->>NormalReader: DocIdList1
end
alt 属性查询
NormalReader->>AttributeReader: Filter(attrQuery)
AttributeReader-->>NormalReader: DocIdList2
end
alt 主键查询
NormalReader->>PrimaryKeyReader: Lookup(pk)
PrimaryKeyReader-->>NormalReader: DocId3
end
NormalReader->>ResultMerger: MergeResults([List1, List2, DocId3])
ResultMerger->>ResultMerger: 去重
ResultMerger->>ResultMerger: 排序
ResultMerger-->>NormalReader: MergedResult
NormalReader->>AttributeReader: ReadAttributes(MergedResult)
AttributeReader-->>NormalReader: Attributes
NormalReader->>NormalReader: 序列化为JSON
NormalReader-->>Client: jsonResult
查询方式详解：
KVTable 是键值表，支持简单的键值存储：
KVTable 的特点：支持主键查询、简单的键值存储：
flowchart TD
KVTable[KVTable<br/>键值表] --> Main
subgraph Main["主要组件"]
direction LR
A[主键索引<br/>PrimaryKeyIndex<br/>主键查询]
B[键值存储<br/>KeyValueStorage<br/>键值对存储]
C[属性查询<br/>AttributeQuery<br/>属性条件查询]
end
KVTable --> A
KVTable --> B
KVTable --> C
A --> Sub1
B --> Sub2
C --> Sub1
subgraph Sub["子组件"]
direction LR
Sub1[打包属性<br/>PackAttribute<br/>属性打包存储]
Sub2[值读取器<br/>ValueReader<br/>值数据读取]
end
Sub --> Sub1
Sub --> Sub2
style KVTable fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Sub fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Sub1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style Sub2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
主要特点：
KVTable 的架构：
KVTable 的架构：KVTabletReader、KVTabletWriter 等组件：
flowchart TD
KVTable[KVTable<br/>键值表架构] --> Main
subgraph Main["主要组件"]
direction LR
A[KVTabletReader<br/>KV读取器<br/>查询入口]
B[KVTabletWriter<br/>KV写入器<br/>数据写入]
C[KVIndexReader<br/>KV索引读取器<br/>索引查询]
end
KVTable --> A
KVTable --> B
KVTable --> C
A --> Sub1
B --> Sub2
C --> Sub1
subgraph Sub["子组件"]
direction LR
Sub1[PackAttributeFormatter<br/>打包属性格式化器<br/>属性格式化]
Sub2[ValueReader<br/>值读取器<br/>值数据读取]
end
Sub --> Sub1
Sub --> Sub2
style KVTable fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Sub fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Sub1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style Sub2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
核心组件：
KVTable 的查询方式：
KVTable 的查询：主键查询、属性查询等：
flowchart LR
subgraph Query["查询接口层"]
A["主键查询<br/>PrimaryKeyQuery"]
B["批量主键查询<br/>BatchPrimaryKeyQuery"]
C["属性查询<br/>AttributeQuery"]
end
subgraph Read["数据读取层"]
D["值读取<br/>ValueRead<br/>读取完整值"]
E["属性读取<br/>AttributeRead<br/>读取指定属性"]
end
A -->|单主键定位| D
B -->|批量主键定位| D
B -->|指定属性| E
C -->|属性过滤| E
style Query fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Read fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
查询方式：
查询示例：
// table/kv_table/KVTabletReader.h
// JSON 查询格式
{
"pk": ["key1", "key2"], // 主键列表
"pkNumber": [123456, 623445], // 数字主键列表
"attrs": ["attr1", "attr2"], // 要查询的属性
"indexName": "kv1" // 索引名称
}
KKVTable 是键键值表，支持主键+排序键查询：
KKVTable 的特点：支持主键+排序键查询、多值存储：
flowchart LR
subgraph Index["索引层"]
A["主键索引<br/>PrimaryKeyIndex<br/>定位主键"]
B["排序键索引<br/>SortKeyIndex<br/>排序键定位"]
end
subgraph Storage["存储层"]
C["多值存储<br/>MultiValueStorage<br/>一个主键多个值"]
end
subgraph Query["查询层"]
D["范围查询<br/>RangeQuery<br/>排序键范围"]
E["属性查询<br/>AttributeQuery<br/>指定属性"]
end
A -->|主键定位| C
B -->|排序键定位| C
A -->|主键+排序键| D
B -->|排序键范围| D
C -->|读取数据| E
style Index fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Storage fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Query fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#a5d6a7,stroke:#388e3c,stroke-width:1.5px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
主要特点：
KKVTable 的架构：
KKVTable 的架构：KKVTabletReader、KKVTabletWriter 等组件：
flowchart LR
subgraph Read["读取组件"]
A["KKVTabletReader<br/>KKV表查询器<br/>主键+排序键查询"]
C["KKVReader<br/>KKV索引读取器<br/>索引查询"]
end
subgraph Write["写入组件"]
B["KKVTabletWriter<br/>KKV表写入器<br/>键键值构建"]
end
subgraph Support["支持组件"]
D["KKVIterator<br/>KKV迭代器<br/>范围查询迭代"]
E["ValueReader<br/>值读取器<br/>读取存储值"]
end
A -->|使用| D
A -->|使用| E
C -->|使用| D
C -->|使用| E
B -->|写入时读取| E
style Read fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Write fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style Support fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#a5d6a7,stroke:#388e3c,stroke-width:1.5px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
核心组件：
KKVTable 的查询方式：
KKVTable 的查询：主键+排序键查询、范围查询等：
flowchart LR
subgraph Query["查询类型"]
A["主键查询<br/>PrimaryKeyQuery<br/>查询所有值"]
B["主键+排序键查询<br/>PrimaryKey+SortKeyQuery<br/>精确查询"]
C["范围查询<br/>RangeQuery<br/>排序键范围"]
end
subgraph Support["查询能力"]
D["属性查询<br/>AttributeQuery<br/>指定属性"]
E["迭代器查询<br/>IteratorQuery<br/>范围迭代"]
end
A -->|可配合| D
B -->|可配合| D
C -->|使用| E
C -->|可配合| D
E -->|可配合| D
style Query fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Support fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
查询方式：
查询示例：
// table/kkv_table/KKVTabletReader.h
// JSON 查询格式
{
"pk": ["key1"], // 主键
"pkNumber": [123456], // 数字主键
"attrs": ["attr1", "attr2"], // 要查询的属性
"skey": ["skey1", "skey2"] // 排序键列表
}
不同索引类型的 TabletReader 实现差异：
TabletReader 的实现差异：NormalTabletReader、KVTabletReader、KKVTabletReader：
flowchart LR
subgraph Reader["TabletReader 类型"]
A["NormalTabletReader<br/>标准表读取器<br/>全文检索+属性+主键"]
B["KVTabletReader<br/>键值表读取器<br/>主键查询"]
C["KKVTabletReader<br/>键键值表读取器<br/>主键+排序键"]
end
subgraph Component["核心组件"]
D["IndexReader<br/>索引读取器<br/>索引定位"]
E["AttributeReader<br/>属性读取器<br/>属性读取"]
end
A -->|使用| D
A -->|使用| E
B -->|使用| D
B -->|使用| E
C -->|使用| D
C -->|使用| E
style Reader fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Component fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
实现差异：
不同索引类型的 TabletWriter 实现差异：
TabletWriter 的实现差异：NormalTabletWriter、KVTabletWriter、KKVTabletWriter：
flowchart TB
subgraph Writer["TabletWriter 类型"]
A["NormalTabletWriter<br/>标准表写入器"]
B["KVTabletWriter<br/>键值表写入器"]
C["KKVTabletWriter<br/>键键值表写入器"]
end
subgraph Build["构建流程"]
A1["文档构建<br/>DocumentBuilder<br/>倒排+正排+主键"]
B1["键值构建<br/>IndexBuilder<br/>主键索引"]
C1["键键值构建<br/>IndexBuilder<br/>主键+排序键"]
end
subgraph Output["构建输出"]
A2["NormalTable<br/>倒排索引+正排索引+主键索引"]
B2["KVTable<br/>主键索引+值存储"]
C2["KKVTable<br/>主键索引+排序键索引+多值存储"]
end
A -->|完整构建流程| A1
B -->|简化构建流程| B1
C -->|扩展构建流程| C1
A1 -->|生成| A2
B1 -->|生成| B2
C1 -->|生成| C2
style Writer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Build fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Output fill:#c8e6c9,stroke:#388e3c,stroke-width:2px
style A fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style B fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style C fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style A1 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style B1 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style A2 fill:#a5d6a7,stroke:#388e3c,stroke-width:1.5px
style B2 fill:#a5d6a7,stroke:#388e3c,stroke-width:1.5px
style C2 fill:#a5d6a7,stroke:#388e3c,stroke-width:1.5px
实现差异：
不同索引类型的索引构建差异：
索引构建的差异：Normal、KV、KKV 的索引构建流程：
flowchart TD
subgraph Main["主要组件"]
A["NormalTable构建<br/>倒排+正排+主键"]
B["KVTable构建<br/>主键索引"]
C["KKVTable构建<br/>主键+排序键"]
end
subgraph Sub["子组件"]
D["索引构建器<br/>IndexBuilder"]
E["文档构建器<br/>DocumentBuilder"]
end
A --> D
B --> E
C --> D
style Main fill:#e3f2fd
style Sub fill:#fff3e0
构建差异：
选择 NormalTable 的场景：
选择 NormalTable 的场景：全文检索、复杂查询等场景：
flowchart LR
subgraph Scene["NormalTable 适用场景"]
direction LR
A["全文检索场景<br/>FullTextSearch<br/>文本搜索需求"]
B["复杂查询场景<br/>ComplexQuery<br/>多条件组合查询"]
C["多字段查询场景<br/>MultiFieldQuery<br/>多字段联合查询"]
end
subgraph Feature["NormalTable 核心查询能力"]
direction LR
D["范围查询<br/>RangeQuery<br/>数值/时间范围"]
E["聚合查询<br/>AggregationQuery<br/>统计聚合"]
F["排序查询<br/>SortQuery<br/>结果排序"]
G["过滤查询<br/>FilterQuery<br/>条件过滤"]
end
Scene ==>|提供完整支持| Feature
style Scene fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Feature fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
适用场景：
选择 KVTable 的场景：
选择 KVTable 的场景：简单的键值存储、主键查询等场景：
flowchart LR
subgraph Scene["KVTable 适用场景"]
direction LR
A["简单存储场景<br/>SimpleStorage<br/>键值存储需求"]
B["主键查询场景<br/>PrimaryKeyQuery<br/>主键快速查询"]
C["高性能场景<br/>HighPerformance<br/>高性能要求"]
end
subgraph Feature["KVTable 核心能力"]
direction LR
D["键值存储<br/>KeyValueStorage<br/>简单键值对"]
E["快速查询<br/>FastQuery<br/>主键定位"]
F["简化接口<br/>SimpleInterface<br/>易于使用"]
end
Scene ==>|提供| Feature
style Scene fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Feature fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
适用场景：
选择 KKVTable 的场景：
选择 KKVTable 的场景：多值存储、主键+排序键查询等场景：
flowchart LR
subgraph Scene["KKVTable 适用场景"]
direction LR
A["多值存储场景<br/>MultiValueStorage<br/>一主键多值"]
B["排序键查询场景<br/>SortKeyQuery<br/>主键+排序键"]
C["范围查询场景<br/>RangeQuery<br/>排序键范围"]
end
subgraph Feature["KKVTable 核心能力"]
direction LR
D["有序存储<br/>OrderedStorage<br/>排序键有序"]
E["迭代器查询<br/>IteratorQuery<br/>范围迭代"]
F["多值管理<br/>MultiValueManagement<br/>值集合管理"]
end
Scene ==>|提供| Feature
style Scene fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Feature fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
适用场景：
不同索引类型的查询性能对比：
查询性能对比：Normal、KV、KKV 的查询性能特点：
flowchart TB
subgraph Type["索引类型"]
direction LR
A["NormalTable<br/>标准表<br/>全文检索性能高"]
B["KVTable<br/>键值表<br/>主键查询性能最高"]
C["KKVTable<br/>键键值表<br/>主键+排序键性能高"]
end
subgraph Query["查询场景性能"]
direction LR
D["全文检索<br/>FullTextSearch<br/>NormalTable 高"]
E["主键查询<br/>PrimaryKeyQuery<br/>KVTable 最高"]
F["主键+排序键<br/>PK+SortKeyQuery<br/>KKVTable 高"]
G["范围查询<br/>RangeQuery<br/>KKVTable 中等"]
H["复杂查询<br/>ComplexQuery<br/>NormalTable 中等"]
end
A -->|擅长| D
A -->|支持| H
B -->|最优| E
C -->|擅长| F
C -->|支持| G
style Type fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Query fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style H fill:#ffcc80,stroke:#f57c00,stroke-width:2px
性能特点：
不同索引类型的存储性能对比：
存储性能对比：Normal、KV、KKV 的存储性能特点：
flowchart TB
subgraph Type["索引类型"]
direction LR
A["NormalTable<br/>存储空间：较大<br/>需要多种索引"]
B["KVTable<br/>存储空间：较小<br/>只需主键索引"]
C["KKVTable<br/>存储空间：中等<br/>主键+排序键索引"]
end
subgraph Storage["存储组成"]
direction LR
D["倒排索引<br/>InvertedIndex<br/>NormalTable 需要"]
E["正排索引<br/>ForwardIndex<br/>NormalTable 需要"]
F["主键索引<br/>PrimaryKeyIndex<br/>三种都需要"]
G["排序键索引<br/>SortKeyIndex<br/>KKVTable 需要"]
H["数据存储<br/>DataStorage<br/>三种都需要"]
end
A -->|包含| D
A -->|包含| E
A -->|包含| F
A -->|包含| H
B -->|包含| F
B -->|包含| H
C -->|包含| F
C -->|包含| G
C -->|包含| H
style Type fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Storage fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style H fill:#ffcc80,stroke:#f57c00,stroke-width:2px
存储特点：
不同索引类型的构建性能对比：
构建性能对比：Normal、KV、KKV 的构建性能特点：
flowchart TB
subgraph Type["索引类型"]
direction LR
A["NormalTable<br/>构建时间：较长<br/>需要构建多种索引"]
B["KVTable<br/>构建时间：最短<br/>构建流程简化"]
C["KKVTable<br/>构建时间：中等<br/>主键+排序键索引"]
end
subgraph Build["构建流程"]
direction LR
D["文档构建<br/>DocumentBuild<br/>NormalTable 需要"]
E["倒排索引构建<br/>InvertedIndexBuild<br/>NormalTable 需要"]
F["正排索引构建<br/>ForwardIndexBuild<br/>NormalTable 需要"]
G["主键索引构建<br/>PrimaryKeyIndexBuild<br/>三种都需要"]
H["排序键索引构建<br/>SortKeyIndexBuild<br/>KKVTable 需要"]
end
A -->|包含| D
A -->|包含| E
A -->|包含| F
A -->|包含| G
B -->|包含| G
C -->|包含| G
C -->|包含| H
style Type fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Build fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style G fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style H fill:#ffcc80,stroke:#f57c00,stroke-width:2px
构建特点：
IndexLib 支持自定义索引类型：
自定义索引类型：通过实现接口扩展索引类型：
flowchart TB
subgraph Step["扩展步骤"]
direction LR
A["实现 TabletReader<br/>自定义读取器<br/>实现查询接口"]
B["实现 TabletWriter<br/>自定义写入器<br/>实现写入接口"]
C["实现索引构建<br/>自定义构建逻辑<br/>实现构建流程"]
end
subgraph Action["扩展操作"]
direction LR
D["注册索引类型<br/>RegisterType<br/>注册到系统"]
E["扩展接口<br/>ExtendInterface<br/>扩展查询能力"]
F["配置索引类型<br/>ConfigureType<br/>配置参数"]
end
A -->|完成后| D
B -->|完成后| D
C -->|完成后| D
D -->|支持| E
D -->|需要| F
style Step fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Action fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
扩展方式：
索引类型的演进：
索引类型的演进：从 Normal 到 KV、KKV 的演进过程：
flowchart LR
subgraph Evolution["索引类型演进"]
direction LR
A["NormalTable<br/>最早版本<br/>完整功能支持"]
B["KVTable<br/>简单场景优化<br/>主键查询优化"]
C["KKVTable<br/>多值存储优化<br/>排序键支持"]
end
subgraph Optimization["优化方向"]
direction LR
D["功能演进<br/>FeatureEvolution<br/>功能扩展"]
E["性能优化<br/>PerformanceOptimization<br/>查询性能提升"]
F["场景优化<br/>ScenarioOptimization<br/>特定场景优化"]
end
A -->|演进| B
B -->|演进| C
A -.->|带来| D
B -.->|带来| E
C -.->|带来| F
style Evolution fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Optimization fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
演进过程：
索引类型的统一接口设计：
统一的接口设计：ITabletReader、ITabletWriter 等统一接口：
flowchart TB
subgraph Interface["统一接口层"]
direction LR
A["ITabletReader<br/>统一查询接口<br/>定义查询规范"]
B["ITabletWriter<br/>统一写入接口<br/>定义写入规范"]
C["ITabletSchema<br/>统一Schema接口<br/>定义Schema规范"]
end
subgraph Implementation["接口实现层"]
direction LR
D["NormalTable实现<br/>完整功能实现"]
E["KVTable实现<br/>简化功能实现"]
F["KKVTable实现<br/>扩展功能实现"]
end
A -->|实现| D
A -->|实现| E
A -->|实现| F
B -->|实现| D
B -->|实现| E
B -->|实现| F
C -->|实现| D
C -->|实现| E
C -->|实现| F
style Interface fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Implementation fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
索引类型的灵活扩展设计：
灵活的扩展设计：支持自定义索引类型和扩展功能：
flowchart TB
subgraph Design["扩展设计机制"]
direction LR
A["接口抽象<br/>InterfaceAbstraction<br/>统一接口定义"]
B["插件机制<br/>PluginMechanism<br/>动态加载扩展"]
C["配置驱动<br/>ConfigurationDriven<br/>配置选择类型"]
end
subgraph Extension["扩展能力"]
direction LR
D["类型扩展<br/>TypeExtension<br/>自定义索引类型"]
E["功能扩展<br/>FeatureExtension<br/>扩展查询功能"]
F["接口扩展<br/>InterfaceExtension<br/>扩展接口能力"]
end
A -->|支持| D
A -->|支持| F
B -->|支持| D
B -->|支持| E
C -->|支持| D
style Design fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Extension fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
索引类型的性能优化设计：
性能优化设计：针对不同索引类型的性能优化策略：
flowchart TB
subgraph Strategy["优化策略"]
direction LR
A["针对性优化<br/>TargetedOptimization<br/>针对索引类型特点"]
B["查询优化<br/>QueryOptimization<br/>优化查询路径"]
C["构建优化<br/>BuildOptimization<br/>优化构建流程"]
end
subgraph Optimization["优化手段"]
direction LR
D["性能调优<br/>PerformanceTuning<br/>提升查询性能"]
E["资源优化<br/>ResourceOptimization<br/>降低资源消耗"]
F["索引优化<br/>IndexOptimization<br/>优化索引结构"]
end
A -->|采用| D
A -->|采用| F
B -->|采用| D
B -->|采用| F
C -->|采用| E
C -->|采用| F
style Strategy fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Optimization fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C fill:#90caf9,stroke:#1976d2,stroke-width:2px
style D fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style E fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F fill:#ffcc80,stroke:#f57c00,stroke-width:2px
设计要点：
选择策略：
优化策略：
优化策略：
索引类型是 IndexLib 的核心功能，通过 NormalTable、KVTable、KKVTable 三种类型支持不同的应用场景。通过本文的深入解析，我们了解到：
核心类型：
设计亮点：
性能特点：
理解索引类型，是掌握 IndexLib 索引功能的关键。在下一篇文章中，我们将深入介绍 Locator 与数据一致性的实现细节，包括 Locator 的完整结构、比较逻辑、更新机制、数据一致性保证等各个方面的实现原理和性能优化策略。
]]>内存管理与资源控制概览：从内存配额到内存回收的完整机制：
flowchart TB
Start([内存管理与资源控制概览<br/>Memory Management & Resource Control Overview]) --> QuotaLayer[内存配额层<br/>Memory Quota Layer]
subgraph QuotaGroup["内存配额 Memory Quota"]
direction TB
Q1[MemoryQuotaController<br/>配额控制器<br/>管理内存配额和分配]
Q2[层级配额管理<br/>Hierarchical Quota<br/>支持多级配额管理]
Q3[配额分配<br/>Quota Allocation<br/>动态分配内存配额]
Q1 --> Q2
Q2 --> Q3
end
QuotaLayer --> CalculateLayer[内存计算层<br/>Memory Calculation Layer]
subgraph CalculateGroup["内存计算 Memory Calculation"]
direction TB
C1[TabletMemoryCalculator<br/>内存计算器<br/>计算Tablet内存使用]
C2[实时统计<br/>Real-time Statistics<br/>实时统计内存使用]
C3[分类统计<br/>Categorized Statistics<br/>按类型统计内存]
C1 --> C2
C2 --> C3
end
CalculateLayer --> ReclaimLayer[内存回收层<br/>Memory Reclaim Layer]
subgraph ReclaimGroup["内存回收 Memory Reclaim"]
direction TB
R1[IIndexMemoryReclaimer<br/>内存回收器<br/>回收不再使用的内存]
R2[延迟回收<br/>Delayed Reclaim<br/>延迟回收避免频繁操作]
R3[按需回收<br/>On-Demand Reclaim<br/>内存紧张时按需回收]
R1 --> R2
R2 --> R3
end
ReclaimLayer --> ResourceLayer[资源控制层<br/>Resource Control Layer]
subgraph ResourceGroup["资源控制 Resource Control"]
direction TB
RE1[BuildResourceCalculator<br/>构建资源计算器<br/>计算构建资源使用]
RE2[资源估算<br/>Resource Estimation<br/>估算资源需求]
RE3[资源预留<br/>Resource Reservation<br/>预留构建资源]
RE1 --> RE2
RE2 --> RE3
end
ResourceLayer --> End([内存管理完成<br/>Memory Management Complete])
QuotaLayer -.->|包含| QuotaGroup
CalculateLayer -.->|包含| CalculateGroup
ReclaimLayer -.->|包含| ReclaimGroup
ResourceLayer -.->|包含| ResourceGroup
Q3 -.->|使用| C1
C3 -.->|触发| R1
R3 -.->|使用| RE1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style QuotaLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CalculateLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ReclaimLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ResourceLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style QuotaGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Q1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Q2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Q3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CalculateGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ReclaimGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style R1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style R3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style ResourceGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style RE1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style RE2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style RE3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
IndexLib 的内存管理包括以下核心概念：
让我们先通过图来理解内存管理的整体架构：
内存管理架构：MemoryQuotaController、TabletMemoryCalculator、IIndexMemoryReclaimer 的关系（已在上面类图中展示，此处不再重复）：
内存管理在 IndexLib 中起到关键作用，是系统稳定性和性能的基础。让我们通过类图来理解内存管理的整体架构：
classDiagram
class MemoryQuotaController {
- string _name
- int64_t _rootQuota
- atomic_int64_t _localFreeQuota
- atomic_int64_t _reservedParentQuota
- MemoryQuotaController _parentController
+ Allocate()
+ TryAllocate()
+ Reserve()
+ Free()
+ GetAllocatedQuota()
}
class TabletMemoryCalculator {
- TabletWriter _tabletWriter
- TabletReaderContainer _tabletReaderContainer
+ GetRtBuiltSegmentsMemsize()
+ GetRtIndexMemsize()
+ GetBuildingSegmentMemsize()
}
class IIndexMemoryReclaimer {
<<interface>>
+ Retire()
+ DropRetireItem()
+ TryReclaim()
+ Reclaim()
}
class BuildResourceCalculator {
+ GetCurrentTotalMemoryUse()
+ EstimateDumpTempMemoryUse()
+ EstimateDumpExpandMemoryUse()
}
MemoryQuotaController --> MemoryQuotaController : 层级关系
TabletMemoryCalculator --> MemoryQuotaController : 使用
IIndexMemoryReclaimer --> MemoryQuotaController : 使用
BuildResourceCalculator --> MemoryQuotaController : 使用
内存管理的核心作用：
MemoryQuotaController 是内存配额控制器，定义在 base/MemoryQuotaController.h 中：
// base/MemoryQuotaController.h
class MemoryQuotaController
{
public:
// 构造函数：创建根配额控制器
MemoryQuotaController(std::string name, int64_t totalQuota);
// 构造函数：创建子配额控制器
MemoryQuotaController(std::string name,
std::shared_ptr<MemoryQuotaController> parentController);
// 分配内存配额
void Allocate(int64_t quota);
Status TryAllocate(int64_t quota); // 尝试分配，不阻塞
// 预留内存配额
Status Reserve(int64_t quota);
// 释放内存配额
void Free(int64_t quota);
// 获取内存配额信息
int64_t GetAllocatedQuota() const; // 已分配配额
int64_t GetFreeQuota() const; // 可用配额
int64_t GetTotalQuota() const; // 总配额
private:
std::string _name; // 控制器名称
const int64_t _rootQuota; // 根配额（根控制器）
std::atomic<int64_t> _localFreeQuota; // 本地可用配额
std::atomic<int64_t> _reservedParentQuota; // 从父控制器预留的配额
std::shared_ptr<MemoryQuotaController> _parentController; // 父控制器
};
MemoryQuotaController 的关键字段：
MemoryQuotaController 的结构：包含配额信息和父控制器：
flowchart TD
subgraph Controller["MemoryQuotaController"]
C1["name<br/>控制器名称"]
C2["rootQuota<br/>根配额"]
C3["localFreeQuota<br/>本地可用配额"]
C4["reservedParentQuota<br/>预留父配额"]
C5["parentController<br/>父控制器引用"]
end
subgraph Methods["关键方法"]
M1["Allocate<br/>分配配额"]
M2["TryAllocate<br/>尝试分配"]
M3["Reserve<br/>预留配额"]
M4["Free<br/>释放配额"]
end
C1 --> M1
C2 --> M1
C3 --> M1
C4 --> M3
C5 --> M1
style Controller fill:#e3f2fd
style Methods fill:#fff3e0
内存配额的分配机制：
内存配额分配：从父控制器到子控制器的配额分配（已在上面详细展示，此处不再重复）：
内存配额分配流程图：
flowchart TD
Start[请求分配内存<br/>Allocate请求] --> CheckLocal[检查本地配额<br/>localFreeQuota]
CheckLocal --> LocalEnough{本地配额足够?}
LocalEnough -->|是| AllocateLocal[从本地分配<br/>减少localFreeQuota]
LocalEnough -->|否| RequestParent[向父控制器请求<br/>parentController.Allocate]
AllocateLocal --> UpdateLocal[更新本地配额<br/>更新localFreeQuota]
RequestParent --> CheckParent{父控制器有配额?}
CheckParent -->|是| AllocateParent[从父控制器分配<br/>减少父控制器的配额]
CheckParent -->|否| HandleNoQuota[处理配额不足]
AllocateParent --> ReserveParent[预留父配额<br/>增加reservedParentQuota]
ReserveParent --> UpdateLocal
HandleNoQuota --> WaitOrReject{等待或拒绝?}
WaitOrReject -->|等待| WaitQuota[等待配额释放<br/>阻塞或轮询]
WaitOrReject -->|拒绝| RejectAlloc[拒绝分配<br/>返回失败]
WaitQuota --> CheckLocal
UpdateLocal --> Success[分配完成<br/>返回成功]
RejectAlloc --> Fail[分配失败<br/>返回错误]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CheckLocal fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style LocalEnough fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style AllocateLocal fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style RequestParent fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckParent fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style AllocateParent fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ReserveParent fill:#fff3e0,stroke:#f57c00,stroke-width:1px
style UpdateLocal fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style HandleNoQuota fill:#ffebee,stroke:#c62828,stroke-width:1px
style WaitOrReject fill:#ffebee,stroke:#c62828,stroke-width:1px
style WaitQuota fill:#fff9c4,stroke:#f57f17,stroke-width:1px
style RejectAlloc fill:#ffebee,stroke:#c62828,stroke-width:2px
style Success fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Fail fill:#ffebee,stroke:#c62828,stroke-width:2px
分配机制详解：
内存配额的分配是内存管理的核心机制。让我们通过序列图来理解完整的分配流程：
sequenceDiagram
participant Component as 组件
participant ChildCtrl as 子控制器
participant ParentCtrl as 父控制器
participant RootCtrl as 根控制器
Component->>ChildCtrl: Allocate(quota)
ChildCtrl->>ChildCtrl: 检查本地配额
alt 本地配额足够
ChildCtrl->>ChildCtrl: 从本地分配
ChildCtrl-->>Component: Success
else 本地配额不足
ChildCtrl->>ParentCtrl: 请求配额
ParentCtrl->>ParentCtrl: 检查配额
alt 父控制器有配额
ParentCtrl->>ChildCtrl: 分配配额
ChildCtrl->>ChildCtrl: 更新本地配额
ChildCtrl-->>Component: Success
else 父控制器配额不足
ParentCtrl->>RootCtrl: 请求配额
RootCtrl->>ParentCtrl: 分配配额
ParentCtrl->>ChildCtrl: 分配配额
ChildCtrl->>ChildCtrl: 更新本地配额
ChildCtrl-->>Component: Success
end
end
分配机制详解：
分配策略的优势：
MemoryQuotaController 支持层级配额管理：
层级配额管理：从根控制器到子控制器的层级结构：
flowchart TD
subgraph Root["根控制器"]
R1["总配额<br/>Total Quota"]
end
subgraph Partition["分区控制器"]
P1["分区配额<br/>Partition Quota"]
P2["分区配额<br/>Partition Quota"]
end
subgraph Tablet["Tablet 控制器"]
T1["Tablet配额<br/>Tablet Quota"]
T2["Tablet配额<br/>Tablet Quota"]
end
R1 --> P1
R1 --> P2
P1 --> T1
P2 --> T2
style Root fill:#e3f2fd
style Partition fill:#fff3e0
style Tablet fill:#f3e5f5
层级结构：
层级配额管理是 IndexLib 内存管理的核心设计。让我们通过类图来理解层级结构：
classDiagram
class RootController {
- int64_t _rootQuota = 100GB
+ Allocate()
+ GetFreeQuota()
}
class PartitionController {
- MemoryQuotaController _parent
- int64_t _localFreeQuota
+ Allocate()
+ GetFreeQuota()
}
class TabletController {
- MemoryQuotaController _parent
- int64_t _localFreeQuota
+ Allocate()
+ GetFreeQuota()
}
class BuildController {
- MemoryQuotaController _parent
- int64_t _localFreeQuota
+ Allocate()
}
class QueryController {
- MemoryQuotaController _parent
- int64_t _localFreeQuota
+ Allocate()
}
RootController --> PartitionController : 分配配额
PartitionController --> TabletController : 分配配额
TabletController --> BuildController : 分配配额
TabletController --> QueryController : 分配配额
层级结构详解：
层级管理的优势：
配额分配的策略：
配额分配策略：按需分配、预留分配等策略：
flowchart TD
subgraph Strategies["分配策略"]
S1["按需分配<br/>On-Demand"]
S2["预留分配<br/>Reserved"]
S3["阻塞分配<br/>Blocking"]
S4["非阻塞分配<br/>Non-Blocking"]
end
subgraph Methods["分配方法"]
M1["Allocate<br/>阻塞分配"]
M2["TryAllocate<br/>非阻塞分配"]
M3["Reserve<br/>预留分配"]
end
S1 --> M1
S2 --> M3
S3 --> M1
S4 --> M2
style Strategies fill:#e3f2fd
style Methods fill:#fff3e0
分配策略：
TabletMemoryCalculator 是 Tablet 内存计算器，定义在 framework/TabletMemoryCalculator.h 中：
// framework/TabletMemoryCalculator.h
class TabletMemoryCalculator final
{
public:
TabletMemoryCalculator(const std::shared_ptr<TabletWriter>& tabletWriter,
const std::shared_ptr<TabletReaderContainer>& tabletReaderContainer);
// 获取各种内存使用量
size_t GetRtBuiltSegmentsMemsize() const; // 实时已构建 Segment 内存
size_t GetRtIndexMemsize() const; // 实时索引内存
size_t GetIncIndexMemsize() const; // 增量索引内存
size_t GetBuildingSegmentMemsize() const; // 构建中 Segment 内存
size_t GetDumpingSegmentMemsize() const; // 转储中 Segment 内存
size_t GetBuildingSegmentDumpExpandMemsize() const; // 转储扩展内存
private:
std::shared_ptr<TabletWriter> _tabletWriter;
std::shared_ptr<TabletReaderContainer> _tabletReaderContainer;
};
TabletMemoryCalculator 的关键方法：
TabletMemoryCalculator 的方法：计算各种内存使用量：
flowchart TD
subgraph Calculator["TabletMemoryCalculator"]
C1["GetRtBuiltSegmentsMemsize<br/>实时构建段内存"]
C2["GetRtIndexMemsize<br/>实时索引内存"]
C3["GetBuildingSegmentMemsize<br/>构建中段内存"]
C4["GetDumpingSegmentMemsize<br/>转储中段内存"]
end
subgraph Components["统计组件"]
CO1["TabletWriter<br/>写入器内存"]
CO2["TabletReaderContainer<br/>查询器容器内存"]
CO3["IndexReader<br/>索引读取器内存"]
end
C1 --> CO1
C2 --> CO2
C3 --> CO1
C4 --> CO1
style Calculator fill:#e3f2fd
style Components fill:#fff3e0
内存使用统计的流程：
内存使用统计：从 Tablet 组件到内存使用量的统计流程（已在上面详细展示，此处不再重复）：
统计流程：
内存使用统计是监控和调优的基础。让我们通过序列图来理解完整的统计流程：
sequenceDiagram
participant Calculator as TabletMemoryCalculator
participant Writer as TabletWriter
participant ReaderContainer as TabletReaderContainer
participant MemSeg as MemSegment
participant DiskSeg as DiskSegment
participant Indexer as Indexer
Calculator->>Writer: GetBuildingSegment()
Writer-->>Calculator: MemSegment
Calculator->>MemSeg: EvaluateCurrentMemUsed()
MemSeg->>Indexer: GetMemUsed()
Indexer-->>MemSeg: memUsed
MemSeg-->>Calculator: buildingMemSize
Calculator->>Writer: GetDumpingSegment()
Writer-->>Calculator: DumpingSegment
Calculator->>DumpingSegment: EvaluateCurrentMemUsed()
DumpingSegment-->>Calculator: dumpingMemSize
Calculator->>ReaderContainer: GetTabletReaders()
ReaderContainer-->>Calculator: TabletReaders
loop 遍历每个TabletReader
Calculator->>DiskSeg: GetIndexMemsize()
DiskSeg-->>Calculator: indexMemSize
end
Calculator->>Calculator: 汇总所有内存使用
Calculator-->>Calculator: 返回统计结果
统计流程详解：
统计的用途：
IIndexMemoryReclaimer 是索引内存回收器的接口，定义在 framework/mem_reclaimer/IIndexMemoryReclaimer.h 中：
// framework/mem_reclaimer/IIndexMemoryReclaimer.h
class IIndexMemoryReclaimer
{
public:
// 回收内存：将内存加入回收队列
virtual int64_t Retire(void* addr, std::function<void(void*)> deAllocator) = 0;
// 取消回收：从回收队列中移除
virtual void DropRetireItem(int64_t itemId) = 0;
// 尝试回收：尝试回收一些内存
virtual void TryReclaim() = 0;
// 强制回收：强制回收所有可回收的内存
virtual void Reclaim() = 0;
};
IIndexMemoryReclaimer 的关键方法：
IIndexMemoryReclaimer 接口：提供内存回收的抽象：
flowchart TD
subgraph Interface["IIndexMemoryReclaimer 接口"]
I1["Retire<br/>标记待回收"]
I2["DropRetireItem<br/>删除待回收项"]
I3["TryReclaim<br/>尝试回收"]
I4["Reclaim<br/>强制回收"]
end
subgraph Lifecycle["生命周期"]
L1["使用中<br/>In Use"]
L2["待回收<br/>Retired"]
L3["已回收<br/>Reclaimed"]
L1 -->|Retire| L2
L2 -->|Reclaim| L3
end
I1 --> L2
I3 --> L3
style Interface fill:#e3f2fd
style Lifecycle fill:#fff3e0
内存回收的机制：
内存回收机制：从 Retire 到 Reclaim 的回收流程：
flowchart TB
Start([内存回收流程<br/>Memory Reclaim Flow]) --> UseLayer[使用中阶段<br/>In Use Phase]
subgraph UseGroup["使用中 In Use"]
direction TB
U1[内存使用中<br/>Memory In Use<br/>内存正在被使用]
end
UseLayer --> RetireLayer[标记待回收阶段<br/>Retire Phase]
subgraph RetireGroup["标记待回收 Retire"]
direction TB
R1[标记待回收<br/>Retire<br/>标记为待回收状态]
R2[加入回收队列<br/>Add to Reclaim Queue<br/>加入延迟回收队列]
R1 --> R2
end
RetireLayer --> DelayLayer[延迟回收阶段<br/>Delayed Reclaim Phase]
subgraph DelayGroup["延迟回收 Delayed Reclaim"]
direction TB
D1[延迟回收<br/>Delayed Reclaim<br/>延迟一段时间后回收]
D2{内存紧张?<br/>Memory Pressure?}
D1 --> D2
end
DelayLayer --> ReclaimLayer[执行回收阶段<br/>Reclaim Phase]
subgraph ReclaimGroup["执行回收 Execute Reclaim"]
direction TB
E1[执行回收<br/>Reclaim<br/>尝试回收内存]
E2[释放内存<br/>Free Memory<br/>释放内存空间]
E1 --> E2
end
ReclaimLayer --> End([回收完成<br/>Reclaim Complete])
UseLayer -.->|包含| UseGroup
RetireLayer -.->|包含| RetireGroup
DelayLayer -.->|包含| DelayGroup
ReclaimLayer -.->|包含| ReclaimGroup
U1 --> R1
R2 --> D1
D2 -->|是| E1
D2 -->|否| D1
E2 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style UseLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style RetireLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style DelayLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ReclaimLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style UseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style U1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style RetireGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style R1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style R2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style DelayGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style D1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style D2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style ReclaimGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style E1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style E2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
内存回收流程图：
graph TD
A[内存不再使用] --> B[Retire 加入回收队列]
B --> C[延迟回收]
C --> D{内存是否紧张?}
D -->|否| E[继续延迟]
D -->|是| F[TryReclaim 尝试回收]
E --> D
F --> G{回收成功?}
G -->|是| H[释放内存]
G -->|否| I[Reclaim 强制回收]
I --> H
H --> J[回收完成]
style C fill:#e3f2fd
style F fill:#fff3e0
style I fill:#f3e5f5
style H fill:#e8f5e9
回收机制详解：
内存回收是保证系统稳定性的关键机制。让我们通过序列图来理解完整的回收流程：
sequenceDiagram
participant Component as 组件
participant Reclaimer as IIndexMemoryReclaimer
participant RetireQueue as RetireQueue
participant MemoryQuota as MemoryQuotaController
Component->>Reclaimer: Retire(addr, deAllocator)
Reclaimer->>RetireQueue: AddRetireItem(addr, deAllocator)
RetireQueue-->>Reclaimer: itemId
Reclaimer-->>Component: itemId
Note over Reclaimer: 延迟回收，等待合适时机
MemoryQuota->>Reclaimer: 内存紧张，触发回收
Reclaimer->>Reclaimer: TryReclaim()
Reclaimer->>RetireQueue: GetRetireItems()
RetireQueue-->>Reclaimer: RetireItems
loop 遍历回收项
Reclaimer->>Reclaimer: 检查是否可以回收
alt 可以回收
Reclaimer->>Component: deAllocator(addr)
Component->>MemoryQuota: Free(quota)
MemoryQuota-->>Component: Success
end
end
alt 内存仍然紧张
Reclaimer->>Reclaimer: Reclaim()
Reclaimer->>RetireQueue: ForceReclaimAll()
RetireQueue-->>Reclaimer: Success
end
回收机制详解：
回收策略的优势：
内存回收的策略：
内存回收策略：延迟回收、按需回收等策略：
flowchart TD
subgraph Strategies["回收策略"]
S1["延迟回收<br/>Delayed Reclaim"]
S2["按需回收<br/>On-Demand Reclaim"]
S3["批量回收<br/>Batch Reclaim"]
end
subgraph Triggers["触发条件"]
T1["内存紧张<br/>Memory Pressure"]
T2["定期回收<br/>Periodic Reclaim"]
T3["手动触发<br/>Manual Trigger"]
end
T1 --> S2
T2 --> S1
T3 --> S3
style Strategies fill:#e3f2fd
style Triggers fill:#fff3e0
回收策略：
BuildResourceCalculator 是构建资源计算器，定义在 util/memory_control/BuildResourceCalculator.h 中：
// util/memory_control/BuildResourceCalculator.h
class BuildResourceCalculator
{
public:
// 获取当前总内存使用
static int64_t GetCurrentTotalMemoryUse(const BuildResourceMetricsPtr& metrics);
// 估算转储临时内存使用
static int64_t EstimateDumpTempMemoryUse(const BuildResourceMetricsPtr& metrics,
int dumpThreadCount);
// 估算转储扩展内存使用
static int64_t EstimateDumpExpandMemoryUse(const BuildResourceMetricsPtr& metrics);
// 估算转储文件大小
static int64_t EstimateDumpFileSize(const BuildResourceMetricsPtr& metrics);
};
BuildResourceCalculator 的关键方法：
BuildResourceCalculator 的方法：计算构建资源使用：
flowchart TD
subgraph Calculator["BuildResourceCalculator"]
C1["GetCurrentTotalMemoryUse<br/>当前总内存使用"]
C2["EstimateDumpTempMemoryUse<br/>估算转储临时内存"]
C3["EstimateDumpExpandMemoryUse<br/>估算转储扩展内存"]
end
subgraph Metrics["资源指标"]
M1["构建内存<br/>Build Memory"]
M2["转储内存<br/>Dump Memory"]
M3["索引内存<br/>Index Memory"]
end
C1 --> M1
C2 --> M2
C3 --> M2
style Calculator fill:#e3f2fd
style Metrics fill:#fff3e0
构建资源估算的流程：
构建资源估算：从 BuildResourceMetrics 到资源使用量的估算流程：
flowchart TD
subgraph Metrics["BuildResourceMetrics"]
M1["文档数量<br/>Doc Count"]
M2["索引大小<br/>Index Size"]
M3["字段数量<br/>Field Count"]
end
subgraph Estimate["资源估算"]
E1["估算构建内存<br/>Estimate Build Memory"]
E2["估算转储内存<br/>Estimate Dump Memory"]
E3["估算总内存<br/>Estimate Total Memory"]
end
M1 --> E1
M2 --> E2
M3 --> E1
E1 --> E3
E2 --> E3
style Metrics fill:#e3f2fd
style Estimate fill:#fff3e0
估算流程：
内存分配的策略：
内存分配策略：按需分配、预留分配等策略（已在上面详细展示，此处不再重复）：
分配策略：
内存分配的优化：
内存分配优化：批量分配、内存池等优化策略：
flowchart TD
subgraph Optimization["优化策略"]
O1["批量分配<br/>Batch Allocation"]
O2["内存池<br/>Memory Pool"]
O3["对齐分配<br/>Aligned Allocation"]
end
subgraph Benefits["优化收益"]
B1["减少分配次数<br/>Reduce Allocations"]
B2["减少内存碎片<br/>Reduce Fragmentation"]
B3["提高访问效率<br/>Improve Access"]
end
O1 --> B1
O2 --> B2
O3 --> B3
style Optimization fill:#e3f2fd
style Benefits fill:#fff3e0
优化策略：
内存回收的时机：
内存回收时机：延迟回收、按需回收等时机：
flowchart TD
subgraph Timing["回收时机"]
T1["延迟回收<br/>Delayed Reclaim<br/>延迟一段时间后回收"]
T2["按需回收<br/>On-Demand Reclaim<br/>内存紧张时回收"]
T3["定期回收<br/>Periodic Reclaim<br/>定期触发回收"]
end
subgraph Conditions["触发条件"]
C1["内存使用率超过阈值"]
C2["配额不足"]
C3["定时器触发"]
end
C1 --> T2
C2 --> T2
C3 --> T3
style Timing fill:#e3f2fd
style Conditions fill:#fff3e0
回收时机：
内存回收的优化：
内存回收优化：批量回收、延迟回收等优化策略（已在上面详细展示，此处不再重复）：
优化策略：
内存使用的优化：
内存使用优化：内存池、缓存控制等优化策略：
flowchart TD
subgraph Optimization["优化策略"]
O1["内存池<br/>Memory Pool"]
O2["缓存控制<br/>Cache Control"]
O3["懒加载<br/>Lazy Loading"]
end
subgraph Benefits["优化收益"]
B1["减少内存分配开销"]
B2["控制内存使用上限"]
B3["按需加载减少内存占用"]
end
O1 --> B1
O2 --> B2
O3 --> B3
style Optimization fill:#e3f2fd
style Benefits fill:#fff3e0
优化策略：
内存监控与告警：
内存监控与告警：实时监控内存使用，及时告警：
flowchart TD
subgraph Monitor["监控"]
M1["实时统计<br/>Real-time Statistics"]
M2["内存使用率<br/>Memory Usage Rate"]
M3["配额使用率<br/>Quota Usage Rate"]
end
subgraph Alert["告警"]
A1["阈值告警<br/>Threshold Alert"]
A2["异常告警<br/>Anomaly Alert"]
A3["趋势告警<br/>Trend Alert"]
end
M1 --> A1
M2 --> A2
M3 --> A3
style Monitor fill:#e3f2fd
style Alert fill:#fff3e0
监控与告警：
层级配额管理的设计：
层级配额管理：从根控制器到子控制器的层级结构：
flowchart TD
subgraph Root["根控制器"]
R1["总配额<br/>Total Quota"]
end
subgraph Partition["分区控制器"]
P1["分区配额<br/>Partition Quota"]
P2["分区配额<br/>Partition Quota"]
end
subgraph Tablet["Tablet 控制器"]
T1["Tablet配额<br/>Tablet Quota"]
T2["Tablet配额<br/>Tablet Quota"]
end
R1 --> P1
R1 --> P2
P1 --> T1
P2 --> T2
style Root fill:#e3f2fd
style Partition fill:#fff3e0
style Tablet fill:#f3e5f5
设计要点：
内存回收的设计：
内存回收设计：延迟回收、按需回收等设计：
flowchart TD
subgraph Main["主要组件"]
A["延迟回收<br/>DelayedRecycle"]
B["按需回收<br/>OnDemandRecycle"]
C["并发安全<br/>ConcurrentSafe"]
end
subgraph Sub["子组件"]
D["资源释放<br/>ResourceRelease"]
E["内存清理<br/>MemoryCleanup"]
end
A --> D
B --> E
C --> D
style Main fill:#e3f2fd
style Sub fill:#fff3e0
设计要点：
性能优化的设计：
性能优化设计：内存池、批量操作等优化策略：
flowchart TD
subgraph Main["主要组件"]
A["内存池<br/>MemoryPool"]
B["批量操作<br/>BatchOperation"]
C["缓存优化<br/>CacheOptimization"]
end
subgraph Sub["子组件"]
D["资源控制<br/>ResourceControl"]
E["性能调优<br/>PerformanceTuning"]
end
A --> D
B --> E
C --> D
style Main fill:#e3f2fd
style Sub fill:#fff3e0
设计要点：
优化策略：
优化策略：
优化策略：
内存管理与资源控制是 IndexLib 的核心功能，通过 MemoryQuotaController、TabletMemoryCalculator、IIndexMemoryReclaimer 等组件实现。通过本文的深入解析，我们了解到：
核心组件：
设计亮点：
性能优化：
理解内存管理与资源控制，是掌握 IndexLib 资源管理机制的关键。在下一篇文章中，我们将深入介绍索引类型的实现细节，包括 NormalTable、KVTable、KKVTable 等不同索引类型的特点、实现原理和使用场景。
]]>Segment 合并策略概览：从合并策略到合并执行的完整流程：
flowchart TB
Start([Segment 合并策略概览<br/>Segment Merge Strategy Overview]) --> StrategyLayer[合并策略层<br/>Merge Strategy Layer]
subgraph StrategyGroup["合并策略 Merge Strategy"]
direction TB
S1[MergeStrategy<br/>策略接口<br/>合并策略抽象接口]
S2[OptimizeMergeStrategy<br/>优化合并策略<br/>优化索引结构]
S3[RealtimeMergeStrategy<br/>实时合并策略<br/>实时合并小Segment]
S4[ShardBasedMergeStrategy<br/>分片合并策略<br/>按分片合并]
S1 --> S2
S1 --> S3
S1 --> S4
end
StrategyLayer --> PlanLayer[合并计划层<br/>Merge Plan Layer]
subgraph PlanGroup["合并计划 Merge Plan"]
direction TB
P1[MergePlan<br/>合并计划<br/>合并任务计划]
P2[SegmentMergePlan<br/>Segment合并计划<br/>Segment合并详情]
P3[目标版本<br/>Target Version<br/>合并后的目标版本]
P1 --> P2
P1 --> P3
end
PlanLayer --> ExecutionLayer[合并执行层<br/>Merge Execution Layer]
subgraph ExecutionGroup["合并执行 Merge Execution"]
direction TB
E1[VersionMerger<br/>版本合并器<br/>执行版本合并]
E2[IndexMergeOperation<br/>合并操作<br/>索引合并操作]
E3[读取源Segment<br/>Read Source Segments<br/>读取待合并Segment]
E4[合并索引数据<br/>Merge Index Data<br/>合并倒排/正排索引]
E5[写入目标Segment<br/>Write Target Segment<br/>写入合并后的Segment]
E1 --> E2
E2 --> E3
E3 --> E4
E4 --> E5
end
ExecutionLayer --> CommitLayer[版本提交层<br/>Version Commit Layer]
subgraph CommitGroup["版本提交 Version Commit"]
direction TB
C1[创建新版本<br/>Create New Version<br/>创建包含合并Segment的版本]
C2[Fence机制<br/>Fence Mechanism<br/>原子性保证]
C3[提交版本<br/>Commit Version<br/>提交新版本]
C4[清理旧Segment<br/>Cleanup Old Segments<br/>删除不再需要的Segment]
C1 --> C2
C2 --> C3
C3 --> C4
end
CommitLayer --> End([合并完成<br/>Merge Complete])
StrategyLayer -.->|包含| StrategyGroup
PlanLayer -.->|包含| PlanGroup
ExecutionLayer -.->|包含| ExecutionGroup
CommitLayer -.->|包含| CommitGroup
S2 -.->|生成| P1
P2 -.->|执行| E1
E5 -.->|创建| C1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style PlanLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ExecutionLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style CommitLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style PlanGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style P1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ExecutionGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style E1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style E2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style E3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style E4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style E5 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style CommitGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style C1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C4 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
Segment 合并的主要目的包括：
让我们先通过图来理解 Segment 合并的整体流程：
Segment 合并流程：从合并策略到合并执行的完整过程（已在上面详细展示，此处不再重复）：
Segment 合并流程图：
flowchart TD
Start([开始合并]) --> GetVersion[获取当前版本<br/>从TabletData获取Version]
GetVersion --> SelectStrategy[选择合并策略<br/>MergeStrategy]
SelectStrategy --> StrategyType{合并策略类型}
StrategyType -->|优化合并| OptimizeStrategy[优化合并策略<br/>选择需要合并的Segment]
StrategyType -->|实时合并| RealtimeStrategy[实时合并策略<br/>实时合并小Segment]
StrategyType -->|分片合并| ShardStrategy[分片合并策略<br/>按分片合并]
OptimizeStrategy --> CreatePlan
RealtimeStrategy --> CreatePlan
ShardStrategy --> CreatePlan
CreatePlan[创建MergePlan<br/>包含Segment列表和目标版本] --> ValidatePlan[验证MergePlan<br/>检查Segment有效性]
ValidatePlan --> PlanValid{验证通过?}
PlanValid -->|否| AdjustStrategy[调整策略<br/>重新选择Segment]
AdjustStrategy -.->|重新选择| SelectStrategy
PlanValid -->|是| Merge
subgraph Merge["执行合并"]
direction LR
EM1[创建合并操作<br/>IndexMergeOperation]
EM2[读取源Segment<br/>从多个Segment读取]
EM3[合并索引数据<br/>倒排/正排/主键索引]
EM4[写入目标Segment<br/>写入合并后的数据]
EM1 --> EM2 --> EM3 --> EM4
end
Merge --> EM1
EM4 --> CreateVersion[创建新版本<br/>包含合并后的Segment]
CreateVersion --> CommitVersion[提交新版本<br/>使用Fence机制保证原子性]
CommitVersion --> Cleanup[清理旧Segment<br/>删除不再需要的Segment文件]
Cleanup --> End([完成合并])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style GetVersion fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style SelectStrategy fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style StrategyType fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style OptimizeStrategy fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style RealtimeStrategy fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style ShardStrategy fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style CreatePlan fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ValidatePlan fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style PlanValid fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style AdjustStrategy fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Merge fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style EM1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style EM2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style EM3 fill:#ba68c8,stroke:#7b1fa2,stroke-width:1.5px
style EM4 fill:#ab47bc,stroke:#7b1fa2,stroke-width:1.5px
style CreateVersion fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CommitVersion fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Cleanup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
Segment 合并包括以下核心组件，它们协同工作完成合并任务。让我们通过类图来理解各组件的关系：
classDiagram
class MergeStrategy {
<<interface>>
+ GetName()
+ CreateMergePlan()
}
class OptimizeMergeStrategy {
- OptimizeMergeParams _params
+ CreateMergePlan()
}
class MergePlan {
- vector_SegmentMergePlan _mergePlan
- Version _targetVersion
+ AddMergePlan()
+ GetTargetVersion()
}
class SegmentMergePlan {
- vector_segmentid_t _srcSegments
- segmentid_t _targetSegment
+ AddSrcSegment()
+ SetTargetSegment()
}
class VersionMerger {
- ITabletMergeController _controller
- IIndexTaskPlanCreator _planCreator
+ ExecuteTask()
+ Run()
}
class IndexMergeOperation {
- vector_Segment _srcSegments
- Segment _targetSegment
+ Execute()
+ MergeIndex()
}
MergeStrategy <|-- OptimizeMergeStrategy : 实现
MergeStrategy --> MergePlan : 创建
MergePlan --> SegmentMergePlan : 包含
VersionMerger --> MergeStrategy : 使用
VersionMerger --> IndexMergeOperation : 执行
IndexMergeOperation --> SegmentMergePlan : 使用
核心组件详解：
MergeStrategy 是合并策略的抽象接口，定义在 table/index_task/merger/MergeStrategy.h 中：
// table/index_task/merger/MergeStrategy.h
class MergeStrategy
{
public:
virtual ~MergeStrategy() {}
// 获取策略名称
virtual std::string GetName() const = 0;
// 创建合并计划：根据 Context 创建合并计划
virtual std::pair<Status, std::shared_ptr<MergePlan>>
CreateMergePlan(const framework::IndexTaskContext* context) = 0;
};
MergeStrategy 的关键方法：
MergeStrategy 接口：提供合并策略的抽象：
flowchart TD
subgraph Interface["MergeStrategy 接口"]
I1[MergeStrategy<br/>策略接口]
I2[GetName<br/>获取策略名称]
I3[CreateMergePlan<br/>创建合并计划]
I1 --> I2
I1 --> I3
end
subgraph Context["IndexTaskContext"]
C1[当前版本<br/>Current Version]
C2[Segment列表<br/>Segment List]
C3[合并参数<br/>Merge Parameters]
C1 --> C2
C2 --> C3
end
subgraph Result["返回结果"]
R1[Status<br/>状态码]
R2[MergePlan<br/>合并计划]
R1 --> R2
end
I3 --> C1
C3 --> I3
I3 --> R1
style Interface fill:#e3f2fd
style Context fill:#fff3e0
style Result fill:#f3e5f5
IndexLib 支持多种合并策略：
合并策略类型：Optimize、Realtime、ShardBased 等：
flowchart TB
Start([合并策略体系<br/>Merge Strategy System]) --> BaseLayer[基础接口层<br/>Base Interface Layer]
subgraph BaseGroup["基础接口 Base Interface"]
direction TB
B1[MergeStrategy<br/>策略接口<br/>合并策略抽象接口]
end
BaseLayer --> StrategyLayer[策略实现层<br/>Strategy Implementation Layer]
subgraph StrategyGroup["合并策略实现 Merge Strategy Implementations"]
direction TB
S1[OptimizeMergeStrategy<br/>优化合并策略<br/>合并所有符合条件的Segment]
S2[RealtimeMergeStrategy<br/>实时合并策略<br/>实时合并小Segment]
S3[ShardBasedMergeStrategy<br/>分片合并策略<br/>按分片合并Segment]
S4[KeyValueOptimizeMergeStrategy<br/>KV优化合并策略<br/>针对KV表的优化合并]
end
StrategyLayer --> FeatureLayer[策略特性层<br/>Strategy Features Layer]
subgraph FeatureGroup["策略特性 Strategy Features"]
direction TB
F1[全量合并<br/>Full Merge<br/>合并所有Segment]
F2[实时合并<br/>Realtime Merge<br/>实时合并小Segment]
F3[分片合并<br/>Shard Merge<br/>按分片合并]
F4[KV优化<br/>KV Optimize<br/>KV表优化合并]
end
FeatureLayer --> End([策略体系完成<br/>Strategy System Complete])
BaseLayer -.->|包含| BaseGroup
StrategyLayer -.->|包含| StrategyGroup
FeatureLayer -.->|包含| FeatureGroup
B1 -.->|实现| S1
B1 -.->|实现| S2
B1 -.->|实现| S3
B1 -.->|实现| S4
S1 -.->|提供| F1
S2 -.->|提供| F2
S3 -.->|提供| F3
S4 -.->|提供| F4
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style BaseLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style StrategyLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FeatureLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style BaseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style B1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style StrategyGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FeatureGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style F1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
合并策略类型：
OptimizeMergeStrategy 是优化合并策略的实现，定义在 table/index_task/merger/OptimizeMergeStrategy.h 中：
// table/index_task/merger/OptimizeMergeStrategy.h
class OptimizeMergeStrategy : public MergeStrategy
{
public:
std::string GetName() const override {
return MergeStrategyDefine::OPTIMIZE_MERGE_STRATEGY_NAME;
}
// 创建合并计划
std::pair<Status, std::shared_ptr<MergePlan>>
CreateMergePlan(const framework::IndexTaskContext* context) override;
private:
// 合并参数
struct OptimizeMergeParams {
uint32_t maxDocCount; // 参与合并的 Segment 的最大文档数
uint64_t afterMergeMaxDocCount; // 合并后的最小文档数
uint32_t afterMergeMaxSegmentCount; // 合并后的最大 Segment 数
bool skipSingleMergedSegment; // 是否跳过单个已合并的 Segment
};
OptimizeMergeParams _params;
};
OptimizeMergeStrategy 的关键参数：
OptimizeMergeStrategy 参数：控制合并行为的关键参数：
flowchart TD
subgraph Params["OptimizeMergeParams 参数"]
P1[maxDocCount<br/>参与合并的最大文档数<br/>只有小于等于该值的Segment参与合并]
P2[afterMergeMaxDocCount<br/>合并后的最小文档数<br/>控制合并后Segment的大小]
P3[afterMergeMaxSegmentCount<br/>合并后的最大Segment数<br/>控制合并后Segment的数量]
P4[skipSingleMergedSegment<br/>是否跳过单个已合并的Segment<br/>避免重复合并]
end
subgraph Impact["参数影响"]
I1[控制参与合并的Segment<br/>maxDocCount越大包含越多]
I2[控制合并后Segment大小<br/>afterMergeMaxDocCount越大Segment越大]
I3[控制合并后Segment数量<br/>afterMergeMaxSegmentCount越小Segment越少]
I4[控制合并策略<br/>skipSingleMergedSegment避免重复合并]
end
P1 --> I1
P2 --> I2
P3 --> I3
P4 --> I4
style Params fill:#e3f2fd
style Impact fill:#fff3e0
合并策略的选择逻辑：
合并策略的选择逻辑：根据 Segment 特征选择合并策略（已在上面详细展示，此处不再重复）：
选择逻辑：
合并策略的选择逻辑是合并流程的关键。让我们通过流程图来理解详细的选择过程：
flowchart TD
Start([开始创建合并计划]) --> Collect[收集源Segment<br/>从TabletData获取所有Segment]
Collect --> Filter[过滤Segment<br/>筛选符合条件的Segment]
Filter --> Check{检查maxDocCount<br/>文档数量限制}
Check -->|docCount <= maxDocCount| Keep[保留Segment<br/>符合合并条件]
Check -->|docCount > maxDocCount| Skip[跳过Segment<br/>超过限制，不合并]
Keep --> Group
Skip --> Group
Group[分组Segment<br/>将Segment进行分组] --> Calculate[计算目标Segment数<br/>根据合并后大小计算]
Calculate --> GroupBy[根据afterMergeMaxDocCount分组<br/>按目标文档数分组]
GroupBy --> CreatePlan[创建SegmentMergePlan<br/>为每组创建合并计划]
CreatePlan --> SetTarget[设置目标Segment<br/>指定合并后的Segment]
SetTarget --> CreateMergePlan[创建MergePlan<br/>包含所有合并计划]
CreateMergePlan --> SetVersion[设置目标版本<br/>指定合并后的版本号]
SetVersion --> End([完成])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Collect fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Filter fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Check fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Keep fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style Skip fill:#ffcdd2,stroke:#c62828,stroke-width:1.5px
style Group fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Calculate fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style GroupBy fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CreatePlan fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style SetTarget fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style CreateMergePlan fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style SetVersion fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
选择逻辑详解：
ST_BUILT）maxDocCount 过滤 Segment，只保留符合条件的 Segment
maxDocCount 的 SegmentskipSingleMergedSegment 为 true，跳过单个已合并的 SegmentafterMergeMaxDocCount 和 afterMergeMaxSegmentCount 分组 Segment
afterMergeMaxDocCount 计算目标 Segment 数afterMergeMaxDocCountMergePlan 是合并计划，定义在 table/index_task/merger/MergePlan.h 中：
// table/index_task/merger/MergePlan.h
class MergePlan : public framework::IndexTaskResource,
public autil::legacy::Jsonizable
{
public:
// 添加合并计划
void AddMergePlan(const SegmentMergePlan& segmentMergePlan);
// 获取合并计划
const SegmentMergePlan& GetSegmentMergePlan(size_t index);
// 获取目标版本
const framework::Version& GetTargetVersion() const;
void SetTargetVersion(framework::Version targetVersion);
// 创建新版本
static framework::Version CreateNewVersion(
const std::shared_ptr<MergePlan>& mergePlan,
const framework::IndexTaskContext* taskContext);
private:
std::vector<SegmentMergePlan> _mergePlan; // 合并计划列表
framework::Version _targetVersion; // 目标版本
};
MergePlan 的关键字段：
MergePlan 的结构：包含 SegmentMergePlan 列表和目标版本：
flowchart TD
subgraph MergePlan["MergePlan 对象"]
MP[MergePlan<br/>合并计划]
end
subgraph Fields["核心字段"]
F1[SegmentMergePlan列表<br/>vector SegmentMergePlan<br/>每个计划合并一组Segment]
F2[目标版本<br/>Version targetVersion<br/>合并后的目标版本]
end
subgraph SegmentMergePlan["SegmentMergePlan 结构"]
SM1[源Segment列表<br/>vector segmentid_t<br/>要合并的Segment]
SM2[目标Segment<br/>segmentid_t<br/>合并后的Segment]
SM1 --> SM2
end
subgraph TargetVersion["目标版本"]
TV1[VersionId<br/>版本号]
TV2[Segment列表<br/>合并后的Segment列表]
TV3[Locator<br/>位置信息]
TV1 --> TV2
TV2 --> TV3
end
MP --> F1
MP --> F2
F1 --> SM1
F2 --> TV1
style MergePlan fill:#e3f2fd
style Fields fill:#fff3e0
style SegmentMergePlan fill:#f3e5f5
style TargetVersion fill:#e8f5e9
SegmentMergePlan 是单个 Segment 合并计划：
// table/index_task/merger/SegmentMergePlan.h
class SegmentMergePlan
{
public:
// 添加源 Segment
void AddSrcSegment(segmentid_t segmentId);
// 设置目标 Segment
void SetTargetSegment(segmentid_t segmentId);
// 获取源 Segment 列表
const std::vector<segmentid_t>& GetSrcSegments() const;
// 获取目标 Segment
segmentid_t GetTargetSegment() const;
private:
std::vector<segmentid_t> _srcSegments; // 源 Segment 列表
segmentid_t _targetSegment; // 目标 Segment
};
SegmentMergePlan 的关键字段：
SegmentMergePlan 的结构：包含源 Segment 列表和目标 Segment（已在上面详细展示，此处不再重复）：
合并计划的创建流程：
合并计划的创建流程：从收集 Segment 到创建合并计划（已在上面详细展示，此处不再重复）：
创建流程：
VersionMerger 是版本合并器，管理合并流程和版本更新，定义在 framework/VersionMerger.h 中：
// framework/VersionMerger.h
class VersionMerger
{
public:
// 执行合并任务
future_lite::coro::Lazy<std::pair<Status, versionid_t>>
ExecuteTask(const Version& sourceVersion,
const std::string& taskType,
const std::string& taskName,
const std::map<std::string, std::string>& params);
// 运行合并流程
future_lite::coro::Lazy<std::pair<Status, versionid_t>> Run();
// 获取合并后的版本信息
std::shared_ptr<MergedVersionInfo> GetMergedVersionInfo();
// 判断是否需要提交
bool NeedCommit() const;
private:
std::string _tabletName;
std::shared_ptr<ITabletMergeController> _controller;
std::unique_ptr<IIndexTaskPlanCreator> _planCreator;
Version _currentBaseVersion;
std::shared_ptr<MergedVersionInfo> _mergedVersionInfo;
};
VersionMerger 的关键组件：
VersionMerger 的结构：管理合并流程和版本更新：
flowchart TD
subgraph VersionMerger["VersionMerger 对象"]
VM[VersionMerger<br/>版本合并器]
end
subgraph Components["核心组件"]
C1[MergeController<br/>ITabletMergeController<br/>管理合并任务执行]
C2[PlanCreator<br/>IIndexTaskPlanCreator<br/>创建合并计划]
C3[CurrentBaseVersion<br/>Version<br/>当前基础版本]
C4[MergedVersionInfo<br/>合并后的版本信息<br/>包含基础版本和目标版本]
end
subgraph Methods["关键方法"]
M1[ExecuteTask<br/>执行合并任务]
M2[Run<br/>运行合并流程]
M3[GetMergedVersionInfo<br/>获取合并后的版本信息]
M4[NeedCommit<br/>判断是否需要提交]
end
VM --> C1
VM --> C2
VM --> C3
VM --> C4
C1 --> M1
C2 --> M2
C4 --> M3
C3 --> M4
style VersionMerger fill:#e3f2fd
style Components fill:#fff3e0
style Methods fill:#f3e5f5
合并执行的完整流程：
合并执行流程：从创建合并计划到提交新版本（已在上面详细展示，此处不再重复）：
执行流程：
合并执行是合并流程的核心，需要高效地处理大量 Segment。让我们通过序列图来理解完整的执行流程：
sequenceDiagram
participant Controller as MergeController
participant Strategy as MergeStrategy
participant Plan as MergePlan
participant Merger as VersionMerger
participant Operation as IndexMergeOperation
participant Seg1 as Segment1
participant Seg2 as Segment2
participant Seg3 as Segment3
participant TargetSeg as TargetSegment
participant VersionCommitter as VersionCommitter
Controller->>Strategy: CreateMergePlan(Context)
Strategy->>Strategy: 收集源Segment
Strategy->>Strategy: 过滤Segment
Strategy->>Strategy: 分组Segment
Strategy->>Plan: 创建MergePlan
Plan-->>Strategy: MergePlan
Strategy-->>Controller: MergePlan
Controller->>Merger: ExecuteTask(Version, MergePlan)
Merger->>Operation: CreateIndexMergeOperation(MergePlan)
Operation-->>Merger: IndexMergeOperation
Merger->>Operation: Execute()
Operation->>Seg1: ReadIndexData()
Operation->>Seg2: ReadIndexData()
Operation->>Seg3: ReadIndexData()
Seg1-->>Operation: IndexData1
Seg2-->>Operation: IndexData2
Seg3-->>Operation: IndexData3
Operation->>Operation: MergeIndexData([Data1, Data2, Data3])
Operation->>TargetSeg: WriteIndexData(MergedData)
TargetSeg-->>Operation: Success
Operation-->>Merger: Success
Merger->>VersionCommitter: Commit(NewVersion)
VersionCommitter->>VersionCommitter: CreateFence()
VersionCommitter->>VersionCommitter: WriteVersion()
VersionCommitter->>VersionCommitter: AtomicSwitch()
VersionCommitter-->>Merger: VersionMeta
Merger->>Merger: CleanupOldSegments()
Merger-->>Controller: Success
执行流程详解：
CreateMergePlan() 创建合并计划IndexMergeOperation 是合并操作，执行实际的合并工作：
IndexMergeOperation：执行实际的合并工作（已在上面详细展示，此处不再重复）：
合并操作的关键步骤：
IndexMergeOperation 是合并执行的核心，负责实际的合并工作。让我们通过流程图来理解详细的合并过程：
flowchart TD
Start([开始合并操作]) --> Read[读取源Segment<br/>从多个Segment读取数据]
Read --> MergeInverted[合并倒排索引<br/>InvertedIndex]
Read --> MergeAttribute[合并正排索引<br/>AttributeIndex]
Read --> MergePrimaryKey[合并主键索引<br/>PrimaryKeyIndex]
MergeInverted --> MergeDoc[合并文档数据<br/>合并所有索引数据]
MergeAttribute --> MergeDoc
MergePrimaryKey --> MergeDoc
MergeDoc --> Dedup[去重处理<br/>去除重复文档]
Dedup --> Sort[排序处理<br/>按文档ID排序]
Sort --> Write[写入目标Segment<br/>写入合并后的数据]
Write --> UpdateMeta[更新元数据<br/>更新Segment元数据]
UpdateMeta --> End([完成合并])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Read fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style MergeInverted fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style MergeAttribute fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style MergePrimaryKey fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style MergeDoc fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Dedup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Sort fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Write fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style UpdateMeta fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
合并操作的关键步骤详解：
合并操作的性能优化：
OptimizeMergeStrategy 的合并逻辑：
OptimizeMergeStrategy 的合并逻辑：根据参数决定合并行为（已在上面详细展示，此处不再重复）：
合并逻辑：
maxDocCount 过滤，只保留文档数小于等于该值的 SegmentafterMergeMaxDocCount 和 afterMergeMaxSegmentCount 计算目标 Segment 数合并参数对合并行为的影响：
合并参数的影响：maxDocCount、afterMergeMaxDocCount 等参数的作用（已在上面详细展示，此处不再重复）：
参数影响：
不同场景下的合并策略选择：
合并策略的选择：根据场景选择不同的合并策略：
flowchart TB
Start([合并策略选择<br/>Merge Strategy Selection]) --> ScenarioLayer[应用场景层<br/>Application Scenarios Layer]
subgraph ScenarioGroup["应用场景 Application Scenarios"]
direction TB
SC1[全量合并场景<br/>Full Merge Scenario<br/>需要合并所有Segment]
SC2[实时合并场景<br/>Realtime Merge Scenario<br/>实时合并小Segment]
SC3[分片合并场景<br/>Shard Merge Scenario<br/>按分片合并Segment]
SC4[KV表场景<br/>KV Table Scenario<br/>KV表优化合并]
end
ScenarioLayer --> StrategyLayer[合并策略层<br/>Merge Strategy Layer]
subgraph StrategyGroup["合并策略 Merge Strategies"]
direction TB
ST1[OptimizeMergeStrategy<br/>优化合并策略<br/>合并所有符合条件的Segment]
ST2[RealtimeMergeStrategy<br/>实时合并策略<br/>实时合并小Segment]
ST3[ShardBasedMergeStrategy<br/>分片合并策略<br/>按分片合并Segment]
ST4[KeyValueOptimizeMergeStrategy<br/>KV优化合并策略<br/>针对KV表优化]
end
StrategyLayer --> FeatureLayer[策略特点层<br/>Strategy Features Layer]
subgraph FeatureGroup["策略特点 Strategy Features"]
direction TB
F1[合并所有符合条件的Segment<br/>Merge All Eligible Segments<br/>适用于全量合并]
F2[实时合并小Segment<br/>Realtime Merge Small Segments<br/>适用于实时场景]
F3[按分片合并Segment<br/>Merge by Shard<br/>适用于分片场景]
F4[针对KV表优化<br/>KV Table Optimization<br/>适用于KV表]
end
FeatureLayer --> End([策略选择完成<br/>Strategy Selection Complete])
ScenarioLayer -.->|包含| ScenarioGroup
StrategyLayer -.->|包含| StrategyGroup
FeatureLayer -.->|包含| FeatureGroup
SC1 -.->|选择| ST1
SC2 -.->|选择| ST2
SC3 -.->|选择| ST3
SC4 -.->|选择| ST4
ST1 -.->|提供| F1
ST2 -.->|提供| F2
ST3 -.->|提供| F3
ST4 -.->|提供| F4
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style ScenarioLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style StrategyLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FeatureLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ScenarioGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SC1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SC2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SC3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SC4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style StrategyGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style ST1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ST2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ST3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style ST4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FeatureGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style F1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style F4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
策略选择：
合并的触发条件：
合并触发条件：Segment 数量、大小等条件：
flowchart TD
subgraph Conditions["触发条件"]
C1[Segment数量<br/>Segment Count<br/>超过阈值时触发]
C2[Segment大小<br/>Segment Size<br/>小Segment数量过多时触发]
C3[查询性能<br/>Query Performance<br/>查询性能下降时触发]
C4[存储空间<br/>Storage Space<br/>存储空间不足时触发]
C5[手动触发<br/>Manual Trigger<br/>支持手动触发]
end
subgraph Thresholds["阈值设置"]
T1[Segment数量阈值<br/>如100个Segment]
T2[小Segment数量阈值<br/>如50个小Segment]
T3[查询延迟阈值<br/>如P99延迟超过100ms]
T4[存储空间阈值<br/>如使用率超过80%]
end
subgraph Actions["触发动作"]
A1[创建合并计划<br/>Create Merge Plan]
A2[提交合并任务<br/>Submit Merge Task]
A3[执行合并<br/>Execute Merge]
end
C1 --> T1
C2 --> T2
C3 --> T3
C4 --> T4
C5 --> A1
T1 --> A1
T2 --> A1
T3 --> A1
T4 --> A1
A1 --> A2
A2 --> A3
style Conditions fill:#e3f2fd
style Thresholds fill:#fff3e0
style Actions fill:#f3e5f5
触发条件：
合并时机的选择：
合并时机的选择：在线合并、离线合并等：
flowchart TB
Start([合并时机选择<br/>Merge Timing Selection]) --> TimingLayer[合并时机层<br/>Merge Timing Layer]
subgraph TimingGroup["合并时机 Merge Timing"]
direction TB
T1[在线合并<br/>Online Merge<br/>服务运行期间进行]
T2[离线合并<br/>Offline Merge<br/>服务停止时进行]
T3[定时合并<br/>Scheduled Merge<br/>定期触发合并]
T4[按需合并<br/>On-Demand Merge<br/>根据需求触发]
end
TimingLayer --> AdvantageLayer[时机优势层<br/>Timing Advantages Layer]
subgraph AdvantageGroup["时机优势 Timing Advantages"]
direction TB
A1[不影响服务可用性<br/>No Impact on Availability<br/>在线合并的优势]
A2[更彻底优化索引<br/>More Thorough Optimization<br/>离线合并的优势]
A3[保持索引结构优化<br/>Maintain Index Structure<br/>定时合并的优势]
A4[灵活控制合并<br/>Flexible Control<br/>按需合并的优势]
end
AdvantageLayer --> TradeoffLayer[权衡层<br/>Tradeoffs Layer]
subgraph TradeoffGroup["权衡 Tradeoffs"]
direction TB
TR1[可能影响查询性能<br/>May Impact Query Performance<br/>在线合并的权衡]
TR2[需要停止服务<br/>Require Service Stop<br/>离线合并的权衡]
TR3[固定时间触发<br/>Fixed Time Trigger<br/>定时合并的权衡]
TR4[需要手动干预<br/>Require Manual Intervention<br/>按需合并的权衡]
end
TradeoffLayer --> End([时机选择完成<br/>Timing Selection Complete])
TimingLayer -.->|包含| TimingGroup
AdvantageLayer -.->|包含| AdvantageGroup
TradeoffLayer -.->|包含| TradeoffGroup
T1 -.->|提供| A1
T2 -.->|提供| A2
T3 -.->|提供| A3
T4 -.->|提供| A4
T1 -.->|权衡| TR1
T2 -.->|权衡| TR2
T3 -.->|权衡| TR3
T4 -.->|权衡| TR4
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style TimingLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style AdvantageLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style TradeoffLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style TimingGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style T1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style T4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style AdvantageGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style A1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style A2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style A3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style A4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style TradeoffGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style TR1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style TR2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style TR3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style TR4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
合并时机：
合并性能优化的策略：
合并性能优化：并行合并、增量合并等策略：
flowchart TD
subgraph Optimization["优化策略"]
O1[并行合并<br/>Parallel Merge<br/>多个Segment并行合并]
O2[增量合并<br/>Incremental Merge<br/>只合并变更的Segment]
O3[合并优先级<br/>Merge Priority<br/>根据重要性设置优先级]
O4[资源控制<br/>Resource Control<br/>控制CPU、内存、IO资源]
end
subgraph Parallel["并行合并"]
P1[Segment并行<br/>多个Segment并行合并]
P2[索引并行<br/>多个索引并行合并]
P3[IO并行<br/>读取和写入并行进行]
end
subgraph Incremental["增量合并"]
I1[变更检测<br/>只检测变更的Segment]
I2[增量合并<br/>只合并变更的索引]
I3[增量写入<br/>只写入变更的数据]
end
O1 --> P1
O2 --> I1
P1 --> P2
P2 --> P3
I1 --> I2
I2 --> I3
style Optimization fill:#e3f2fd
style Parallel fill:#fff3e0
style Incremental fill:#f3e5f5
优化策略：
合并时的资源控制：
合并的资源控制：CPU、内存、IO 资源的控制：
flowchart TD
subgraph Resources["资源类型"]
R1[CPU控制<br/>CPU Control<br/>限制CPU使用率]
R2[内存控制<br/>Memory Control<br/>限制内存使用]
R3[IO控制<br/>IO Control<br/>限制IO带宽]
R4[并发控制<br/>Concurrency Control<br/>控制并发任务数]
end
subgraph CPU["CPU控制"]
C1[限制CPU使用率<br/>如不超过50%]
C2[避免影响查询性能<br/>Avoid Impact on Query]
C3[动态调整<br/>Dynamic Adjustment]
end
subgraph Memory["内存控制"]
M1[限制内存使用<br/>如不超过总内存的30%]
M2[避免内存溢出<br/>Avoid Memory Overflow]
M3[流式处理<br/>Streaming Processing]
end
subgraph IO["IO控制"]
IO1[限制IO带宽<br/>如不超过总带宽的40%]
IO2[避免影响查询IO<br/>Avoid Impact on Query IO]
IO3[IO优先级<br/>IO Priority]
end
R1 --> C1
R2 --> M1
R3 --> IO1
C1 --> C2
M1 --> M2
IO1 --> IO2
style Resources fill:#e3f2fd
style CPU fill:#fff3e0
style Memory fill:#f3e5f5
style IO fill:#e8f5e9
资源控制：
在全量合并场景中，合并的应用：
全量合并场景：合并所有符合条件的 Segment：
flowchart TD
Start([全量合并开始]) --> Step1[1. 收集所有Segment<br/>从TabletData获取]
Step1 --> Step2[2. 过滤Segment<br/>筛选符合条件的Segment]
Step2 --> Step3[3. 创建合并计划<br/>MergePlan，合并所有Segment]
Step3 --> Step4[4. 执行合并操作<br/>IndexMergeOperation]
Step4 --> Step5[5. 合并索引数据<br/>合并倒排/正排/主键索引]
Step5 --> Step6[6. 创建目标Segment<br/>生成合并后的Segment]
Step6 --> Step7[7. 提交新版本<br/>使用Fence机制保证原子性]
Step7 --> Step8[8. 更新TabletData<br/>切换到新版本]
Step8 --> Step9[9. 清理旧Segment<br/>删除不再需要的文件]
Step9 --> End([全量合并完成])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Step1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Step2 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Step3 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step4 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step5 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step6 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step7 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Step8 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Step9 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
全量合并流程：
在增量合并场景中，合并的应用：
增量合并场景：只合并新增或变更的 Segment：
flowchart TD
subgraph Identify["识别变更"]
I1[识别变更Segment<br/>Identify Changed Segments]
I2[新增Segment<br/>New Segments]
I3[变更Segment<br/>Changed Segments]
I1 --> I2
I1 --> I3
end
subgraph Plan["创建合并计划"]
P1[创建合并计划<br/>Create Merge Plan]
P2[只合并变更Segment<br/>Merge Only Changed Segments]
P3[优化合并范围<br/>Optimize Merge Scope]
P1 --> P2
P2 --> P3
end
subgraph Execute["执行合并"]
E1[执行合并操作<br/>Execute Merge]
E2[合并变更Segment<br/>Merge Changed Segments]
E3[创建目标Segment<br/>Create Target Segments]
E1 --> E2
E2 --> E3
end
subgraph Commit["提交版本"]
CO1[提交新版本<br/>Commit New Version]
CO2[更新TabletData<br/>Update TabletData]
CO3[清理旧Segment<br/>Cleanup Old Segments]
CO1 --> CO2
CO2 --> CO3
end
I2 --> P1
I3 --> P1
P3 --> E1
E3 --> CO1
style Identify fill:#e3f2fd
style Plan fill:#fff3e0
style Execute fill:#f3e5f5
style Commit fill:#e8f5e9
增量合并流程：
合并的原子性保证：
合并的原子性：通过 Fence 机制保证合并的原子性：
flowchart LR
subgraph Fence["Fence 机制"]
F1[创建Fence目录<br/>Create Fence Directory]
F2[写入合并结果<br/>Write Merge Result]
F3[原子重命名<br/>Atomic Rename]
F1 --> F2
F2 --> F3
end
subgraph Atomicity["原子性保证"]
A1[要么全部成功<br/>All or Nothing]
A2[要么全部失败<br/>Rollback on Failure]
A3[避免部分写入<br/>Avoid Partial Write]
A1 --> A2
A2 --> A3
end
subgraph Error["错误处理"]
E1[合并失败<br/>Merge Failure]
E2[清理Fence目录<br/>Cleanup Fence]
E3[不影响已有版本<br/>No Impact on Existing]
E1 --> E2
E2 --> E3
end
subgraph Advantage["设计优势"]
AD1[原子性<br/>Atomicity]
AD2[可靠性<br/>Reliability]
AD3[简单性<br/>Simplicity]
AD4[性能高<br/>High Performance]
end
F3 --> A1
A3 --> E1
E3 --> AD1
style Fence fill:#e3f2fd
style Atomicity fill:#fff3e0
style Error fill:#f3e5f5
style Advantage fill:#e8f5e9
原子性保证：
合并的一致性保证：
合并的一致性：保证合并后数据的一致性：
flowchart TD
subgraph Consistency["一致性保证"]
C1[数据完整性<br/>Data Integrity<br/>保证不丢失数据]
C2[索引一致性<br/>Index Consistency<br/>保证索引结构正确]
C3[版本一致性<br/>Version Consistency<br/>保证版本信息正确]
end
subgraph Data["数据完整性"]
D1[文档去重<br/>Document Deduplication]
D2[数据校验<br/>Data Validation]
D3[完整性检查<br/>Integrity Check]
D1 --> D2
D2 --> D3
end
subgraph Index["索引一致性"]
I1[索引结构正确<br/>Correct Index Structure]
I2[索引数据完整<br/>Complete Index Data]
I3[索引关系正确<br/>Correct Index Relations]
I1 --> I2
I2 --> I3
end
subgraph Version["版本一致性"]
V1[版本号递增<br/>Version ID Increment]
V2[Segment列表正确<br/>Correct Segment List]
V3[Locator信息正确<br/>Correct Locator Info]
V1 --> V2
V2 --> V3
end
C1 --> D1
C2 --> I1
C3 --> V1
style Consistency fill:#e3f2fd
style Data fill:#fff3e0
style Index fill:#f3e5f5
style Version fill:#e8f5e9
一致性保证：
合并的性能优化：
合并的性能优化：并行合并、资源控制等优化策略（已在上面详细展示，此处不再重复）：
性能优化：
优化策略：
优化策略：
监控指标：
Segment 合并策略是 IndexLib 的核心功能，通过 MergeStrategy 和 MergePlan 实现。通过本文的深入解析，我们了解到：
核心机制：
maxDocCount、afterMergeMaxDocCount 等参数控制合并行为设计亮点：
性能优化：
理解 Segment 合并策略，是掌握 IndexLib 索引优化机制的关键。在下一篇文章中，我们将深入介绍内存管理与资源控制的实现细节，包括 MemoryQuotaController、TabletMemoryCalculator、IIndexMemoryReclaimer 等各个组件的实现原理和性能优化策略。
]]>版本管理与增量更新概览：Version 与 Locator 的协同工作：
flowchart TB
Start([版本管理与增量更新<br/>Version Management & Incremental Update]) --> InputLayer[数据输入层<br/>Data Input Layer]
subgraph InputGroup["数据输入 Data Input"]
direction TB
I1[数据源<br/>DataSource<br/>数据来源]
I2[文档流<br/>Document Stream<br/>文档数据流]
I1 --> I2
end
InputLayer --> LocatorLayer[位置信息层<br/>Locator Layer]
subgraph LocatorGroup["位置信息 Locator"]
direction TB
L1[Locator<br/>数据处理位置<br/>记录处理进度]
L2[Timestamp<br/>时间戳<br/>数据时间信息]
L3[MultiProgress<br/>多进度信息<br/>多分片进度]
L4[HashId<br/>分片标识<br/>数据分片ID]
L1 --> L2
L1 --> L3
L3 --> L4
end
LocatorLayer --> UpdateLayer[增量更新层<br/>Incremental Update Layer]
subgraph UpdateGroup["增量更新 Incremental Update"]
direction TB
U1[IsFasterThan<br/>比较判断<br/>判断数据是否已处理]
U2[数据过滤<br/>Filter Processed<br/>过滤已处理数据]
U3[处理新数据<br/>Process New Data<br/>构建新索引]
U4[更新Locator<br/>Update Locator<br/>记录处理进度]
U1 --> U2
U2 --> U3
U3 --> U4
end
UpdateLayer --> VersionLayer[版本管理层<br/>Version Management Layer]
subgraph VersionGroup["版本管理 Version Management"]
direction TB
V1[Version<br/>版本信息<br/>版本元数据]
V2[VersionId<br/>版本号递增<br/>单调递增版本号]
V3[Segments<br/>Segment列表<br/>包含的Segment]
V4[Schema演进<br/>Schema Evolution<br/>SchemaId映射]
V1 --> V2
V1 --> V3
V1 --> V4
end
VersionLayer --> CommitLayer[版本提交层<br/>Version Commit Layer]
subgraph CommitGroup["版本提交 Version Commit"]
direction TB
C1[VersionCommitter<br/>版本提交器<br/>提交新版本]
C2[Fence机制<br/>Fence Mechanism<br/>原子性保证]
C3[持久化<br/>Persistence<br/>写入磁盘]
C1 --> C2
C2 --> C3
end
CommitLayer --> End([版本管理完成<br/>Version Management Complete])
InputLayer -.->|包含| InputGroup
LocatorLayer -.->|包含| LocatorGroup
UpdateLayer -.->|包含| UpdateGroup
VersionLayer -.->|包含| VersionGroup
CommitLayer -.->|包含| CommitGroup
I2 --> L1
L1 --> U1
U4 --> V1
V1 --> C1
C3 --> V3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style InputLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style LocatorLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style UpdateLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style VersionLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style CommitLayer fill:#fce4ec,stroke:#ef4444,stroke-width:3px
style InputGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style I1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style I2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style LocatorGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style L1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style UpdateGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style U1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style U2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style U3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style U4 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style VersionGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style V1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CommitGroup fill:#fce4ec,stroke:#ef4444,stroke-width:3px
style C1 fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
style C2 fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
style C3 fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
IndexLib 的版本管理包括以下核心概念：
让我们先通过图来理解版本管理的整体架构：
版本管理架构：Version、Locator、Segment 的关系：
flowchart TB
Start([版本管理架构<br/>Version Management Architecture]) --> VersionLayer[版本信息层<br/>Version Information Layer]
subgraph VersionGroup["Version 版本信息 Version Information"]
direction TB
V1[VersionId<br/>版本号单调递增<br/>每次Commit递增]
V2[Segments<br/>Segment列表<br/>包含的Segment集合]
V3[Locator<br/>位置信息<br/>数据处理位置]
V4[Timestamp<br/>时间戳<br/>版本创建时间]
V5[Sealed<br/>封存状态<br/>是否封存]
V6[SchemaId<br/>Schema标识<br/>当前Schema版本]
V1 --> V2
V1 --> V3
V1 --> V4
V1 --> V5
V1 --> V6
end
VersionLayer --> SegmentLayer[索引段层<br/>Segment Layer]
subgraph SegmentGroup["Segment 索引段 Segment"]
direction TB
S1[SegmentId<br/>段标识<br/>唯一标识Segment]
S2[SchemaId<br/>段Schema<br/>Segment的Schema版本]
S3[IndexFiles<br/>索引文件<br/>索引数据文件]
S4[SegmentInfo<br/>段信息<br/>Segment元数据]
S1 --> S2
S1 --> S3
S1 --> S4
end
SegmentLayer --> LocatorLayer[位置信息层<br/>Locator Layer]
subgraph LocatorGroup["Locator 位置信息 Locator"]
direction TB
L1[SourceId<br/>数据源标识<br/>数据来源ID]
L2[Timestamp<br/>时间戳<br/>数据时间信息]
L3[ConcurrentIdx<br/>并发索引<br/>并发处理索引]
L4[HashId<br/>分片标识<br/>数据分片ID]
L5[MultiProgress<br/>多进度信息<br/>多分片处理进度]
L1 --> L2
L2 --> L3
L3 --> L4
L4 --> L5
end
LocatorLayer --> CommitLayer[版本提交层<br/>Version Commit Layer]
subgraph CommitGroup["版本提交 Version Commit"]
direction TB
C1[VersionCommitter<br/>版本提交器<br/>提交新版本]
C2[Fence目录<br/>Fence Directory<br/>临时目录保证原子性]
C3[原子切换<br/>Atomic Switch<br/>重命名操作]
C4[持久化<br/>Persistence<br/>写入磁盘]
C1 --> C2
C2 --> C3
C3 --> C4
end
CommitLayer --> End([版本管理完成<br/>Version Management Complete])
VersionLayer -.->|包含| VersionGroup
SegmentLayer -.->|包含| SegmentGroup
LocatorLayer -.->|包含| LocatorGroup
CommitLayer -.->|包含| CommitGroup
V2 -.->|包含| S1
V3 -.->|包含| L1
V1 -.->|提交| C1
C4 -.->|更新| V1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style VersionLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SegmentLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style LocatorLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style CommitLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style VersionGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style V1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V6 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SegmentGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style LocatorGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style L1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style L5 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style CommitGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style C1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C4 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
版本管理在 IndexLib 中起到关键作用，是系统稳定性和数据一致性的基础。让我们通过类图来理解版本管理的整体架构：
classDiagram
class Version {
- versionid_t _versionId
- vector_SegmentInVersion _segments
- Locator _locator
- int64_t _timestamp
- bool _sealed
+ GetVersionId()
+ AddSegment()
+ SetLocator()
+ IncVersionId()
}
class Locator {
- uint64_t _src
- MultiProgress _multiProgress
- string _userData
+ IsFasterThan()
+ Update()
+ Serialize()
}
class VersionCommitter {
+ Commit()
+ CreateFence()
+ WriteVersion()
}
class VersionLoader {
+ Load()
+ Validate()
}
Version --> Locator : 包含
VersionCommitter --> Version : 创建
VersionLoader --> Version : 加载
版本管理的核心作用：
sourceIdx 区分数据源，支持多数据源场景Version 记录索引的版本信息，定义在 framework/Version.h 中：
// framework/Version.h
class Version : public autil::legacy::Jsonizable
{
private:
struct SegmentInVersion {
segmentid_t segmentId = INVALID_SEGMENTID;
schemaid_t schemaId = DEFAULT_SCHEMAID; // 每个 Segment 可以有不同的 Schema
};
public:
// 版本信息
versionid_t GetVersionId() const { return _versionId; }
void IncVersionId() { ++_versionId; } // 每次 Commit 时递增
// Segment 管理
void AddSegment(segmentid_t segmentId, schemaid_t schemaId);
void RemoveSegment(segmentid_t segmentId);
size_t GetSegmentCount() const { return _segments.size(); }
// Locator：数据位置信息
void SetLocator(const Locator& locator);
const Locator& GetLocator() const { return _locator; }
// 时间戳
void SetTimestamp(int64_t timestamp) { _timestamp = timestamp; }
int64_t GetTimestamp() const { return _timestamp; }
// 封存状态
void SetSealed() { _sealed = true; }
bool IsSealed() const { return _sealed; }
private:
versionid_t _versionId; // 版本号，单调递增
std::vector<SegmentInVersion> _segments; // Segment 列表（有序）
Locator _locator; // 位置信息，用于增量更新
int64_t _timestamp; // 时间戳
bool _sealed = false; // 是否封存
schemaid_t _schemaId; // Schema ID
std::string _fenceName; // Fence 名称
};
Version 的关键字段：
Version 的结构：包含 VersionId、Segments、Locator 等关键信息：
flowchart TD
subgraph Version["Version 对象"]
V[Version<br/>版本信息]
end
subgraph Fields["核心字段"]
F1[VersionId<br/>版本号<br/>单调递增]
F2[Segments<br/>Segment列表<br/>vector SegmentInVersion]
F3[Locator<br/>位置信息<br/>用于增量更新]
F4[Timestamp<br/>时间戳<br/>版本创建时间]
F5[Sealed<br/>封存状态<br/>是否封存]
F6[SchemaId<br/>Schema标识<br/>当前Schema版本]
F7[FenceName<br/>Fence名称<br/>临时目录名]
end
subgraph SegmentInVersion["SegmentInVersion 结构"]
S1[SegmentId<br/>段标识<br/>INVALID_SEGMENTID]
S2[SchemaId<br/>段Schema<br/>DEFAULT_SCHEMAID]
S1 --> S2
end
V --> F1
V --> F2
V --> F3
V --> F4
V --> F5
V --> F6
V --> F7
F2 -->|包含| S1
style Version fill:#e3f2fd
style Fields fill:#fff3e0
style SegmentInVersion fill:#f3e5f5
每次 Commit 都会创建新版本，版本号递增：
Version 演进：从 V1 到 V2 的版本变化：
flowchart TB
Start([Version 演进流程<br/>Version Evolution Flow]) --> V1Layer[Version 1 层<br/>Version 1 Layer]
subgraph V1Group["Version 1 版本信息"]
direction TB
V1_ID[VersionId: 1<br/>版本号1]
V1_SEG[Segments: 1, 2<br/>包含Segment 1和2]
V1_LOC[Locator: timestamp=100<br/>处理到时间戳100]
V1_SCHEMA[SchemaId: 0<br/>Schema版本0]
V1_ID --> V1_SEG
V1_ID --> V1_LOC
V1_ID --> V1_SCHEMA
end
V1Layer --> CommitLayer[提交操作层<br/>Commit Operation Layer]
subgraph CommitGroup["Commit 操作 Commit Operation"]
direction TB
C1[收集Segment<br/>Collect Segments<br/>收集所有Segment]
C2[更新Locator<br/>Update Locator<br/>更新处理位置]
C3[递增VersionId<br/>Increment VersionId<br/>版本号递增]
C4[创建新版本<br/>Create New Version<br/>创建Version 2]
C1 --> C2
C2 --> C3
C3 --> C4
end
CommitLayer --> V2Layer[Version 2 层<br/>Version 2 Layer]
subgraph V2Group["Version 2 版本信息"]
direction TB
V2_ID[VersionId: 2<br/>版本号递增为2]
V2_SEG[Segments: 1, 2, 3<br/>新增Segment 3]
V2_LOC[Locator: timestamp=200<br/>处理到时间戳200]
V2_SCHEMA[SchemaId: 0<br/>Schema版本0保持不变]
V2_ID --> V2_SEG
V2_ID --> V2_LOC
V2_ID --> V2_SCHEMA
end
V2Layer --> MergeLayer[合并操作层<br/>Merge Operation Layer]
subgraph MergeGroup["合并操作 Merge Operation"]
direction TB
M1[合并Segment 1和2<br/>Merge Segments 1 and 2<br/>合并为Segment 4]
end
MergeLayer --> V3Layer[Version 3 层<br/>Version 3 Layer]
subgraph V3Group["Version 3 版本信息"]
direction TB
V3_ID[VersionId: 3<br/>版本号递增为3]
V3_SEG[Segments: 4<br/>合并后的Segment 4]
V3_LOC[Locator: timestamp=300<br/>处理到时间戳300]
V3_SCHEMA[SchemaId: 0<br/>Schema版本0保持不变]
V3_ID --> V3_SEG
V3_ID --> V3_LOC
V3_ID --> V3_SCHEMA
end
V3Layer --> End([版本演进完成<br/>Version Evolution Complete])
V1Layer -.->|包含| V1Group
CommitLayer -.->|包含| CommitGroup
V2Layer -.->|包含| V2Group
MergeLayer -.->|包含| MergeGroup
V3Layer -.->|包含| V3Group
V1Group -.->|提交| CommitGroup
CommitGroup -.->|创建| V2Group
V2Group -.->|合并| MergeGroup
MergeGroup -.->|创建| V3Group
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style V1Layer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CommitLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style V2Layer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style MergeLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style V3Layer fill:#fce4ec,stroke:#ef4444,stroke-width:3px
style V1Group fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style V1_ID fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V1_SEG fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V1_LOC fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V1_SCHEMA fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CommitGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V2Group fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style V2_ID fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style V2_SEG fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style V2_LOC fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style V2_SCHEMA fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style MergeGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style M1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style V3Group fill:#fce4ec,stroke:#ef4444,stroke-width:3px
style V3_ID fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
style V3_SEG fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
style V3_LOC fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
style V3_SCHEMA fill:#f8bbd0,stroke:#ef4444,stroke-width:2px
版本演进示例：
版本演进的关键设计：
版本演进是 IndexLib 版本管理的核心机制。让我们通过序列图来理解版本演进的完整过程：
sequenceDiagram
participant Writer as TabletWriter
participant MemSeg as MemSegment
participant DiskSeg as DiskSegment
participant Version as Version
participant VersionCommitter as VersionCommitter
participant TabletData as TabletData
Writer->>MemSeg: Build(documents)
MemSeg-->>Writer: Success
Writer->>MemSeg: NeedDump()?
MemSeg-->>Writer: true
Writer->>MemSeg: CreateSegmentDumpItems()
MemSeg-->>Writer: DumpItems
Writer->>DiskSeg: Dump(DumpItems)
DiskSeg-->>Writer: Success
Writer->>Version: AddSegment(segmentId, schemaId)
Version->>Version: IncVersionId()
Version-->>Writer: newVersionId
Writer->>Version: SetLocator(locator)
Version-->>Writer: Success
Writer->>VersionCommitter: Commit(TabletData, Schema, Options)
VersionCommitter->>VersionCommitter: CreateFence()
VersionCommitter->>VersionCommitter: WriteVersion(Version)
VersionCommitter->>VersionCommitter: AtomicSwitch()
VersionCommitter-->>Writer: VersionMeta
Writer->>TabletData: UpdateVersion(Version)
TabletData-->>Writer: Success
版本演进的关键设计：
VersionId 自动递增，保证版本顺序
SchemaId，支持 Schema 变更
Version 需要持久化到磁盘，通过 Fence 机制保证原子性：
Version 持久化：通过 Fence 机制保证原子性：
flowchart TD
Start([Version 持久化开始]) --> P1
subgraph Prepare["1. 准备阶段"]
direction LR
P1[PrepareVersion<br/>准备版本信息]
P2[CollectSegments<br/>收集Segment列表]
P3[PrepareLocator<br/>准备Locator信息]
P1 --> P2 --> P3
end
P3 --> F1
subgraph Fence["2. Fence 机制"]
direction LR
F1[CreateFenceDirectory<br/>创建临时目录]
F2[WriteVersionFile<br/>写入版本文件]
F3[IncVersionId<br/>递增版本号]
F4[AtomicRename<br/>原子重命名]
F1 --> F2 --> F3 --> F4
end
F4 --> PE1
subgraph Persist["3. 持久化"]
direction LR
PE1[序列化Version<br/>JSON格式]
PE2[写入version文件<br/>version.0]
PE3[写入元数据<br/>时间戳、Locator]
PE1 --> PE2 --> PE3
end
PE3 --> U1
subgraph Update["4. 更新内存"]
direction LR
U1[UpdateTabletData<br/>更新TabletData]
U2[更新Version引用<br/>切换到新版本]
U1 --> U2
end
U2 --> Success([持久化成功])
F2 -.->|异常| Error
F4 -.->|异常| Error
PE2 -.->|异常| Error
U1 -.->|异常| Error
subgraph Error["错误处理"]
direction TB
E1[CleanupFence<br/>清理临时目录]
E2[不影响已有版本<br/>保证一致性]
E1 --> E2
end
Error --> E1
E2 --> Fail([持久化失败])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Prepare fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style P1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style P3 fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style Fence fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style F2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style F3 fill:#ffb74d,stroke:#f57c00,stroke-width:1.5px
style F4 fill:#ffa726,stroke:#f57c00,stroke-width:1.5px
style Persist fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style PE1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style PE2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style PE3 fill:#ba68c8,stroke:#7b1fa2,stroke-width:1.5px
style Update fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style U1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style U2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
style Error fill:#fce4ec,stroke:#c2185b,stroke-width:2px
style E1 fill:#f8bbd0,stroke:#c2185b,stroke-width:1.5px
style E2 fill:#f48fb1,stroke:#c2185b,stroke-width:1.5px
style Success fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
style Fail fill:#ffcdd2,stroke:#c62828,stroke-width:3px
持久化流程：
Version 的持久化是版本管理的核心，通过 Fence 机制保证原子性。让我们通过序列图来理解完整的持久化流程：
sequenceDiagram
participant Committer as VersionCommitter
participant Version as Version
participant FileSys as FileSystem
participant TabletData as TabletData
Committer->>Version: PrepareVersion()
Version->>Version: CollectSegments()
Version->>Version: PrepareLocator()
Version-->>Committer: Version对象
Committer->>FileSys: CreateFenceDirectory()
FileSys-->>Committer: FencePath
Committer->>FileSys: WriteVersionFile(Version, FencePath)
FileSys->>FileSys: 序列化Version为JSON
FileSys->>FileSys: 写入version文件
FileSys-->>Committer: Success
Committer->>Version: IncVersionId()
Version-->>Committer: newVersionId
Committer->>FileSys: AtomicRename(FencePath, VersionPath)
FileSys-->>Committer: Success
Committer->>TabletData: UpdateVersion(Version)
TabletData-->>Committer: Success
alt 提交失败
Committer->>FileSys: CleanupFence(FencePath)
FileSys-->>Committer: Success
end
持久化流程详解：
version.fence.1234567890）version.0）rename）Fence 机制的设计优势：
Version 的加载通过 VersionLoader 实现：
Version 加载：从磁盘加载版本信息：
flowchart TB
Start([Version 加载开始<br/>Version Load Start]) --> LoadLayer[加载阶段<br/>Load Phase]
subgraph LoadGroup["1. 加载阶段 Load Phase"]
direction TB
L1[VersionLoader.Load<br/>加载版本信息<br/>从磁盘读取]
L2[读取版本文件<br/>version.0, version.1等<br/>按版本号顺序<br/>找到最新版本]
L3[解析JSON<br/>反序列化Version对象<br/>转换为内存结构]
L1 --> L2
L2 --> L3
end
LoadLayer --> ValidateLayer[验证阶段<br/>Validate Phase]
subgraph ValidateGroup["2. 验证阶段 Validate Phase"]
direction TB
V1[ValidateVersion<br/>验证版本有效性<br/>检查基本格式]
V2[检查Segment存在性<br/>Segment文件是否存在<br/>验证文件完整性]
V3[检查Schema兼容性<br/>Schema版本映射检查<br/>确保兼容性]
V4[检查Locator有效性<br/>Locator格式正确性<br/>验证数据一致性]
V1 --> V2
V2 --> V3
V3 --> V4
end
ValidateLayer --> SegmentLayer[加载Segment阶段<br/>Load Segment Phase]
subgraph SegmentGroup["3. 加载Segment Load Segment"]
direction TB
S1[根据Segment列表<br/>加载Segment信息<br/>遍历所有Segment]
S2[OpenSegment<br/>打开Segment文件<br/>初始化文件句柄]
S3[加载索引文件<br/>按需加载索引数据<br/>延迟加载策略]
S4[创建DiskSegment<br/>DiskSegment对象<br/>封装Segment信息]
S1 --> S2
S2 --> S3
S3 --> S4
end
SegmentLayer --> InitLayer[初始化TabletData阶段<br/>Initialize TabletData Phase]
subgraph InitGroup["4. 初始化TabletData Initialize TabletData"]
direction TB
I1[UpdateVersion<br/>更新Version引用<br/>设置当前版本]
I2[设置Segment列表<br/>添加到TabletData<br/>建立索引关系]
I3[初始化查询器<br/>准备查询功能<br/>创建Reader对象]
I1 --> I2
I2 --> I3
end
InitLayer --> End([完成加载<br/>Load Complete])
LoadLayer -.->|包含| LoadGroup
ValidateLayer -.->|包含| ValidateGroup
SegmentLayer -.->|包含| SegmentGroup
InitLayer -.->|包含| InitGroup
L3 --> V1
V4 --> S1
S4 --> I1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style LoadLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ValidateLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style SegmentLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style InitLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style LoadGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style L1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style ValidateGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style V1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style SegmentGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style S1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style InitGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style I1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style I2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style I3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
加载流程：
Locator 是增量更新的核心，记录数据的位置信息：
Locator 的作用：记录数据处理位置，支持增量更新：
flowchart LR
subgraph Role["Locator 核心作用"]
R1[增量更新<br/>判断数据是否已处理]
R2[数据一致性<br/>保证不重复不丢失]
R3[进度追踪<br/>记录每个HashId进度]
R4[并发控制<br/>处理时间戳相同情况]
end
subgraph Compare["比较机制"]
C1[IsFasterThan<br/>比较两个Locator]
C2[LCR_FULLY_FASTER<br/>完全更快]
C3[LCR_SLOWER<br/>更慢]
C4[LCR_PARTIAL_FASTER<br/>部分更快]
C5[LCR_INVALID<br/>无效比较]
C1 --> C2
C1 --> C3
C1 --> C4
C1 --> C5
end
subgraph Update["更新机制"]
U1[Update<br/>更新Locator]
U2[条件检查<br/>新Locator必须更快]
U3[更新MultiProgress<br/>记录最新进度]
U4[保证一致性<br/>只向前推进]
U1 --> U2
U2 --> U3
U3 --> U4
end
subgraph Application["应用场景"]
A1[实时写入<br/>实时接收数据流]
A2[批量更新<br/>批量处理数据]
A3[多数据源<br/>支持多数据源场景]
A4[故障恢复<br/>故障恢复时判断]
end
R1 --> C1
R2 --> U1
R3 --> A1
R4 --> A2
C2 --> A3
U4 --> A4
style Role fill:#e3f2fd
style Compare fill:#fff3e0
style Update fill:#f3e5f5
style Application fill:#e8f5e9
Locator 的关键作用：
IsFasterThan() 判断哪些数据已处理，避免重复处理concurrentIdx 处理时间戳相同的情况Locator 的结构定义在 framework/Locator.h 中：
// framework/Locator.h
class Locator final
{
public:
// Locator 比较结果
enum class LocatorCompareResult {
LCR_INVALID, // 无效
LCR_SLOWER, // 比这个 locator 慢
LCR_PARTIAL_FASTER, // 部分 hash id 更快
LCR_FULLY_FASTER // 完全比这个 locator 快（包括相等）
};
// 文档信息：记录文档在数据源中的位置
struct DocInfo {
int64_t timestamp; // 时间戳
uint32_t concurrentIdx; // 并发索引（时间戳相同时的序号）
uint16_t hashId; // Hash ID（用于分片）
uint8_t sourceIdx; // 数据源索引
};
// 比较两个 Locator：判断数据是否已处理
LocatorCompareResult IsFasterThan(const Locator& other,
bool ignoreLegacyDiffSrc) const;
private:
uint64_t _src; // 数据源标识
base::Progress::Offset _minOffset; // 最小偏移量
base::MultiProgress _multiProgress; // 多进度信息（每个 hashId 的进度）
std::string _userData; // 用户数据
};
Locator 的关键字段：
Locator 的结构：包含 timestamp、concurrentIdx、hashId 等信息：
flowchart TD
Locator[Locator 对象<br/>━━━━━━━━━━<br/>位置信息<br/>记录数据处理进度] --> Fields
subgraph Fields["核心字段"]
direction LR
F1["SourceId<br/>━━━━━━━━━━<br/>数据源标识<br/>uint64_t _src"]
F2["MinOffset<br/>━━━━━━━━━━<br/>最小偏移量<br/>Progress::Offset"]
F3["MultiProgress<br/>━━━━━━━━━━<br/>多进度信息<br/>每个HashId的进度"]
F4["UserData<br/>━━━━━━━━━━<br/>用户数据<br/>string _userData"]
end
Locator --> F1
Locator --> F2
Locator --> F3
Locator --> F4
F3 --> Progress
F2 --> Progress
subgraph Progress["Progress 进度信息<br/>用于MultiProgress和MinOffset"]
direction LR
P1["Offset<br/>━━━━━━━━━━<br/>偏移量<br/>包含时间信息"]
P2["Timestamp<br/>━━━━━━━━━━<br/>时间戳<br/>int64_t"]
P3["ConcurrentIdx<br/>━━━━━━━━━━<br/>并发索引<br/>uint32_t"]
P1 --> P2 --> P3
end
Progress --> P1
F3 --> DocInfo
subgraph DocInfo["DocInfo 文档信息<br/>用于构建Progress"]
direction LR
D1["Timestamp<br/>━━━━━━━━━━<br/>时间戳<br/>int64_t"]
D2["ConcurrentIdx<br/>━━━━━━━━━━<br/>并发索引<br/>uint32_t"]
D3["HashId<br/>━━━━━━━━━━<br/>分片标识<br/>uint16_t"]
D4["SourceIdx<br/>━━━━━━━━━━<br/>数据源索引<br/>uint8_t"]
D1 --> D2 --> D3 --> D4
end
DocInfo --> D1
style Locator fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Fields fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style F2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style F3 fill:#ffb74d,stroke:#f57c00,stroke-width:1.5px
style F4 fill:#ffa726,stroke:#f57c00,stroke-width:1.5px
style Progress fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style P1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style P2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
style P3 fill:#81c784,stroke:#2e7d32,stroke-width:1.5px
style DocInfo fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style D1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style D2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style D3 fill:#ba68c8,stroke:#7b1fa2,stroke-width:1.5px
style D4 fill:#ab47bc,stroke:#7b1fa2,stroke-width:1.5px
Locator 的比较逻辑用于判断数据是否已处理：
Locator 比较：判断数据是否已处理的逻辑（已在上面详细展示，此处不再重复）：
比较示例：
LCR_FULLY_FASTER），说明 B 包含 A 的所有数据比较逻辑：
Locator 的比较逻辑是增量更新的核心算法。让我们通过流程图来理解详细的比较过程：
flowchart TD
Start([IsFasterThan 调用]) --> CheckSource{数据源是否相同?}
CheckSource -->|否| Invalid[返回 LCR_INVALID<br/>数据源不同，无法比较]
CheckSource -->|是| Loop[遍历 multiProgress<br/>遍历所有 hashId]
Loop --> CheckHashId{当前 hashId<br/>是否存在?}
CheckHashId -->|不存在| CheckOther{other 中<br/>是否存在?}
CheckHashId -->|存在| Compare[比较 Progress<br/>CompareProgress方法]
CheckOther -->|存在| Partial[返回 LCR_PARTIAL_FASTER<br/>部分更快]
CheckOther -->|不存在| Next[继续下一个 hashId]
Compare --> CheckResult{比较结果}
CheckResult -->|LCR_FULLY_FASTER| Next
CheckResult -->|LCR_SLOWER| Return[返回 LCR_SLOWER<br/>更慢]
CheckResult -->|LCR_PARTIAL_FASTER| Return2[返回 LCR_PARTIAL_FASTER<br/>部分更快]
Next --> CheckComplete{是否遍历完<br/>所有 hashId?}
CheckComplete -->|否| Loop
CheckComplete -->|是| Full[返回 LCR_FULLY_FASTER<br/>完全更快]
Invalid --> End([结束])
Partial --> End
Return --> End
Return2 --> End
Full --> End
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CheckSource fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Invalid fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Loop fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckHashId fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckOther fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Compare fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Partial fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Next fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CheckResult fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Return fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Return2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CheckComplete fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Full fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
比较逻辑详解：
// framework/Locator.h
LocatorCompareResult Locator::IsFasterThan(const Locator& other,
bool ignoreLegacyDiffSrc) const
{
// 1. 检查数据源是否相同
if (!IsSameSrc(other, ignoreLegacyDiffSrc)) {
return LCR_INVALID; // 数据源不同，无法比较
}
// 2. 比较每个 hashId 的进度
for (size_t i = 0; i < _multiProgress.size(); ++i) {
if (i >= other._multiProgress.size()) {
// 当前 Locator 有更多的 hashId，部分更快
return LCR_PARTIAL_FASTER;
}
// 比较该 hashId 的进度
auto result = CompareProgress(_multiProgress[i], other._multiProgress[i]);
if (result != LCR_FULLY_FASTER) {
// 如果该 hashId 不是完全更快，返回结果
return result;
}
}
// 3. 所有 hashId 都完全更快，返回完全更快
return LCR_FULLY_FASTER;
}
比较算法的性能优化：
LCR_INVALID，避免遍历 ProgressLocator 的更新通过 Update() 方法实现：
Locator 更新：更新数据处理位置：
flowchart TD
Start([Locator 更新开始]) --> Input[接收输入<br/>新 Locator + 当前 Locator]
Input --> Compare[IsFasterThan 比较<br/>判断新Locator是否完全更快]
Compare --> Decision{是否完全更快?<br/>LCR_FULLY_FASTER}
Decision -->|否| Fail[更新失败<br/>保持原Locator不变]
Decision -->|是| Step1[1. 更新 MultiProgress<br/>合并每个HashId的进度信息]
Step1 --> Step2[2. 更新 MinOffset<br/>取最小偏移量]
Step2 --> Step3[3. 更新 UserData<br/>保留用户自定义数据]
Step3 --> Step4[4. 保证一致性<br/>确保只向前推进，不后退]
Step4 --> Success[更新成功<br/>Locator已更新]
Fail --> End([结束])
Success --> End
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Input fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Compare fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Decision fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style Step1 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:2px
style Step3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Step4 fill:#ba68c8,stroke:#7b1fa2,stroke-width:2px
style Success fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style Fail fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
更新逻辑：
multiProgress，记录最新的数据处理位置增量更新通过 Locator 判断哪些数据已处理：
增量更新流程：通过 Locator 判断数据是否已处理（已在上面详细展示，此处不再重复）：
增量更新流程图：
graph TD
A[读取数据源] --> B[获取数据 Locator]
B --> C[比较 Locator]
C --> D{IsFasterThan?}
D -->|LCR_FULLY_FASTER| E[数据已处理]
D -->|LCR_SLOWER| F[处理新数据]
D -->|LCR_PARTIAL_FASTER| G[部分处理]
E --> H[跳过数据]
F --> I[构建索引]
G --> I
I --> J[更新 Locator]
J --> K[提交版本]
K --> L[更新 Version Locator]
style C fill:#e3f2fd
style F fill:#fff3e0
style J fill:#f3e5f5
style K fill:#e8f5e9
增量更新流程：
IsFasterThan() 判断数据是否已处理增量更新的判断通过 Locator 比较实现：
增量更新判断：通过 Locator 比较判断数据是否已处理：
flowchart TD
subgraph Input["输入"]
I1[数据Locator<br/>Data Locator]
I2[版本Locator<br/>Version Locator]
end
subgraph Compare["比较判断"]
C1[IsFasterThan<br/>比较两个Locator]
C2{比较结果}
end
subgraph Result["判断结果"]
R1[LCR_FULLY_FASTER<br/>数据已处理<br/>跳过数据]
R2[LCR_SLOWER<br/>数据未处理<br/>需要处理]
R3[LCR_PARTIAL_FASTER<br/>部分数据已处理<br/>需要部分处理]
R4[LCR_INVALID<br/>数据源不同<br/>无法比较]
end
subgraph Action["处理动作"]
A1[跳过数据<br/>不处理]
A2[处理新数据<br/>构建索引]
A3[部分处理<br/>处理未处理部分]
A4[无法判断<br/>需要人工处理]
end
I1 --> C1
I2 --> C1
C1 --> C2
C2 -->|完全更快| R1
C2 -->|更慢| R2
C2 -->|部分更快| R3
C2 -->|无效| R4
R1 --> A1
R2 --> A2
R3 --> A3
R4 --> A4
style Input fill:#e3f2fd
style Compare fill:#fff3e0
style Result fill:#f3e5f5
style Action fill:#e8f5e9
判断逻辑：
增量更新适用于以下场景：
增量更新场景：实时写入、批量更新等：
flowchart TD
subgraph Realtime["1. 实时写入场景"]
direction LR
R1[实时接收数据流<br/>━━━━━━━━━━<br/>Continuous Data Stream<br/>持续接收新数据]
R2[检查Locator<br/>━━━━━━━━━━<br/>IsFasterThan判断<br/>判断是否需要处理]
R3[处理新数据<br/>━━━━━━━━━━<br/>只处理未处理数据<br/>避免重复处理]
R4[更新Locator<br/>━━━━━━━━━━<br/>记录最新进度<br/>更新处理位置]
R5[定期Commit<br/>━━━━━━━━━━<br/>提交版本<br/>持久化进度]
R1 --> R2 --> R3 --> R4 --> R5
end
subgraph Batch["2. 批量更新场景"]
direction LR
B1[批量读取数据源<br/>━━━━━━━━━━<br/>Batch Read<br/>一次性读取大量数据]
B2[检查Locator<br/>━━━━━━━━━━<br/>过滤已处理数据<br/>跳过已处理部分]
B3[处理新数据<br/>━━━━━━━━━━<br/>批量构建索引<br/>高效处理]
B4[更新Locator<br/>━━━━━━━━━━<br/>更新进度<br/>记录处理位置]
B5[批量Commit<br/>━━━━━━━━━━<br/>提交版本<br/>批量持久化]
B1 --> B2 --> B3 --> B4 --> B5
end
subgraph MultiSource["3. 多数据源场景"]
direction LR
M1[多个数据源<br/>━━━━━━━━━━<br/>Multiple Data Sources<br/>来自不同来源]
M2[区分SourceIdx<br/>━━━━━━━━━━<br/>区分数据源<br/>标识来源]
M3[分别处理<br/>━━━━━━━━━━<br/>独立处理每个数据源<br/>独立进度跟踪]
M4[保证一致性<br/>━━━━━━━━━━<br/>数据不重复不丢失<br/>确保完整性]
M1 --> M2 --> M3 --> M4
end
subgraph Recovery["4. 故障恢复场景"]
direction LR
F1[故障恢复<br/>━━━━━━━━━━<br/>Failure Recovery<br/>系统重启或恢复]
F2[检查Locator<br/>━━━━━━━━━━<br/>判断需要重新处理的数据<br/>定位断点]
F3[重新处理<br/>━━━━━━━━━━<br/>处理未处理数据<br/>从断点继续]
F4[恢复完成<br/>━━━━━━━━━━<br/>恢复正常状态<br/>继续正常运行]
F1 --> F2 --> F3 --> F4
end
style Realtime fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style R1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1.5px
style R2 fill:#90caf9,stroke:#1976d2,stroke-width:1.5px
style R3 fill:#64b5f6,stroke:#1976d2,stroke-width:1.5px
style R4 fill:#42a5f5,stroke:#1976d2,stroke-width:1.5px
style R5 fill:#2196f3,stroke:#1976d2,stroke-width:1.5px
style Batch fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style B1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style B2 fill:#ffcc80,stroke:#f57c00,stroke-width:1.5px
style B3 fill:#ffb74d,stroke:#f57c00,stroke-width:1.5px
style B4 fill:#ffa726,stroke:#f57c00,stroke-width:1.5px
style B5 fill:#ff9800,stroke:#f57c00,stroke-width:1.5px
style MultiSource fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style M1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style M2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style M3 fill:#ba68c8,stroke:#7b1fa2,stroke-width:1.5px
style M4 fill:#ab47bc,stroke:#7b1fa2,stroke-width:1.5px
style Recovery fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style F1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style F2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
style F3 fill:#81c784,stroke:#2e7d32,stroke-width:1.5px
style F4 fill:#66bb6a,stroke:#2e7d32,stroke-width:1.5px
使用场景：
版本提交通过 VersionCommitter 实现：
版本提交流程：从准备到持久化的完整过程（已在上面详细展示，此处不再重复）：
版本提交流程图：
flowchart TD
Start([版本提交开始]) --> Check[检查提交条件<br/>━━━━━━━━━━<br/>判断是否有新Segment<br/>是否有数据变更]
Check --> Decision{需要提交?}
Decision -->|否| Skip[跳过提交<br/>━━━━━━━━━━<br/>无变更，无需提交<br/>保持当前版本]
Decision -->|是| Prepare[准备版本信息<br/>━━━━━━━━━━<br/>收集版本元数据<br/>准备提交内容]
Prepare --> Collect[收集 Segment<br/>━━━━━━━━━━<br/>收集所有已构建Segment<br/>构建Segment列表]
Collect --> Locator[准备 Locator<br/>━━━━━━━━━━<br/>准备位置信息<br/>记录处理进度]
Locator --> Fence[创建 Fence<br/>━━━━━━━━━━<br/>创建临时目录<br/>保证原子性]
Fence --> Write[写入版本文件<br/>━━━━━━━━━━<br/>序列化Version<br/>写入JSON文件]
Write --> Update[更新版本号<br/>━━━━━━━━━━<br/>递增版本ID<br/>生成新版本号]
Update --> Persist[持久化到磁盘<br/>━━━━━━━━━━<br/>原子重命名<br/>完成持久化]
Persist --> Success[完成提交<br/>━━━━━━━━━━<br/>版本提交成功<br/>更新TabletData]
Skip --> End([结束])
Success --> End
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Check fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Decision fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Skip fill:#ffcdd2,stroke:#c62828,stroke-width:2px
style Prepare fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Collect fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Locator fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Fence fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Write fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Update fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Persist fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Success fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
提交流程：
版本加载通过 VersionLoader 实现：
版本加载流程：从磁盘加载版本信息（已在上面详细展示，此处不再重复）：
加载流程：
版本回滚支持回滚到历史版本：
flowchart TD
Start([版本回滚开始]) --> Step1[1. 选择目标版本<br/>指定要回滚的版本号]
Step1 --> Step2[2. 验证版本<br/>检查版本文件和Segment存在性]
Step2 --> Step3[3. 加载Version<br/>从磁盘读取版本信息]
Step3 --> Step4[4. 加载Segment列表<br/>读取Segment元数据]
Step4 --> Step5[5. 加载Locator<br/>读取位置信息]
Step5 --> Step6[6. 加载Schema映射<br/>读取Schema版本映射]
Step6 --> Step7[7. 更新TabletData<br/>切换到目标版本]
Step7 --> Step8[8. 设置Version引用<br/>设置当前版本]
Step8 --> Step9[9. 设置Segment列表<br/>恢复Segment结构]
Step9 --> Step10[10. 初始化查询器<br/>重建查询组件]
Step10 --> Success[回滚成功<br/>系统已恢复到目标版本]
Success --> End([回滚完成])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Step1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Step2 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Step3 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step4 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step5 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step6 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step7 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step8 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step9 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Step10 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Success fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
回滚流程：
IndexLib 支持 Schema 演进，每个 Segment 可以有不同的 Schema：
Schema 演进：支持 Schema 变更，每个 Segment 记录自己的 SchemaId：
flowchart LR
subgraph Schema["Schema 演进机制"]
S1[Segment SchemaId<br/>每个Segment记录自己的SchemaId]
S2[Schema版本映射<br/>SchemaVersionRoadMap]
S3[兼容性检查<br/>Schema兼容性验证]
S1 --> S2
S2 --> S3
end
subgraph Version["版本演进"]
V1[Version 1<br/>SchemaId: 0]
V2[Version 2<br/>SchemaId: 0]
V3[Version 3<br/>SchemaId: 1]
V1 -->|Schema变更| V2
V2 -->|新Segment使用新Schema| V3
end
subgraph Segment["Segment Schema"]
SE1[Segment 1<br/>SchemaId: 0]
SE2[Segment 2<br/>SchemaId: 0]
SE3[Segment 3<br/>SchemaId: 1]
SE4[Segment 4<br/>SchemaId: 1]
end
subgraph Compatibility["兼容性保证"]
C1[向后兼容<br/>新Schema向后兼容旧Schema]
C2[渐进式迁移<br/>新Segment使用新Schema]
C3[旧Segment保持<br/>旧Segment保持原样]
C1 --> C2
C2 --> C3
end
V1 --> SE1
V2 --> SE2
V3 --> SE3
V3 --> SE4
S3 --> C1
style Schema fill:#e3f2fd
style Version fill:#fff3e0
style Segment fill:#f3e5f5
style Compatibility fill:#e8f5e9
Schema 演进机制：
SchemaIdSchemaVersionRoadMap，记录 Schema 版本映射Schema 变更的流程：
Schema 变更流程：从 Schema 变更到版本提交：
flowchart TD
subgraph Check["检查兼容性"]
C1[检查新Schema<br/>Check New Schema]
C2[检查兼容性<br/>Check Compatibility]
C3{兼容性检查<br/>通过?}
C1 --> C2
C2 --> C3
end
subgraph Seal["Seal 当前Segment"]
S1[Seal当前Segment<br/>Seal Current Segment]
S2[停止接收新文档<br/>Stop Receiving Documents]
S3[等待转储完成<br/>Wait for Dump]
S1 --> S2
S2 --> S3
end
subgraph Create["创建新Segment"]
CR1[使用新Schema<br/>Use New Schema]
CR2[创建新MemSegment<br/>Create New MemSegment]
CR3[开始接收新文档<br/>Start Receiving Documents]
CR1 --> CR2
CR2 --> CR3
end
subgraph Commit["提交版本"]
CO1[更新SchemaId<br/>Update SchemaId]
CO2[更新SchemaVersionRoadMap<br/>Update RoadMap]
CO3[提交Version<br/>Commit Version]
CO1 --> CO2
CO2 --> CO3
end
C3 -->|通过| S1
C3 -->|失败| Error[Schema变更失败]
S3 --> CR1
CR3 --> CO1
CO3 --> Success[Schema变更成功]
style Check fill:#e3f2fd
style Seal fill:#fff3e0
style Create fill:#f3e5f5
style Commit fill:#e8f5e9
变更流程：
版本清理用于清理不再需要的旧版本文件：
版本清理：清理不再需要的旧版本文件：
flowchart TD
subgraph Identify["识别清理目标"]
I1[保留版本列表<br/>Keep Recent N Versions]
I2[识别旧版本<br/>Identify Old Versions]
I3[检查版本引用<br/>Check Version References]
I1 --> I2
I2 --> I3
end
subgraph CleanSegment["清理Segment"]
CS1[检查Segment引用<br/>Check Segment References]
CS2{Segment是否<br/>被引用?}
CS3[清理Segment文件<br/>Delete Segment Files]
CS4[清理索引文件<br/>Delete Index Files]
CS1 --> CS2
CS2 -->|否| CS3
CS3 --> CS4
end
subgraph CleanVersion["清理版本文件"]
CV1[清理版本文件<br/>Delete Version Files]
CV2[清理Fence目录<br/>Delete Fence Directories]
CV3[清理元数据<br/>Delete Metadata]
CV1 --> CV2
CV2 --> CV3
end
subgraph Result["清理结果"]
R1[释放存储空间<br/>Free Storage Space]
R2[保持系统稳定<br/>Maintain System Stability]
R1 --> R2
end
I3 --> CS1
CS2 -->|是| Skip[跳过清理]
CS4 --> CV1
CV3 --> R1
style Identify fill:#e3f2fd
style CleanSegment fill:#fff3e0
style CleanVersion fill:#f3e5f5
style Result fill:#e8f5e9
清理机制：
版本清理的策略：
版本清理策略：保留版本数量、清理时机等：
flowchart LR
subgraph Strategy["清理策略"]
S1[保留版本数<br/>Keep N Versions]
S2[清理时机<br/>Cleanup Timing]
S3[清理范围<br/>Cleanup Scope]
S1 --> S2
S2 --> S3
end
subgraph Keep["保留策略"]
K1[保留最近N个版本<br/>Keep Recent N Versions]
K2[保留活跃版本<br/>Keep Active Versions]
K3[保留重要版本<br/>Keep Important Versions]
end
subgraph Timing["清理时机"]
T1[Commit时清理<br/>Cleanup on Commit]
T2[定期清理<br/>Periodic Cleanup]
T3[手动清理<br/>Manual Cleanup]
end
subgraph Scope["清理范围"]
SC1[版本文件<br/>Version Files]
SC2[Segment文件<br/>Segment Files]
SC3[索引文件<br/>Index Files]
SC4[元数据文件<br/>Metadata Files]
end
S1 --> K1
S2 --> T1
S3 --> SC1
K1 --> T1
T1 --> SC1
SC1 --> SC2
SC2 --> SC3
SC3 --> SC4
style Strategy fill:#e3f2fd
style Keep fill:#fff3e0
style Timing fill:#f3e5f5
style Scope fill:#e8f5e9
清理策略：
在实时写入场景中，增量更新的应用：
实时写入场景中的增量更新：通过 Locator 判断数据是否已处理：
flowchart TD
subgraph Receive["接收数据"]
R1[实时接收数据流<br/>Receive Data Stream]
R2[解析文档<br/>Parse Documents]
R3[提取Locator<br/>Extract Locator]
R1 --> R2
R2 --> R3
end
subgraph Check["检查Locator"]
C1[获取Version Locator<br/>Get Version Locator]
C2[IsFasterThan比较<br/>Compare Locators]
C3{数据是否<br/>已处理?}
C1 --> C2
C2 --> C3
end
subgraph Process["处理新数据"]
P1[处理新数据<br/>Process New Data]
P2[构建索引<br/>Build Index]
P3[更新Locator<br/>Update Locator]
P1 --> P2
P2 --> P3
end
subgraph Commit["提交版本"]
CO1[定期Commit<br/>Periodic Commit]
CO2[更新Version Locator<br/>Update Version Locator]
CO3[持久化版本<br/>Persist Version]
CO1 --> CO2
CO2 --> CO3
end
R3 --> C1
C3 -->|未处理| P1
C3 -->|已处理| Skip[跳过数据]
P3 --> CO1
style Receive fill:#e3f2fd
style Check fill:#fff3e0
style Process fill:#f3e5f5
style Commit fill:#e8f5e9
实时写入流程：
IsFasterThan() 判断数据是否已处理在批量更新场景中，增量更新的应用：
批量更新场景中的增量更新：批量处理数据，避免重复处理：
flowchart TD
subgraph Read["读取数据源"]
RD1[批量读取数据<br/>Batch Read Data]
RD2[解析文档<br/>Parse Documents]
RD3[提取Locator<br/>Extract Locators]
RD1 --> RD2
RD2 --> RD3
end
subgraph Filter["过滤已处理数据"]
F1[获取Version Locator<br/>Get Version Locator]
F2[批量比较Locator<br/>Batch Compare Locators]
F3[过滤已处理数据<br/>Filter Processed Data]
F1 --> F2
F2 --> F3
end
subgraph Process["处理新数据"]
P1[批量处理新数据<br/>Batch Process New Data]
P2[批量构建索引<br/>Batch Build Index]
P3[更新Locator<br/>Update Locator]
P1 --> P2
P2 --> P3
end
subgraph Commit["提交版本"]
CO1[批量Commit<br/>Batch Commit]
CO2[更新Version Locator<br/>Update Version Locator]
CO3[持久化版本<br/>Persist Version]
CO1 --> CO2
CO2 --> CO3
end
RD3 --> F1
F3 --> P1
P3 --> CO1
style Read fill:#e3f2fd
style Filter fill:#fff3e0
style Process fill:#f3e5f5
style Commit fill:#e8f5e9
批量更新流程：
IsFasterThan() 判断哪些数据已处理版本管理的原子性通过 Fence 机制保证：
版本管理的原子性：通过 Fence 机制保证版本提交的原子性：
flowchart LR
subgraph Fence["Fence 机制"]
F1[创建Fence目录<br/>Create Fence Directory]
F2[写入版本文件<br/>Write Version File]
F3[原子重命名<br/>Atomic Rename]
F1 --> F2
F2 --> F3
end
subgraph Atomicity["原子性保证"]
A1[要么全部成功<br/>All or Nothing]
A2[要么全部失败<br/>Rollback on Failure]
A3[避免部分写入<br/>Avoid Partial Write]
A1 --> A2
A2 --> A3
end
subgraph Error["错误处理"]
E1[提交失败<br/>Commit Failure]
E2[清理Fence目录<br/>Cleanup Fence]
E3[不影响已有版本<br/>No Impact on Existing]
E1 --> E2
E2 --> E3
end
subgraph Advantage["设计优势"]
AD1[原子性<br/>Atomicity]
AD2[性能高<br/>High Performance]
AD3[可靠性<br/>Reliability]
AD4[简单性<br/>Simplicity]
end
F3 --> A1
A3 --> E1
E3 --> AD1
style Fence fill:#e3f2fd
style Atomicity fill:#fff3e0
style Error fill:#f3e5f5
style Advantage fill:#e8f5e9
原子性保证：
版本管理保证数据一致性：
版本管理的数据一致性：通过 Locator 保证数据不重复、不丢失：
flowchart TD
subgraph Consistency["数据一致性保证"]
C1[不重复保证<br/>No Duplication]
C2[不丢失保证<br/>No Loss]
C3[多数据源支持<br/>Multi-Source Support]
end
subgraph Locator["Locator 机制"]
L1[Locator比较<br/>Locator Comparison]
L2[IsFasterThan<br/>判断数据是否已处理]
L3[Update机制<br/>保证只向前推进]
L1 --> L2
L2 --> L3
end
subgraph MultiSource["多数据源支持"]
M1[SourceIdx区分<br/>Distinguish by SourceIdx]
M2[独立处理<br/>Independent Processing]
M3[保证一致性<br/>Ensure Consistency]
M1 --> M2
M2 --> M3
end
subgraph Concurrent["并发控制"]
CO1[ConcurrentIdx<br/>处理时间戳相同]
CO2[HashId分片<br/>Sharding by HashId]
CO3[保证顺序<br/>Ensure Order]
CO1 --> CO2
CO2 --> CO3
end
C1 --> L1
C2 --> L3
C3 --> M1
L3 --> CO1
style Consistency fill:#e3f2fd
style Locator fill:#fff3e0
style MultiSource fill:#f3e5f5
style Concurrent fill:#e8f5e9
数据一致性保证：
sourceIdx 区分数据源concurrentIdx 处理时间戳相同的情况版本管理的性能优化：
版本管理的性能优化：版本缓存、懒加载等：
flowchart LR
subgraph Cache["版本缓存"]
CA1[缓存常用版本<br/>Cache Common Versions]
CA2[LRU淘汰策略<br/>LRU Eviction]
CA3[减少磁盘读取<br/>Reduce Disk Reads]
CA1 --> CA2
CA2 --> CA3
end
subgraph Lazy["懒加载"]
L1[按需加载版本<br/>Load on Demand]
L2[减少启动时间<br/>Reduce Startup Time]
L3[并行加载<br/>Parallel Loading]
L1 --> L2
L2 --> L3
end
subgraph Batch["批量操作"]
B1[批量处理版本<br/>Batch Process Versions]
B2[批量清理<br/>Batch Cleanup]
B3[提高效率<br/>Improve Efficiency]
B1 --> B2
B2 --> B3
end
subgraph Optimize["优化策略"]
O1[快速路径<br/>Fast Path]
O2[短路优化<br/>Short Circuit]
O3[缓存优化<br/>Cache Optimization]
O1 --> O2
O2 --> O3
end
CA3 --> L1
L3 --> B1
B3 --> O1
style Cache fill:#e3f2fd
style Lazy fill:#fff3e0
style Batch fill:#f3e5f5
style Optimize fill:#e8f5e9
性能优化策略：
优化策略：
优化策略：
优化策略：
版本管理和增量更新是 IndexLib 的核心功能，通过 Version 和 Locator 两个机制实现。通过本文的深入解析，我们了解到：
核心机制：
IsFasterThan() 判断数据是否已处理sourceIdx 区分数据源，支持多数据源场景设计亮点：
性能优化：
理解版本管理和增量更新，是掌握 IndexLib 数据管理机制的关键。在下一篇文章中，我们将深入介绍 Segment 合并策略的实现细节，包括合并策略的选择、合并计划的创建、合并执行的流程等各个组件的实现原理和性能优化策略。
]]>查询流程图：
flowchart TD
Start[接收JSON查询请求] --> ParseGroup
subgraph ParseGroup["1. 查询解析层：解析查询请求"]
direction TB
P1[接收JSON查询]
P2[解析JSON格式]
P3[提取查询类型<br/>TermQuery/RangeQuery/BooleanQuery]
P4[提取查询条件<br/>字段名/字段值/范围]
P5[创建Query对象<br/>内部查询对象]
P1 --> P2
P2 --> P3
P3 --> P4
P4 --> P5
end
ParseGroup --> PrepareGroup
subgraph PrepareGroup["2. 索引准备层：准备查询资源"]
direction TB
PR1[获取TabletReader<br/>从Tablet获取Reader实例]
PR2[获取IndexReader<br/>根据索引类型和名称获取]
PR3[遍历Segment列表<br/>获取所有ST_BUILT状态的Segment]
PR4[准备QueryContext<br/>查询上下文和参数]
PR1 --> PR2
PR2 --> PR3
PR3 --> PR4
end
PrepareGroup --> QueryGroup
subgraph QueryGroup["3. 并行查询层：多Segment并行查询"]
direction TB
Q1[启动并行查询<br/>多线程并行执行]
Q2[Segment1查询<br/>使用LocalDocId]
Q3[Segment2查询<br/>使用LocalDocId]
Q4[SegmentN查询<br/>使用LocalDocId]
Q5[倒排索引查询<br/>InvertedIndexReader.Search]
Q6[正排索引查询<br/>AttributeIndexReader.Read]
Q7[主键索引查询<br/>PrimaryKeyIndexReader.Lookup]
Q8[收集各Segment结果<br/>包含LocalDocId和分数]
Q1 --> Q2
Q1 --> Q3
Q1 --> Q4
Q2 --> Q5
Q2 --> Q6
Q2 --> Q7
Q3 --> Q5
Q3 --> Q6
Q3 --> Q7
Q4 --> Q5
Q4 --> Q6
Q4 --> Q7
Q5 --> Q8
Q6 --> Q8
Q7 --> Q8
end
QueryGroup --> ProcessGroup
subgraph ProcessGroup["4. 结果处理层：合并和处理结果"]
direction TB
PS1[合并各Segment结果<br/>收集所有查询结果]
PS2[DocId转换<br/>LocalDocId转GlobalDocId]
PS3[DocId去重<br/>去除重复的DocId]
PS4[按相关性排序<br/>按分数或指定字段排序]
PS5[分页处理<br/>offset和limit截取]
PS1 --> PS2
PS2 --> PS3
PS3 --> PS4
PS4 --> PS5
end
ProcessGroup --> ReturnGroup
subgraph ReturnGroup["5. 结果返回层：序列化和返回"]
direction TB
R1[序列化为JSON<br/>转换为JSON格式]
R2[返回查询结果<br/>包含文档列表和总数]
end
ReturnGroup --> End[查询完成]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ParseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style P1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style P4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style P5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style PrepareGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style PR1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style PR2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style PR3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style PR4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style QueryGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Q1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Q2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style Q3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style Q4 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style Q5 fill:#81c784,stroke:#2e7d32,stroke-width:2px
style Q6 fill:#81c784,stroke:#2e7d32,stroke-width:2px
style Q7 fill:#81c784,stroke:#2e7d32,stroke-width:2px
style Q8 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style ProcessGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style PS1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style PS2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style PS3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style PS4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style PS5 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ReturnGroup fill:#fce4ec,stroke:#ef4444,stroke-width:2px
style R1 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style R2 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
IndexLib 的查询流程包括以下核心步骤：
让我们先通过图来理解整个流程：
组件交互序列图：
sequenceDiagram
participant Client
participant TabletReader
participant IndexReader
participant Segment1
participant Segment2
participant Segment3
Client->>TabletReader: JSON 查询
TabletReader->>TabletReader: 解析查询
TabletReader->>IndexReader: 获取 IndexReader
IndexReader->>Segment1: 并行查询
IndexReader->>Segment2: 并行查询
IndexReader->>Segment3: 并行查询
Segment1-->>IndexReader: 查询结果1
Segment2-->>IndexReader: 查询结果2
Segment3-->>IndexReader: 查询结果3
IndexReader->>IndexReader: 合并结果
IndexReader-->>TabletReader: 合并后的结果
TabletReader->>TabletReader: 序列化为 JSON
TabletReader-->>Client: 返回 JSON 结果
查询的核心接口定义在 framework/ITabletReader.h 中：
// framework/ITabletReader.h
class ITabletReader
{
public:
// 搜索：JSON 格式的查询
virtual Status Search(const std::string& jsonQuery, std::string& result) const = 0;
// 获取索引 Reader：根据索引类型和名称获取
virtual std::shared_ptr<index::IIndexReader> GetIndexReader(
const std::string& indexType,
const std::string& indexName) const = 0;
// 获取 Schema
virtual std::shared_ptr<config::ITabletSchema> GetSchema() const = 0;
};
关键设计：
_indexReaderMap 缓存 IndexReader，避免重复创建TabletReader 是查询的入口，定义在 framework/TabletReader.h 中：
// framework/TabletReader.h
class TabletReader : public ITabletReader
{
public:
explicit TabletReader(const std::shared_ptr<config::ITabletSchema>& schema);
// 打开：初始化 TabletData 和读取资源
Status Open(const std::shared_ptr<TabletData>& tabletData,
const framework::ReadResource& readResource);
// 搜索：JSON 格式的查询
Status Search(const std::string& jsonQuery, std::string& result) const override;
// 获取索引 Reader：根据索引类型和名称获取（带缓存）
std::shared_ptr<index::IIndexReader> GetIndexReader(
const std::string& indexType,
const std::string& indexName) const override;
protected:
// 子类实现：具体的打开逻辑
virtual Status DoOpen(const std::shared_ptr<TabletData>& tabletData,
const framework::ReadResource& readResource) = 0;
protected:
using IndexReaderMapKey = std::pair<std::string, std::string>; // (indexType, indexName)
std::shared_ptr<config::ITabletSchema> _schema;
std::map<IndexReaderMapKey, std::shared_ptr<index::IIndexReader>> _indexReaderMap; // 索引 Reader 缓存
std::shared_ptr<IIndexMemoryReclaimer> _indexMemoryReclaimer;
};
TabletReader 的关键组件：
flowchart TD
Start[TabletReader] --> ComponentGroup
subgraph ComponentGroup["TabletReader 关键组件"]
direction TB
C1[TabletReader<br/>查询入口和协调器]
C2[Schema<br/>ITabletSchema]
C3[IndexReaderMap<br/>索引Reader缓存]
C4[TabletData<br/>索引数据容器]
C5[ReadResource<br/>读取资源管理]
C1 --> C2
C1 --> C3
C1 --> C4
C1 --> C5
end
subgraph SchemaGroup["Schema：索引Schema定义"]
direction TB
S1[索引字段定义<br/>字段类型和属性]
S2[索引配置信息<br/>索引类型和参数]
S3[查询验证<br/>验证查询字段有效性]
S2 --> S1
S1 --> S3
end
subgraph IndexReaderMapGroup["IndexReaderMap：IndexReader缓存"]
direction TB
I1[缓存Key<br/>indexType和indexName]
I2[缓存Value<br/>IIndexReader实例]
I3[避免重复创建<br/>提高查询性能]
I1 --> I2
I2 --> I3
end
subgraph TabletDataGroup["TabletData：索引数据容器"]
direction TB
T1[所有Segment列表<br/>已构建的Segment]
T2[Version信息<br/>版本号和Locator]
T3[ResourceMap<br/>资源映射]
T1 --> T2
T2 --> T3
end
subgraph ReadResourceGroup["ReadResource：读取资源管理"]
direction TB
R1[内存配额控制<br/>MemoryQuotaController]
R2[缓存管理<br/>索引数据缓存]
R3[资源回收<br/>IIndexMemoryReclaimer]
R1 --> R2
R2 --> R3
end
C2 --> SchemaGroup
C3 --> IndexReaderMapGroup
C4 --> TabletDataGroup
C5 --> ReadResourceGroup
SchemaGroup --> Function[组件功能]
IndexReaderMapGroup --> Function
TabletDataGroup --> Function
ReadResourceGroup --> Function
Function --> F1[查询验证和字段解析]
Function --> F2[高效索引查询]
Function --> F3[数据访问和遍历]
Function --> F4[资源管理和优化]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ComponentLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ComponentGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style SchemaGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style S1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style S2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style S3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style IndexReaderMapGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style I1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style I2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style I3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style TabletDataGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style T1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style T2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style T3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ReadResourceGroup fill:#fce4ec,stroke:#ef4444,stroke-width:2px
style R1 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style R2 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style R3 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style Function fill:#f5f5f5,stroke:#757575,stroke-width:2px
style F1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F4 fill:#e0e0e0,stroke:#757575,stroke-width:1px
Open() 方法初始化 TabletReader，准备查询：
Open 流程：
TabletReader 的 Open 流程是查询准备的关键步骤。让我们通过序列图来理解完整的 Open 流程：
sequenceDiagram
participant Client
participant TabletReader
participant TabletData
participant ReadResource
participant NormalTabletReader
participant IndexReader
Client->>TabletReader: Open(TabletData, ReadResource)
TabletReader->>TabletReader: 保存TabletData引用
TabletReader->>TabletReader: 保存ReadResource引用
TabletReader->>NormalTabletReader: DoOpen(TabletData, ReadResource)
NormalTabletReader->>TabletData: CreateSlice(ST_BUILT)
TabletData-->>NormalTabletReader: Segments
NormalTabletReader->>IndexReader: CreateMultiFieldIndexReader()
NormalTabletReader->>IndexReader: CreateDeletionMapReader()
NormalTabletReader->>IndexReader: CreatePrimaryKeyReader()
NormalTabletReader->>IndexReader: CreateSummaryReader()
IndexReader-->>NormalTabletReader: Success
NormalTabletReader-->>TabletReader: Success
TabletReader-->>Client: Success
Open 流程详解：
_indexReaderMapSearch() 方法是查询的入口，将 JSON 查询转换为结果：
Search 流程：
Search 方法是查询的核心，负责将 JSON 查询转换为结果。让我们通过详细的序列图来理解完整的查询流程：
sequenceDiagram
participant Client
participant TabletReader
participant QueryParser
participant IndexReader
participant Segment1
participant Segment2
participant Segment3
participant ResultMerger
Client->>TabletReader: Search(jsonQuery)
TabletReader->>QueryParser: ParseQuery(jsonQuery)
QueryParser->>QueryParser: 提取查询类型
QueryParser->>QueryParser: 提取查询条件
QueryParser-->>TabletReader: Query对象
TabletReader->>IndexReader: GetIndexReader(indexType, indexName)
IndexReader-->>TabletReader: IndexReader
TabletReader->>TabletReader: CreateSlice(ST_BUILT)
TabletReader->>Segment1: Search(query)
TabletReader->>Segment2: Search(query)
TabletReader->>Segment3: Search(query)
Segment1-->>TabletReader: Result1
Segment2-->>TabletReader: Result2
Segment3-->>TabletReader: Result3
TabletReader->>ResultMerger: MergeResults([Result1, Result2, Result3])
ResultMerger->>ResultMerger: 去重
ResultMerger->>ResultMerger: 排序
ResultMerger->>ResultMerger: 分页
ResultMerger-->>TabletReader: MergedResult
TabletReader->>TabletReader: SerializeToJson(MergedResult)
TabletReader-->>Client: jsonResult
Search 流程详解：
_indexReaderMap 查找缓存的 IndexReaderTabletData->CreateSlice(ST_BUILT) 获取所有已构建的 Segment
TabletReader 维护 IndexReader 的缓存，避免重复创建：
缓存机制：
IndexReader 缓存是 TabletReader 性能优化的关键设计。让我们通过流程图来理解缓存机制的工作原理：
flowchart TD
Start[GetIndexReader请求] --> CheckCache{检查缓存<br/>IndexReaderMap中查找}
CheckCache -->|缓存命中| ReturnCached[返回缓存的IndexReader<br/>直接返回shared_ptr]
CheckCache -->|缓存未命中| CreateNew[创建新的IndexReader]
subgraph CreateGroup["创建IndexReader流程"]
direction TB
C1[根据indexType创建<br/>InvertedIndexReader/AttributeIndexReader等]
C2[初始化IndexReader<br/>设置Schema和配置]
C3[加载索引数据<br/>从Segment加载索引文件]
C4[缓存IndexReader<br/>存入IndexReaderMap]
C5[返回IndexReader<br/>返回shared_ptr]
C1 --> C2
C2 --> C3
C3 --> C4
C4 --> C5
end
CreateNew --> CreateGroup
CreateGroup --> End1[IndexReader就绪]
ReturnCached --> End1
End1 --> UseIndexReader[使用IndexReader查询]
subgraph UpdateGroup["IndexReader更新机制"]
direction TB
U1[检查是否需要更新<br/>Schema变更/Version变更]
U2{是否需要更新?}
U3[更新缓存<br/>创建新的IndexReader]
U4[替换旧缓存<br/>更新IndexReaderMap]
U5[继续使用<br/>复用现有IndexReader]
U1 --> U2
U2 -->|是| U3
U2 -->|否| U5
U3 --> U4
U4 --> U6[更新完成]
U5 --> U6
end
UseIndexReader --> UpdateGroup
subgraph CacheInfo["缓存机制特点"]
direction TB
CI1[缓存Key<br/>indexType和indexName对]
CI2[缓存Value<br/>shared_ptr IIndexReader]
CI3[性能优势<br/>避免重复创建<br/>提高查询性能]
CI1 --> CI2
CI2 --> CI3
end
UpdateGroup -.-> CacheInfo
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CheckCache fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReturnCached fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CreateNew fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CreateGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C5 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style End1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style UseIndexReader fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style UpdateGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style U1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style U2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style U3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style U4 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style U5 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style U6 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style CacheInfo fill:#f5f5f5,stroke:#757575,stroke-width:2px
style CI1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style CI2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style CI3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
缓存机制详解：
(indexType, indexName) 对
std::map 或 std::unordered_map 实现 O(log n) 或 O(1) 查找std::pair 作为 Key，支持多级索引IIndexReader 指针
shared_ptr 管理内存，自动释放缓存策略：
缓存实现：
// framework/TabletReader.h
std::shared_ptr<index::IIndexReader> TabletReader::GetIndexReader(
const std::string& indexType,
const std::string& indexName) const
{
IndexReaderMapKey key = std::make_pair(indexType, indexName);
auto it = _indexReaderMap.find(key);
if (it != _indexReaderMap.end()) {
return it->second; // 返回缓存的 IndexReader
}
// 创建新的 IndexReader（子类实现）
auto reader = DoGetIndexReader(indexType, indexName);
if (reader) {
_indexReaderMap[key] = reader; // 缓存
}
return reader;
}
IIndexReader 是索引查询的抽象接口，定义在 index/IIndexReader.h 中：
// index/IIndexReader.h
class IIndexReader
{
public:
virtual ~IIndexReader() = default;
// 打开：初始化 IndexReader
virtual Status Open(const std::shared_ptr<config::IIndexConfig>& indexConfig,
const IndexReaderParameter& indexReaderParam) = 0;
// 查询：根据查询条件查询索引
virtual Status Search(const std::shared_ptr<Query>& query,
std::shared_ptr<QueryResult>& result) = 0;
// 获取索引统计信息
virtual IndexStatistics GetStatistics() const = 0;
};
IIndexReader 的关键方法：
flowchart TD
Start[IIndexReader接口] --> OpenGroup
Start --> SearchGroup
Start --> StatisticsGroup
subgraph OpenGroup["1. Open方法：初始化IndexReader"]
direction TB
O1[Open调用<br/>参数: IndexConfig + IndexReaderParameter]
O2[初始化IndexReader<br/>设置配置和参数]
O3[加载索引数据<br/>从Segment加载索引文件到内存]
O4[准备查询资源<br/>初始化查询所需的数据结构]
O5[返回Status<br/>初始化成功或失败]
O1 --> O2
O2 --> O3
O3 --> O4
O4 --> O5
end
subgraph SearchGroup["2. Search方法：查询索引"]
direction TB
S1[Search调用<br/>参数: Query对象]
S2[解析查询条件<br/>提取查询字段和值]
S3[查询索引数据<br/>根据查询条件查找匹配的DocId]
S4[计算相关性分数<br/>根据匹配度计算分数]
S5[构建QueryResult<br/>包含DocId列表和分数]
S6[返回Status和QueryResult<br/>查询结果或错误信息]
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 --> S6
end
subgraph StatisticsGroup["3. GetStatistics方法：获取统计信息"]
direction TB
ST1[GetStatistics调用<br/>无参数]
ST2[统计文档数<br/>docCount]
ST3[统计Term数<br/>termCount]
ST4[统计索引大小<br/>indexSize]
ST5[构建IndexStatistics<br/>包含所有统计信息]
ST6[返回IndexStatistics<br/>统计信息对象]
ST1 --> ST2
ST1 --> ST3
ST1 --> ST4
ST2 --> ST5
ST3 --> ST5
ST4 --> ST5
ST5 --> ST6
end
OpenGroup -.->|必须先调用| SearchGroup
OpenGroup -.->|可以随时调用| StatisticsGroup
SearchGroup -.->|可以随时调用| StatisticsGroup
subgraph Lifecycle["方法调用生命周期"]
direction LR
L1[初始化阶段<br/>Open方法]
L2[查询阶段<br/>Search方法可多次调用]
L3[监控阶段<br/>GetStatistics方法]
L1 --> L2
L2 --> L3
L2 --> L2
end
OpenGroup -.-> Lifecycle
SearchGroup -.-> Lifecycle
StatisticsGroup -.-> Lifecycle
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style OpenGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style O1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style O2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style O5 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style SearchGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style S1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style S2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style S3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style S4 fill:#81c784,stroke:#2e7d32,stroke-width:2px
style S5 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style S6 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style StatisticsGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style ST1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ST2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ST3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ST4 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ST5 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style ST6 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style Lifecycle fill:#f5f5f5,stroke:#757575,stroke-width:2px
style L1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style L2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style L3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
IndexLib 支持多种类型的 IndexReader：
flowchart TD
Start[IndexReader类型体系] --> InvertedGroup
subgraph InvertedGroup["1. InvertedIndexReader：倒排索引Reader"]
direction TB
I1[实现IIndexReader接口]
I2[全文检索功能<br/>TermQuery/RangeQuery/BooleanQuery]
I3[返回匹配的DocId列表<br/>包含相关性分数]
I1 --> I2
I2 --> I3
end
subgraph AttributeGroup["2. AttributeReader：正排索引Reader"]
direction TB
A1[实现IIndexReader接口]
A2[属性查询功能<br/>根据DocId读取属性值]
A3[支持多种数据类型<br/>int/string/float等]
A1 --> A2
A2 --> A3
end
subgraph PrimaryKeyGroup["3. PrimaryKeyIndexReader：主键索引Reader"]
direction TB
P1[实现IIndexReader接口]
P2[主键查询功能<br/>根据主键查找DocId]
P3[支持精确匹配<br/>O1时间复杂度]
P1 --> P2
P2 --> P3
end
subgraph SummaryGroup["4. SummaryReader：摘要Reader"]
direction TB
S1[实现IIndexReader接口]
S2[获取文档摘要<br/>根据DocId读取摘要信息]
S3[支持字段选择<br/>按需读取字段]
S1 --> S2
S2 --> S3
end
subgraph DeletionMapGroup["5. DeletionMapReader：删除映射Reader"]
direction TB
D1[实现IIndexReader接口]
D2[过滤删除文档<br/>检查DocId是否已删除]
D3[支持删除标记<br/>Tombstone机制]
D1 --> D2
D2 --> D3
end
Start --> AttributeGroup
Start --> PrimaryKeyGroup
Start --> SummaryGroup
Start --> DeletionMapGroup
InvertedGroup --> Usage[使用场景]
AttributeGroup --> Usage
PrimaryKeyGroup --> Usage
SummaryGroup --> Usage
DeletionMapGroup --> Usage
Usage --> U1[全文搜索场景<br/>InvertedIndexReader]
Usage --> U2[属性过滤场景<br/>AttributeReader]
Usage --> U3[主键查找场景<br/>PrimaryKeyIndexReader]
Usage --> U4[文档展示场景<br/>SummaryReader]
Usage --> U5[删除过滤场景<br/>DeletionMapReader]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style TypeLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style InvertedGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style I1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style I2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style I3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style AttributeGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style A1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style A2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style A3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style PrimaryKeyGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style P1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style P2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style P3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style SummaryGroup fill:#fce4ec,stroke:#ef4444,stroke-width:2px
style S1 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style S2 fill:#f48fb1,stroke:#ef4444,stroke-width:2px
style S3 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style DeletionMapGroup fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style D1 fill:#fff59d,stroke:#f57f17,stroke-width:1px
style D2 fill:#ffcc02,stroke:#f57f17,stroke-width:2px
style D3 fill:#fff59d,stroke:#f57f17,stroke-width:1px
style Usage fill:#f5f5f5,stroke:#757575,stroke-width:2px
style U1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U4 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style U5 fill:#e0e0e0,stroke:#757575,stroke-width:1px
IndexReader 类型：
InvertedIndexReader 是倒排索引的查询接口：
flowchart TD
A[查询请求<br/>Query对象] --> B[解析查询类型<br/>TermQuery/RangeQuery等]
subgraph Parse["查询解析"]
B1[提取查询Term<br/>分词处理]
B2[提取查询条件<br/>范围/布尔等]
B3[创建内部Query对象]
B --> B1
B1 --> B2
B2 --> B3
end
subgraph Search["索引查找"]
C1[在倒排索引中查找Term<br/>InvertedIndex]
C2[获取Term的倒排列表<br/>PostingList]
C3[DocId列表<br/>包含该Term的文档]
C4[位置信息<br/>Term在文档中的位置]
B3 --> C1
C1 --> C2
C2 --> C3
C2 --> C4
end
subgraph Filter["过滤处理"]
D1[通过DeletionMap过滤<br/>过滤已删除文档]
D2[范围查询过滤<br/>如果包含范围条件]
D3[布尔查询处理<br/>AND/OR/NOT]
C3 --> D1
C4 --> D1
D1 --> D2
D2 --> D3
end
subgraph Score["相关性计算"]
E1[计算相关性分数<br/>TF-IDF/BM25等]
E2[位置信息加权<br/>短语查询]
E3[字段权重<br/>不同字段权重不同]
D3 --> E1
E1 --> E2
E2 --> E3
end
subgraph Result["返回结果"]
F1[DocId列表<br/>匹配的文档ID]
F2[相关性分数<br/>排序依据]
F3[位置信息<br/>用于高亮显示]
E3 --> F1
E3 --> F2
E3 --> F3
end
style Parse fill:#e3f2fd
style Search fill:#fff3e0
style Filter fill:#e8f5e9
style Score fill:#f3e5f5
style Result fill:#fce4ec
倒排索引查询流程：
AttributeReader 是正排索引的查询接口：
flowchart TD
A[查询请求<br/>GlobalDocId + 属性名] --> B[定位Segment<br/>根据GlobalDocId]
subgraph Locate["定位阶段"]
B1[遍历TabletData中的Segment]
B2[计算每个Segment的BaseDocId<br/>累加前面Segment的docCount]
B3{GlobalDocId在范围内?<br/>BaseDocId <= GlobalDocId < BaseDocId + docCount}
B4[找到对应Segment]
B --> B1
B1 --> B2
B2 --> B3
B3 -->|是| B4
B3 -->|否| B1
end
subgraph Convert["DocId转换"]
C1[计算LocalDocId<br/>LocalDocId = GlobalDocId - BaseDocId]
C2[验证LocalDocId有效性<br/>0 <= LocalDocId < docCount]
B4 --> C1
C1 --> C2
end
subgraph Read["读取属性"]
D1[根据属性名获取AttributeIndexer<br/>GetAttributeReader]
D2[定位属性数据位置<br/>根据LocalDocId]
D3[读取属性值<br/>从磁盘或内存]
D4[数据类型转换<br/>整数/浮点数/字符串等]
D5[解压缩<br/>如果使用了压缩]
C2 --> D1
D1 --> D2
D2 --> D3
D3 --> D4
D4 --> D5
end
subgraph Return["返回结果"]
E1[返回属性值<br/>AttributeValue]
E2[支持批量读取<br/>多个DocId一次读取]
D5 --> E1
E1 --> E2
end
subgraph Optimize["性能优化"]
O1[缓存常用属性<br/>减少磁盘IO]
O2[批量读取<br/>减少IO次数]
O3[预读机制<br/>预读相邻数据]
D3 -.-> O1
D3 -.-> O2
D3 -.-> O3
end
style Locate fill:#e3f2fd
style Convert fill:#fff3e0
style Read fill:#e8f5e9
style Return fill:#f3e5f5
style Optimize fill:#f5f5f5
正排索引查询流程：
查询解析将 JSON 格式的查询转换为内部查询对象：
flowchart TD
A[JSON查询字符串<br/>jsonQuery] --> B[解析JSON<br/>JsonParser]
subgraph Parse["JSON解析"]
B1[解析JSON对象<br/>提取字段]
B2[验证JSON格式<br/>格式检查]
B3[提取查询类型字段<br/>queryType]
B --> B1
B1 --> B2
B2 --> B3
end
subgraph Extract["提取查询信息"]
C1[提取查询类型<br/>TermQuery/RangeQuery/BoolQuery等]
C2[提取查询条件<br/>term/范围/排序字段]
C3[提取排序信息<br/>sortField/sortOrder]
C4[提取分页信息<br/>offset/limit]
C5[提取聚合信息<br/>aggregation]
B3 --> C1
C1 --> C2
C1 --> C3
C1 --> C4
C1 --> C5
end
subgraph Create["创建查询对象"]
D1[创建TermQuery对象<br/>term查询]
D2[创建RangeQuery对象<br/>范围查询]
D3[创建BoolQuery对象<br/>布尔查询]
D4[创建Query对象<br/>组合查询]
C2 --> D1
C2 --> D2
C2 --> D3
D1 --> D4
D2 --> D4
D3 --> D4
end
subgraph Validate["验证查询"]
E1[Schema验证<br/>字段是否存在]
E2[类型验证<br/>字段类型匹配]
E3[范围验证<br/>查询条件有效性]
D4 --> E1
E1 --> E2
E2 --> E3
end
subgraph Result["查询对象"]
F1[内部Query对象<br/>用于后续查询]
F2[包含所有查询信息<br/>类型/条件/排序等]
E3 --> F1
F1 --> F2
end
style Parse fill:#e3f2fd
style Extract fill:#fff3e0
style Create fill:#e8f5e9
style Validate fill:#f3e5f5
style Result fill:#fce4ec
查询解析流程：
查询时需要遍历多个 Segment，可以并行查询以提高性能：
flowchart TD
A[查询请求<br/>Query对象] --> B[TabletData.CreateSlice<br/>ST_BUILT获取Segment列表]
subgraph Segments["Segment列表"]
S1[Segment1<br/>docCount=1000<br/>BaseDocId=0]
S2[Segment2<br/>docCount=2000<br/>BaseDocId=1000]
S3[Segment3<br/>docCount=1500<br/>BaseDocId=3000]
B --> S1
B --> S2
B --> S3
end
subgraph Parallel["并行查询执行"]
P1[Segment1查询<br/>IndexReader.Search]
P2[Segment2查询<br/>IndexReader.Search]
P3[Segment3查询<br/>IndexReader.Search]
P4[线程池执行<br/>并发查询]
P5[收集查询结果<br/>Result1, Result2, Result3]
P6[错误处理<br/>单个Segment失败不影响其他]
S1 --> P1
S2 --> P2
S3 --> P3
P1 --> P4
P2 --> P4
P3 --> P4
P4 --> P5
P5 --> P6
end
subgraph Merge["结果合并"]
M1[DocId去重<br/>避免重复文档]
M2[按相关性分数排序<br/>或按指定字段排序]
M3[分页处理<br/>offset/limit]
M4[聚合统计<br/>总数/平均值等]
P6 --> M1
M1 --> M2
M2 --> M3
M3 --> M4
end
subgraph Performance["性能优化"]
PF1[并行度控制<br/>线程池大小配置]
PF2[结果流式合并<br/>边查询边合并]
PF3[索引剪枝<br/>跳过不相关Segment]
PF4[Locator剪枝<br/>判断Segment是否包含结果]
P4 -.-> PF1
M1 -.-> PF2
B -.-> PF3
B -.-> PF4
end
M4 --> R[返回合并结果<br/>QueryResult]
style Segments fill:#e3f2fd
style Parallel fill:#fff3e0
style Merge fill:#f3e5f5
style Performance fill:#f5f5f5
style R fill:#e8f5e9
并行查询流程：
TabletData->CreateSlice(ST_BUILT) 获取所有已构建的 Segment查询时需要将全局 DocId 转换为局部 DocId：
flowchart TD
A[查询请求<br/>GlobalDocId] --> B[TabletData.GetSegment<br/>遍历Segment列表]
subgraph Locate["定位Segment"]
L1[遍历所有Segment<br/>按顺序查找]
L2[计算每个Segment的BaseDocId<br/>累加前面Segment的docCount]
L3{GlobalDocId在范围内?<br/>BaseDocId <= GlobalDocId < BaseDocId + docCount}
L4[找到对应Segment]
L5[继续遍历下一个Segment]
B --> L1
L1 --> L2
L2 --> L3
L3 -->|是| L4
L3 -->|否| L5
L5 --> L1
end
subgraph Convert["DocId转换"]
C1[获取Segment的BaseDocId<br/>前面所有Segment的docCount之和]
C2[计算LocalDocId<br/>LocalDocId = GlobalDocId - BaseDocId]
C3[验证LocalDocId有效性<br/>0 <= LocalDocId < docCount]
C4[验证失败处理<br/>返回错误]
L4 --> C1
C1 --> C2
C2 --> C3
C3 -->|无效| C4
end
subgraph Query["Segment内查询"]
Q1[使用LocalDocId查询<br/>IndexReader.Get]
Q2[倒排索引查询<br/>InvertedIndexer]
Q3[正排索引查询<br/>AttributeIndexer]
Q4[主键索引查询<br/>PrimaryKeyIndexer]
Q5[返回文档数据<br/>Document]
C3 -->|有效| Q1
Q1 --> Q2
Q1 --> Q3
Q1 --> Q4
Q2 --> Q5
Q3 --> Q5
Q4 --> Q5
end
subgraph Example["转换示例"]
E1[GlobalDocId = 1500]
E2[Segment1: BaseDocId=0, docCount=1000<br/>范围: 0-999, 不在范围内]
E3[Segment2: BaseDocId=1000, docCount=2000<br/>范围: 1000-2999, 在范围内]
E4[LocalDocId = 1500 - 1000 = 500]
E5[在Segment2内使用LocalDocId=500查询]
E1 --> E2
E2 --> E3
E3 --> E4
E4 --> E5
end
Q5 --> R[返回查询结果]
C4 --> R
style Locate fill:#e3f2fd
style Convert fill:#fff3e0
style Query fill:#f3e5f5
style Example fill:#f5f5f5
style R fill:#e8f5e9
DocId 转换流程：
localDocId = globalDocId - baseDocId查询结果需要合并，包括去重、排序等：
结果合并流程：
结果合并是查询流程的关键步骤，需要高效地处理大量查询结果。让我们通过流程图来理解结果合并的详细过程：
flowchart TD
A["多个Segment的查询结果<br/>Result1, Result2, Result3"] --> B["结果收集<br/>收集所有Segment结果"]
subgraph Collect["结果收集"]
B1["收集DocId列表<br/>来自各Segment"]
B2["收集相关性分数<br/>用于排序"]
B3["收集位置信息<br/>用于高亮"]
B --> B1
B --> B2
B --> B3
end
subgraph Dedup["去重处理"]
C1["DocId去重<br/>避免重复文档"]
C2["去重算法选择<br/>set或unordered_set或双指针"]
C3["有序结果优化<br/>双指针算法时间复杂度O n"]
C4["无序结果<br/>hash set时间复杂度O n"]
B1 --> C1
C1 --> C2
C2 -->|有序| C3
C2 -->|无序| C4
end
subgraph Sort["排序处理"]
D1{"是否需要排序?"}
D2["按相关性分数排序<br/>相关性高的在前"]
D3["按指定字段排序<br/>时间或数值等"]
D4["按DocId排序<br/>默认排序"]
D5["排序算法<br/>堆排序或快速排序"]
D6["Top-K优化<br/>只对Top-K排序"]
C3 --> D1
C4 --> D1
D1 -->|是| D2
D1 -->|是| D3
D1 -->|否| D4
D2 --> D5
D3 --> D5
D5 --> D6
end
subgraph Page["分页处理"]
E1["计算分页范围<br/>offset到offset加limit"]
E2["截取结果<br/>只返回需要的文档"]
E3["分页缓存<br/>缓存分页结果"]
D6 --> E1
D4 --> E1
E1 --> E2
E2 --> E3
end
subgraph Aggregate["聚合统计"]
F1{"是否需要聚合?"}
F2["总数统计<br/>匹配文档总数"]
F3["平均值统计<br/>字段平均值"]
F4["分组统计<br/>按字段分组"]
F5["并行计算聚合<br/>减少开销"]
E3 --> F1
F1 -->|是| F2
F1 -->|是| F3
F1 -->|是| F4
F2 --> F5
F3 --> F5
F4 --> F5
end
subgraph Optimize["合并优化"]
O1["堆合并<br/>时间复杂度O n log k适合Top-K"]
O2["并行合并<br/>充分利用多核CPU"]
O3["流式合并<br/>边查询边合并"]
O4["减少内存占用<br/>提高响应速度"]
C1 -.-> O1
D5 -.-> O2
B1 -.-> O3
O3 -.-> O4
end
F1 -->|否| G["返回结果<br/>QueryResult"]
F5 --> G
style Collect fill:#e3f2fd
style Dedup fill:#fff3e0
style Sort fill:#e8f5e9
style Page fill:#f3e5f5
style Aggregate fill:#fce4ec
style Optimize fill:#f5f5f5
结果合并流程详解：
std::set 或 std::unordered_set 实现 O(n) 去重结果合并的性能优化：
NormalTabletReader 是标准表的查询实现，定义在 table/normal_table/NormalTabletReader.h 中：
// table/normal_table/NormalTabletReader.h
class NormalTabletReader : public framework::TabletReader
{
public:
NormalTabletReader(const std::shared_ptr<config::ITabletSchema>& schema,
const std::shared_ptr<NormalTabletMetrics>& normalTabletMetrics);
// 打开：初始化 TabletData 和读取资源
Status DoOpen(const std::shared_ptr<framework::TabletData>& tabletData,
const framework::ReadResource& readResource) override;
// 搜索：JSON 格式的查询
Status Search(const std::string& jsonQuery, std::string& result) const override;
// 获取各种 IndexReader
std::shared_ptr<indexlib::index::InvertedIndexReader> GetMultiFieldIndexReader() const;
const std::shared_ptr<index::DeletionMapIndexReader>& GetDeletionMapReader() const;
const std::shared_ptr<indexlib::index::PrimaryKeyIndexReader>& GetPrimaryKeyReader() const;
std::shared_ptr<index::SummaryReader> GetSummaryReader() const;
std::shared_ptr<index::AttributeReader> GetAttributeReader(const std::string& attrName) const;
};
NormalTabletReader 的关键组件：
flowchart TD
Start[NormalTabletReader] --> ComponentGroup
subgraph ComponentGroup["NormalTabletReader 关键组件"]
direction TB
C1[NormalTabletReader<br/>普通索引表的查询入口]
C2[MultiFieldIndexReader<br/>多字段倒排索引Reader]
C3[DeletionMapReader<br/>删除映射Reader]
C4[PrimaryKeyReader<br/>主键索引Reader]
C5[SummaryReader<br/>摘要Reader]
C6[AttributeReader<br/>属性Reader]
C1 --> C2
C1 --> C3
C1 --> C4
C1 --> C5
C1 --> C6
end
subgraph MultiFieldGroup["MultiFieldIndexReader：多字段倒排索引"]
direction TB
M1[管理多个字段的倒排索引<br/>支持多字段联合查询]
M2[全文检索功能<br/>TermQuery/RangeQuery等]
M3[返回匹配的DocId列表<br/>包含相关性分数]
M1 --> M2
M2 --> M3
end
subgraph DeletionMapGroup["DeletionMapReader：删除映射"]
direction TB
D1[管理删除文档映射<br/>记录已删除的DocId]
D2[过滤删除文档<br/>查询时过滤已删除文档]
D3[支持Tombstone机制<br/>标记删除状态]
D1 --> D2
D2 --> D3
end
subgraph PrimaryKeyGroup["PrimaryKeyReader：主键索引"]
direction TB
P1[管理主键索引<br/>主键到DocId的映射]
P2[主键查询功能<br/>根据主键查找DocId]
P3[支持精确匹配<br/>O1时间复杂度]
P1 --> P2
P2 --> P3
end
subgraph SummaryGroup["SummaryReader：摘要"]
direction TB
S1[管理文档摘要<br/>存储文档的摘要信息]
S2[获取文档摘要<br/>根据DocId读取摘要]
S3[支持字段选择<br/>按需读取字段]
S1 --> S2
S2 --> S3
end
subgraph AttributeGroup["AttributeReader：属性"]
direction TB
A1[管理属性索引<br/>存储文档的属性值]
A2[属性查询功能<br/>根据DocId读取属性值]
A3[支持多种数据类型<br/>int/string/float等]
A1 --> A2
A2 --> A3
end
C2 --> MultiFieldGroup
C3 --> DeletionMapGroup
C4 --> PrimaryKeyGroup
C5 --> SummaryGroup
C6 --> AttributeGroup
MultiFieldGroup --> Function[组件功能]
DeletionMapGroup --> Function
PrimaryKeyGroup --> Function
SummaryGroup --> Function
AttributeGroup --> Function
Function --> F1[全文搜索<br/>MultiFieldIndexReader]
Function --> F2[删除过滤<br/>DeletionMapReader]
Function --> F3[主键查找<br/>PrimaryKeyReader]
Function --> F4[文档展示<br/>SummaryReader]
Function --> F5[属性查询<br/>AttributeReader]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ComponentLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ComponentGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style C2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C6 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style MultiFieldGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style M1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style M2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style M3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style DeletionMapGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style D1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style D2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style D3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style PrimaryKeyGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style P1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style P2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style P3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style SummaryGroup fill:#fce4ec,stroke:#ef4444,stroke-width:2px
style S1 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style S2 fill:#f48fb1,stroke:#ef4444,stroke-width:2px
style S3 fill:#f8bbd0,stroke:#ef4444,stroke-width:1px
style AttributeGroup fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style A1 fill:#fff59d,stroke:#f57f17,stroke-width:1px
style A2 fill:#ffcc02,stroke:#f57f17,stroke-width:2px
style A3 fill:#fff59d,stroke:#f57f17,stroke-width:1px
style Function fill:#f5f5f5,stroke:#757575,stroke-width:2px
style F1 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F2 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F3 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F4 fill:#e0e0e0,stroke:#757575,stroke-width:1px
style F5 fill:#e0e0e0,stroke:#757575,stroke-width:1px
DoOpen() 方法初始化 NormalTabletReader：
flowchart LR
Start([DoOpen 开始]) --> Step1[1. 初始化<br/>TabletData]
Step1 --> Step2[2. 创建<br/>Reader 组件]
subgraph Readers["Reader 组件创建顺序"]
direction LR
R1["① MultiFieldIndexReader<br/>多字段倒排索引"]
R2["② DeletionMapReader<br/>删除映射"]
R3["③ PrimaryKeyReader<br/>主键索引"]
R4["④ SummaryReader<br/>摘要"]
R5["⑤ AttributeReader<br/>属性"]
R1 --> R2 --> R3 --> R4 --> R5
end
Step2 --> R1
R5 --> Complete([完成初始化])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Step1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Step2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Readers fill:#f5f5f5,stroke:#757575,stroke-width:2px
style R1 fill:#e3f2fd,stroke:#1976d2,stroke-width:1.5px
style R2 fill:#fff3e0,stroke:#f57c00,stroke-width:1.5px
style R3 fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px
style R4 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1.5px
style R5 fill:#e0f2f1,stroke:#00695c,stroke-width:1.5px
style Complete fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
DoOpen 流程：
Search() 方法实现标准表的查询：
flowchart LR
Start([Search 开始]) --> A[解析查询]
A --> Prepare[准备阶段]
subgraph Prepare["准备阶段"]
direction LR
B[获取 IndexReader]
C[遍历 Segment]
B --> C
end
Prepare --> B
C --> Query[查询阶段]
subgraph Query["查询阶段"]
direction LR
D[并行查询]
end
Query --> D
D --> PostProcess[后处理阶段]
subgraph PostProcess["后处理阶段"]
direction LR
E[过滤删除文档]
F[合并结果]
G[返回结果]
E --> F --> G
end
PostProcess --> E
G --> End([完成])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Prepare fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style B fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style C fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
style Query fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style D fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style PostProcess fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style E fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style F fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style G fill:#ba68c8,stroke:#7b1fa2,stroke-width:1.5px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
Search 流程：
查询剪枝可以减少不必要的查询：
flowchart TD
Start([查询剪枝]) --> Strategies[剪枝策略]
subgraph Strategies["三种剪枝策略"]
direction LR
S1[Locator 剪枝]
S2[范围剪枝]
S3[索引剪枝]
end
Strategies --> S1
Strategies --> S2
Strategies --> S3
S1 --> R1[判断 Segment<br/>是否包含结果]
S2 --> R2[减少查询范围<br/>缩小搜索空间]
S3 --> R3[跳过不相关索引<br/>提高查询效率]
R1 --> Benefit[优化效果]
R2 --> Benefit
R3 --> Benefit
Benefit --> End([减少不必要查询<br/>提升性能])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Strategies fill:#f5f5f5,stroke:#757575,stroke-width:2px
style S1 fill:#fff3e0,stroke:#f57c00,stroke-width:1.5px
style S2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:1.5px
style S3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1.5px
style R1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style R2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style R3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style Benefit fill:#fff9c4,stroke:#f9a825,stroke-width:2px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
查询剪枝策略：
查询缓存可以提高查询性能：
flowchart TD
Start([查询缓存机制]) --> CacheLayer[缓存层]
subgraph CacheLayer["缓存层"]
direction TB
subgraph Cache1["结果缓存"]
direction LR
C1[结果缓存] --> E1[避免重复查询<br/>直接返回缓存结果]
end
subgraph Cache2["索引缓存"]
direction LR
C2[索引缓存] --> E2[减少 IO 操作<br/>从内存读取索引]
end
subgraph Cache3["统计缓存"]
direction LR
C3[统计缓存] --> E3[减少计算开销<br/>复用统计信息]
end
end
CacheLayer --> Cache1
CacheLayer --> Cache2
CacheLayer --> Cache3
E1 --> Benefit[综合性能提升]
E2 --> Benefit
E3 --> Benefit
Benefit --> End([提升查询性能<br/>降低系统负载])
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style CacheLayer fill:#f5f5f5,stroke:#757575,stroke-width:2px
style Cache1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1.5px
style E1 fill:#fff8e1,stroke:#f57c00,stroke-width:1.5px
style Cache2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style C2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style E2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
style Cache3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style C3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1.5px
style E3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:1.5px
style Benefit fill:#fff9c4,stroke:#f9a825,stroke-width:3px
style End fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
查询缓存机制：
并行查询可以提高查询性能：
flowchart TB
Start([并行查询优化<br/>Parallel Query Optimization]) --> StrategyLayer[优化策略层<br/>Optimization Strategies Layer]
subgraph StrategyGroup["并行查询策略 Parallel Query Strategies"]
direction TB
S1[Segment 并行<br/>Segment Parallel<br/>多个Segment并行查询<br/>提高查询吞吐量]
S2[索引并行<br/>Index Parallel<br/>多个索引并行查询<br/>充分利用多核CPU]
S3[结果并行合并<br/>Result Parallel Merge<br/>查询结果并行合并<br/>减少合并时间]
end
StrategyLayer --> BenefitLayer[性能提升层<br/>Performance Benefits Layer]
subgraph BenefitGroup["性能提升 Performance Benefits"]
direction TB
B1[缩短查询延迟<br/>Reduce Query Latency<br/>并行执行减少等待时间]
B2[提高查询吞吐量<br/>Increase Throughput<br/>充分利用系统资源]
B3[提升系统效率<br/>Improve Efficiency<br/>优化资源利用率]
end
BenefitLayer --> End([优化完成<br/>Optimization Complete])
StrategyLayer -.->|包含| StrategyGroup
BenefitLayer -.->|包含| BenefitGroup
S1 -.->|实现| B1
S2 -.->|实现| B2
S3 -.->|实现| B3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style BenefitLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style S1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style S3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style BenefitGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style B1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style B2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style B3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
并行查询优化：
索引加载优化可以减少查询延迟：
flowchart TD
Start[索引加载优化] --> Strategies[优化策略]
subgraph Strategies["三种优化策略"]
direction LR
L1[1. 按需加载<br/>只加载查询需要的索引]
L2[2. 懒加载<br/>查询时才加载索引数据]
L3[3. 预加载<br/>预加载常用索引减少延迟]
end
Strategies --> L1
Strategies --> L2
Strategies --> L3
L1 --> Benefit[优化效果]
L2 --> Benefit
L3 --> Benefit
subgraph Effects["优化效果"]
direction LR
E1[减少内存占用]
E2[提升加载效率]
end
Benefit --> Effects
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Strategies fill:#f5f5f5,stroke:#757575,stroke-width:2px
style L1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style L2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style L3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Benefit fill:#fff9c4,stroke:#f9a825,stroke-width:2px
style Effects fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style E1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style E2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:1.5px
索引加载优化：
内存优化可以减少内存使用：
flowchart TB
Start([内存优化<br/>Memory Optimization]) --> StrategyLayer[优化策略层<br/>Optimization Strategies Layer]
subgraph StrategyGroup["内存优化策略 Memory Optimization Strategies"]
direction TB
M1[内存池<br/>Memory Pool<br/>减少内存分配开销<br/>提高分配效率]
M2[缓存控制<br/>Cache Control<br/>控制缓存大小避免溢出<br/>动态调整缓存策略]
M3[内存回收<br/>Memory Reclaim<br/>及时回收不再使用的内存<br/>释放内存空间]
end
StrategyLayer --> BenefitLayer[优化效果层<br/>Optimization Benefits Layer]
subgraph BenefitGroup["优化效果 Optimization Benefits"]
direction TB
B1[降低内存占用<br/>Reduce Memory Usage<br/>减少内存分配和占用]
B2[提升系统稳定性<br/>Improve Stability<br/>避免内存溢出和崩溃]
B3[提高性能<br/>Improve Performance<br/>减少内存分配开销]
end
BenefitLayer --> End([优化完成<br/>Optimization Complete])
StrategyLayer -.->|包含| StrategyGroup
BenefitLayer -.->|包含| BenefitGroup
M1 -.->|实现| B1
M2 -.->|实现| B2
M3 -.->|实现| B3
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style BenefitLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style StrategyGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style BenefitGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style B1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style B2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style B3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
内存优化策略：
IO 优化可以减少 IO 操作：
flowchart TD
Start[IO 优化] --> Strategy[优化策略]
subgraph Strategy["三种优化策略"]
direction LR
I1[1. 批量读取<br/>减少 IO 次数]
I2[2. 预读<br/>减少查询延迟]
I3[3. IO 合并<br/>减少 IO 开销]
end
Strategy --> I1
Strategy --> I2
Strategy --> I3
I1 --> Benefit[优化效果]
I2 --> Benefit
I3 --> Benefit
Benefit --> E1[提升 IO 效率]
Benefit --> E2[降低系统负载]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Strategy fill:#f5f5f5,stroke:#757575,stroke-width:2px
style I1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style I2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style I3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Benefit fill:#fff9c4,stroke:#f9a825,stroke-width:2px
style E1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
style E2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1.5px
IO 优化策略：
在全文检索场景中，查询流程：
flowchart TB
Start([全文检索流程<br/>Full-Text Search Flow]) --> ParseLayer[解析层<br/>Parse Layer]
subgraph ParseGroup["查询解析 Query Parsing"]
direction TB
P1[解析查询<br/>Parse Query<br/>解析term查询条件]
P2[获取 InvertedIndexReader<br/>Get InvertedIndexReader<br/>获取倒排索引Reader]
end
ParseLayer --> SearchLayer[查找层<br/>Search Layer]
subgraph SearchGroup["索引查找 Index Search"]
direction TB
S1[查找 term<br/>Search Term<br/>在倒排索引中查找]
S2[获取倒排列表<br/>Get Posting List<br/>获取term对应的DocId列表]
end
SearchLayer --> FilterLayer[过滤层<br/>Filter Layer]
subgraph FilterGroup["结果过滤 Result Filtering"]
direction TB
F1[过滤删除文档<br/>Filter Deleted Docs<br/>通过DeletionMap过滤]
F2[计算相关性<br/>Calculate Relevance<br/>计算文档相关性分数]
end
FilterLayer --> ResultLayer[结果层<br/>Result Layer]
subgraph ResultGroup["结果处理 Result Processing"]
direction TB
R1[排序返回<br/>Sort and Return<br/>按相关性分数排序]
end
ResultLayer --> End([查询完成<br/>Query Complete])
ParseLayer -.->|包含| ParseGroup
SearchLayer -.->|包含| SearchGroup
FilterLayer -.->|包含| FilterGroup
ResultLayer -.->|包含| ResultGroup
P1 --> P2
P2 --> S1
S1 --> S2
S2 --> F1
F1 --> F2
F2 --> R1
R1 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style ParseLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style SearchLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style FilterLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style ResultLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ParseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style SearchGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style S1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style S2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style FilterGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style F1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style F2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style ResultGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style R1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
全文检索流程：
在属性查询场景中，查询流程：
flowchart TB
Start([属性查询流程<br/>Attribute Query Flow]) --> ParseLayer[解析层<br/>Parse Layer]
subgraph ParseGroup["查询解析 Query Parsing"]
direction TB
P1[解析查询<br/>Parse Query<br/>解析属性查询条件]
P2[获取 AttributeReader<br/>Get AttributeReader<br/>获取属性索引Reader]
end
ParseLayer --> TraverseLayer[遍历层<br/>Traverse Layer]
subgraph TraverseGroup["Segment遍历 Segment Traversal"]
direction TB
T1[遍历 Segment<br/>Traverse Segments<br/>遍历所有已构建的Segment]
end
TraverseLayer --> QueryLayer[查询层<br/>Query Layer]
subgraph QueryGroup["属性查询 Attribute Query"]
direction TB
Q1[查询属性<br/>Query Attribute<br/>在Segment内查询属性值]
Q2[过滤匹配<br/>Filter Matches<br/>过滤匹配查询条件的文档]
end
QueryLayer --> ResultLayer[结果层<br/>Result Layer]
subgraph ResultGroup["结果返回 Result Return"]
direction TB
R1[返回结果<br/>Return Results<br/>返回匹配的文档列表]
end
ResultLayer --> End([查询完成<br/>Query Complete])
ParseLayer -.->|包含| ParseGroup
TraverseLayer -.->|包含| TraverseGroup
QueryLayer -.->|包含| QueryGroup
ResultLayer -.->|包含| ResultGroup
P1 --> P2
P2 --> T1
T1 --> Q1
Q1 --> Q2
Q2 --> R1
R1 --> End
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style ParseLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style TraverseLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style QueryLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style ResultLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ParseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style TraverseGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style T1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style QueryGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style Q1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style Q2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style ResultGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style R1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
属性查询流程：
优化策略：
优化策略：
优化策略：
查询流程是 IndexLib 的核心功能，包括 TabletReader 和 IndexReader 两个层次。通过本文的深入解析，我们了解到：
核心组件：
设计亮点：
性能优化：
理解查询流程，是掌握 IndexLib 查询机制的关键。在下一篇文章中，我们将深入介绍版本管理和增量更新的实现细节，包括 Version 结构、Locator 机制、增量更新流程等各个组件的实现原理和性能优化策略。
]]>索引构建流程图：
flowchart TD
Start[开始构建] --> ReceiveDoc[接收文档批次<br/>IDocumentBatch]
ReceiveDoc --> BuildStart[Build阶段]
subgraph BuildGroup["1. Build阶段：构建索引到内存"]
direction TB
B1[文档验证<br/>格式/Schema验证]
B2[分配DocId<br/>BaseDocId + LocalDocId]
B3[写入Indexer<br/>InvertedIndexer/AttributeIndexer]
B4[更新SegmentInfo<br/>docCount/Locator]
B5[评估内存使用<br/>EvaluateCurrentMemUsed]
B6{是否需要Flush?<br/>内存超阈值/文档数超阈值}
B1 --> B2
B2 --> B3
B3 --> B4
B4 --> B5
B5 --> B6
end
BuildStart --> B1
B6 -->|否，继续构建| ReceiveDoc
B6 -->|是，触发转储| FlushStart[Flush阶段]
subgraph FlushGroup["2. Flush阶段：转储到磁盘"]
direction TB
F1[创建SegmentDumper<br/>准备转储]
F2[转储MemSegment<br/>异步转储索引文件]
F3[创建DiskSegment<br/>加载转储后的Segment]
F4[更新TabletData<br/>添加DiskSegment]
F5{是否需要Seal?<br/>Segment数量/时间间隔}
F1 --> F2
F2 --> F3
F3 --> F4
F4 --> F5
end
FlushStart --> F1
F5 -->|否，继续构建| ReceiveDoc
F5 -->|是，触发封存| SealStart[Seal阶段]
subgraph SealGroup["3. Seal阶段：封存Segment"]
direction TB
S1[封存当前MemSegment<br/>标记为只读]
S2[等待转储完成<br/>确保数据已持久化]
S3[更新Segment状态<br/>ST_BUILT]
S4{是否需要Commit?<br/>版本更新条件}
S1 --> S2
S2 --> S3
S3 --> S4
end
SealStart --> S1
S4 -->|否，继续构建| ReceiveDoc
S4 -->|是，触发提交| CommitStart[Commit阶段]
subgraph CommitGroup["4. Commit阶段：提交版本"]
direction TB
C1[准备新Version<br/>收集Segment列表]
C2[更新Locator<br/>记录最新处理位置]
C3[写入Version文件<br/>序列化为JSON]
C4[创建Fence目录<br/>保证原子性]
C5[原子切换版本<br/>重命名Fence目录]
C6[更新TabletData<br/>切换到新版本]
C1 --> C2
C2 --> C3
C3 --> C4
C4 --> C5
C5 --> C6
end
CommitStart --> C1
C6 --> Continue{继续构建?}
Continue -->|是| ReceiveDoc
Continue -->|否| End[构建完成]
B6 -.->|循环构建| ReceiveDoc
F5 -.->|循环构建| ReceiveDoc
S4 -.->|循环构建| ReceiveDoc
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReceiveDoc fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style BuildStart fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style BuildGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style B6 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style FlushStart fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style FlushGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style F5 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style SealStart fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style SealGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style S4 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style CommitStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CommitGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Continue fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
IndexLib 的索引构建流程包括四个核心阶段：
让我们先通过图来理解整个流程：
流程关系图：
flowchart TB
Start([开始构建<br/>Start Build]) --> BuildLayer[Build阶段<br/>Build Phase]
subgraph BuildGroup["Build 构建索引 Build Index"]
direction TB
B1[Build构建索引<br/>Build Index<br/>接收文档批次]
B2[写入内存<br/>Write to Memory<br/>构建到MemSegment]
B1 --> B2
end
BuildLayer --> MemLayer[MemSegment阶段<br/>MemSegment Phase]
subgraph MemGroup["MemSegment 内存段"]
direction TB
M1[MemSegment内存段<br/>Memory Segment<br/>实时构建和写入]
end
MemLayer --> FlushLayer[Flush阶段<br/>Flush Phase]
subgraph FlushGroup["Flush 转储 Flush"]
direction TB
F1[触发转储<br/>Trigger Flush<br/>内存超阈值或文档数超阈值]
F2[转储到磁盘<br/>Flush to Disk<br/>异步转储索引文件]
F1 --> F2
end
FlushLayer --> DiskLayer[DiskSegment阶段<br/>DiskSegment Phase]
subgraph DiskGroup["DiskSegment 磁盘段"]
direction TB
D1[DiskSegment磁盘段<br/>Disk Segment<br/>持久化存储]
end
DiskLayer --> SealLayer[Seal阶段<br/>Seal Phase]
subgraph SealGroup["Seal 封存 Seal"]
direction TB
S1[触发封存<br/>Trigger Seal<br/>Segment数量或时间间隔]
S2[标记只读<br/>Mark Read-Only<br/>Sealed Segment已封存]
S1 --> S2
end
SealLayer --> CommitLayer[Commit阶段<br/>Commit Phase]
subgraph CommitGroup["Commit 提交版本 Commit Version"]
direction TB
C1[触发提交<br/>Trigger Commit<br/>版本更新条件]
C2[更新版本<br/>Update Version<br/>创建新Version]
C1 --> C2
end
CommitLayer --> VersionLayer[Version阶段<br/>Version Phase]
subgraph VersionGroup["Version 版本"]
direction TB
V1[Version版本<br/>Version<br/>记录Segment列表和Locator]
end
VersionLayer --> DiskLayer2[磁盘存储阶段<br/>Disk Storage Phase]
subgraph DiskGroup2["磁盘存储 Disk Storage"]
direction TB
DS1[持久化<br/>Persistence<br/>写入磁盘]
end
DiskLayer2 --> Continue{继续构建?<br/>Continue Build?}
Continue -->|是| BuildLayer
Continue -->|否| End([构建完成<br/>Build Complete])
BuildLayer -.->|包含| BuildGroup
MemLayer -.->|包含| MemGroup
FlushLayer -.->|包含| FlushGroup
DiskLayer -.->|包含| DiskGroup
SealLayer -.->|包含| SealGroup
CommitLayer -.->|包含| CommitGroup
VersionLayer -.->|包含| VersionGroup
DiskLayer2 -.->|包含| DiskGroup2
B2 --> M1
M1 --> F1
F2 --> D1
D1 --> S1
S2 --> C1
C2 --> V1
V1 --> DS1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style BuildLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style MemLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style FlushLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style DiskLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style SealLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style CommitLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style VersionLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style DiskLayer2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style BuildGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style B1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style MemGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style M1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style FlushGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style F1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style DiskGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style D1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style SealGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style S1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style S2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style CommitGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style C1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style C2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style VersionGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style V1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style DiskGroup2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style DS1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
索引构建的核心接口定义在 framework/ITablet.h 中：
// framework/ITablet.h
class ITablet : private autil::NoCopyable
{
public:
// 构建：接收文档批次并写入内存段
virtual Status Build(const std::shared_ptr<document::IDocumentBatch>& batch) = 0;
// 刷新：将内存数据刷新到磁盘
virtual Status Flush() = 0;
// 封存：封存当前 Segment，准备合并
virtual Status Seal() = 0;
// 提交版本：创建新版本并持久化
virtual std::pair<Status, VersionMeta> Commit(const CommitOptions& commitOptions) = 0;
// 判断是否需要提交
virtual bool NeedCommit() const = 0;
};
关键设计：
Build 阶段负责接收文档批次，将文档写入内存中的索引结构。让我们先通过图来理解 Build 流程：
flowchart TD
subgraph Input["输入阶段"]
A1[接收文档批次<br/>IDocumentBatch]
A2[批次大小配置<br/>平衡内存和性能]
A1 --> A2
end
subgraph Validate["验证阶段"]
B1[文档格式验证<br/>格式检查]
B2[Schema验证<br/>字段定义检查]
B3[数据有效性验证<br/>数值范围/字符串长度]
A2 --> B1
B1 --> B2
B2 --> B3
end
subgraph DocId["DocId分配阶段"]
C1[获取BaseDocId<br/>前面所有Segment的docCount之和]
C2[分配LocalDocId<br/>从0开始递增]
C3[计算GlobalDocId<br/>BaseDocId + LocalDocId]
B3 --> C1
C1 --> C2
C2 --> C3
end
subgraph Indexer["写入Indexer阶段"]
D1[解析文档<br/>提取字段和Term]
D2[写入倒排索引<br/>InvertedIndexer]
D3[写入正排索引<br/>AttributeIndexer]
D4[写入主键索引<br/>PrimaryKeyIndexer]
D5[写入摘要索引<br/>SummaryIndexer]
C3 --> D1
D1 --> D2
D1 --> D3
D1 --> D4
D1 --> D5
end
subgraph Update["更新阶段"]
E1[更新SegmentInfo<br/>docCount递增]
E2[更新Locator<br/>记录数据处理位置]
E3[更新时间戳<br/>最后处理时间]
D2 --> E1
D3 --> E1
D4 --> E1
D5 --> E1
E1 --> E2
E2 --> E3
end
subgraph Check["检查阶段"]
F1[评估内存使用<br/>EvaluateCurrentMemUsed]
F2{转储条件检查<br/>NeedDump?}
F3[内存阈值检查<br/>默认80%]
F4[文档数阈值检查<br/>默认100万]
F5[时间阈值检查<br/>默认5分钟]
E3 --> F1
F1 --> F2
F2 -.-> F3
F2 -.-> F4
F2 -.-> F5
F2 -->|否| A1
F2 -->|是| G1
end
subgraph Flush["Flush触发"]
G1[触发Flush<br/>创建SegmentDumper]
F2 --> G1
end
style Input fill:#e3f2fd
style Validate fill:#fff9c4
style DocId fill:#fff3e0
style Indexer fill:#e8f5e9
style Update fill:#f3e5f5
style Check fill:#fce4ec
style Flush fill:#ffebee
Build 流程包括以下步骤：
Build() 接收 IDocumentBatch
GlobalDocId = BaseDocId + LocalDocIdBuild 流程的序列图：
sequenceDiagram
participant Client
participant TabletWriter
participant MemSegment
participant InvertedIndexer
participant AttributeIndexer
participant SegmentInfo
participant MemCtrl as MemoryQuotaController
Client->>TabletWriter: Build(documentBatch)
TabletWriter->>TabletWriter: ValidateDocuments(batch)
TabletWriter->>TabletWriter: DispatchDocIds(batch)
TabletWriter->>MemSegment: Build(batch)
loop 遍历每个文档
MemSegment->>InvertedIndexer: BuildDocument(doc, docId)
MemSegment->>AttributeIndexer: BuildDocument(doc, docId)
InvertedIndexer-->>MemSegment: Success
AttributeIndexer-->>MemSegment: Success
end
MemSegment->>SegmentInfo: UpdateDocCount()
MemSegment->>SegmentInfo: UpdateLocator()
MemSegment-->>TabletWriter: Success
TabletWriter->>MemCtrl: CheckMemoryQuota()
MemCtrl-->>TabletWriter: quotaStatus
alt 内存不足
TabletWriter-->>Client: NoMem
else 需要转储
TabletWriter-->>Client: NeedDump
else 成功
TabletWriter-->>Client: OK
end
TabletWriter 是构建的核心实现，定义在 framework/TabletWriter.h 中：
// framework/TabletWriter.h
class TabletWriter : private autil::NoCopyable
{
public:
// 构建文档批次
// 返回值：
// - OK: 构建成功
// - NoMem: 内存不足，需要等待内存释放
// - NeedDump: 触发转储，需要转储并重新打开
virtual Status Build(const std::shared_ptr<document::IDocumentBatch>& batch) = 0;
// 创建转储器：准备转储 MemSegment
virtual std::unique_ptr<SegmentDumper> CreateSegmentDumper() = 0;
// 获取总内存使用
virtual size_t GetTotalMemSize() const = 0;
// 获取构建 Segment 转储所需的内存扩展大小
virtual size_t GetBuildingSegmentDumpExpandSize() const = 0;
// 判断是否有未提交的数据
virtual bool IsDirty() const = 0;
};
Build 的返回值：
Build 方法的返回值反映了构建的状态，调用方需要根据返回值采取相应的行动：
CreateSegmentDumper() 创建转储器Flush() 执行转储状态转换图：
stateDiagram-v2
[*] --> Building: Build开始
state Building {
[*] --> Receiving: 接收文档批次
Receiving --> Validating: 文档验证
Validating --> Allocating: 分配DocId
Allocating --> Writing: 写入Indexer
Writing --> Updating: 更新SegmentInfo
Updating --> Evaluating: 评估内存使用
Evaluating --> Checking: 检查转储条件
Checking --> [*]: 继续构建
}
Building --> Building: Build返回OK继续构建
Building --> NeedDump: Build返回NeedDump
Building --> NoMem: Build返回NoMem
state NeedDump {
[*] --> Creating: 创建SegmentDumper
Creating --> [*]
}
NeedDump --> Flushing: CreateSegmentDumper完成
state Flushing {
[*] --> Dumping: 转储MemSegment
Dumping --> CreatingDisk: 创建DiskSegment
CreatingDisk --> UpdatingData: 更新TabletData
UpdatingData --> [*]
}
Flushing --> Dumped: Flush完成
state Dumped {
[*] --> Ready: 转储完成
Ready --> [*]
}
Dumped --> Building: Reopen重新打开创建新MemSegment
state NoMem {
[*] --> WaitingState: 等待内存释放
WaitingState --> [*]
}
NoMem --> Waiting: 进入等待状态
state Waiting {
[*] --> Monitoring: 监控内存状态
Monitoring --> [*]
}
Waiting --> Building: 内存释放继续构建
Waiting --> Flushing: 主动Flush释放内存
Building --> [*]: 构建完成
在 Build 阶段，需要为文档分配 DocId。关键代码（table/normal_table/NormalTabletWriter.h）：
// table/normal_table/NormalTabletWriter.h
class NormalTabletWriter : public table::CommonTabletWriter
{
private:
// 分发 DocId：为文档分配 DocId
void DispatchDocIds(document::IDocumentBatch* batch);
docid_t _buildingSegmentBaseDocId; // 当前构建 Segment 的基础 DocId
std::shared_ptr<NormalMemSegment> _normalBuildingSegment; // 当前构建中的 Segment
};
DocId 分配机制：
flowchart TD
Start[文档写入IDocumentBatch] --> GetMem[获取当前MemSegment]
GetMem --> GetBase[获取BaseDocId]
GetBase --> BaseStart[BaseDocId计算]
BaseStart --> C1[遍历TabletData中的Segment]
C1 --> C2[累加前面Segment的docCount]
C2 --> C3[BaseDocId等于docCount之和]
C3 --> LocalStart[LocalDocId分配]
LocalStart --> D1[获取当前MemSegment的docCount]
D1 --> D2[LocalDocId从0开始]
D2 --> D3[LocalDocId递增每个文档加1]
D3 --> D4[更新docCount递增]
D4 --> GlobalStart[GlobalDocId计算]
GlobalStart --> E1[GlobalDocId等于BaseDocId加LocalDocId]
E1 --> E2[全局唯一文档ID]
E2 --> E3[写入Indexer使用GlobalDocId]
E3 --> End[完成]
subgraph BaseGroup["1. BaseDocId计算"]
C1
C2
C3
end
subgraph LocalGroup["2. LocalDocId分配"]
D1
D2
D3
D4
end
subgraph GlobalGroup["3. GlobalDocId计算"]
E1
E2
E3
end
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetMem fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style GetBase fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style BaseStart fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style BaseGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style C1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style C3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style LocalStart fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style LocalGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style D1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style D2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style D3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style D4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style GlobalStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style GlobalGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style E1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style E2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style E3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
baseDocId + localDocId文档写入各个 Indexer 的过程：
flowchart TD
A[文档对象<br/>IDocument] --> B[解析文档<br/>DocumentParser]
subgraph Parse["解析阶段"]
B1[提取字段<br/>ExtractFields]
B2[提取Term<br/>分词处理]
B3[数据转换<br/>转换为索引格式]
B --> B1
B1 --> B2
B2 --> B3
end
subgraph Inverted["倒排索引写入"]
C1[InvertedIndexer.BuildDocument<br/>doc, docId]
C2[提取文本字段的Term]
C3[建立Term到文档映射<br/>Term → DocId]
C4[更新PostingList<br/>倒排列表]
C5[记录位置信息<br/>用于短语查询]
B3 --> C1
C1 --> C2
C2 --> C3
C3 --> C4
C4 --> C5
end
subgraph Attribute["正排索引写入"]
D1[AttributeIndexer.BuildDocument<br/>doc, docId]
D2[按字段存储属性值]
D3[支持多种数据类型<br/>整数/浮点数/字符串]
D4[压缩存储<br/>减少内存占用]
B3 --> D1
D1 --> D2
D2 --> D3
D3 --> D4
end
subgraph Primary["主键索引写入"]
E1[PrimaryKeyIndexer.BuildDocument<br/>doc, docId]
E2[提取主键字段]
E3[建立主键到DocId映射<br/>PrimaryKey → DocId]
B3 --> E1
E1 --> E2
E2 --> E3
end
subgraph Summary["摘要索引写入"]
F1[SummaryIndexer.BuildDocument<br/>doc, docId]
F2[生成文档摘要<br/>用于搜索结果展示]
F3[存储摘要信息<br/>减少查询时的磁盘IO]
B3 --> F1
F1 --> F2
F2 --> F3
end
subgraph Complete["完成阶段"]
G1[所有Indexer写入完成]
G2[更新SegmentInfo<br/>docCount/Locator]
C5 --> G1
D4 --> G1
E3 --> G1
F3 --> G1
G1 --> G2
end
style Parse fill:#e3f2fd
style Inverted fill:#fff3e0
style Attribute fill:#e8f5e9
style Primary fill:#f3e5f5
style Summary fill:#fce4ec
style Complete fill:#f5f5f5
写入流程：
文档写入 Indexer 是构建的核心步骤，需要高效地处理大量文档。让我们通过序列图来理解详细的写入流程：
sequenceDiagram
participant Writer as TabletWriter
participant MemSeg as MemSegment
participant DocParser as DocumentParser
participant InvertedIdx as InvertedIndexer
participant AttributeIdx as AttributeIndexer
participant SummaryIdx as SummaryIndexer
Writer->>MemSeg: Build(documentBatch)
loop 遍历每个文档
MemSeg->>DocParser: ParseDocument(doc)
DocParser->>DocParser: ExtractFields()
DocParser->>DocParser: ExtractTerms()
DocParser-->>MemSeg: ParsedDocument
MemSeg->>InvertedIdx: BuildDocument(parsedDoc, docId)
InvertedIdx->>InvertedIdx: AddTerm(term, docId)
InvertedIdx->>InvertedIdx: UpdatePostingList()
InvertedIdx-->>MemSeg: Success
MemSeg->>AttributeIdx: BuildDocument(parsedDoc, docId)
AttributeIdx->>AttributeIdx: WriteAttribute(field, value)
AttributeIdx-->>MemSeg: Success
MemSeg->>SummaryIdx: BuildDocument(parsedDoc, docId)
SummaryIdx->>SummaryIdx: UpdateSummary()
SummaryIdx-->>MemSeg: Success
end
MemSeg-->>Writer: Success
写入流程详解：
性能优化：
Build 阶段需要严格控制内存使用，避免内存溢出。关键机制：
内存控制机制：
内存控制是保证系统稳定性的关键。让我们通过流程图来理解完整的内存控制机制：
flowchart TD
A[开始构建<br/>Build调用] --> B[估算内存使用<br/>EstimateMemUsed]
subgraph Estimate["内存估算"]
B1[根据Schema估算<br/>字段类型/数量]
B2[根据文档数估算<br/>批次大小]
B3[根据索引类型估算<br/>倒排/正排/主键]
B4[估算值略大于实际值<br/>保证安全]
B --> B1
B1 --> B2
B2 --> B3
B3 --> B4
end
subgraph Check["配额检查"]
C1[MemoryQuotaController<br/>内存配额控制器]
C2{内存配额充足?}
C3[返回NoMem<br/>拒绝写入]
C4[分配内存<br/>预留内存空间]
B4 --> C1
C1 --> C2
C2 -->|否| C3
C2 -->|是| C4
end
subgraph Build["构建过程"]
D1[Build文档<br/>写入Indexer]
D2[评估实际内存使用<br/>EvaluateCurrentMemUsed]
D3[统计所有Indexer内存<br/>采样评估减少开销]
C4 --> D1
D1 --> D2
D2 --> D3
end
subgraph Monitor["内存监控"]
E1{内存使用检查}
E2[警告阈值: 70%<br/>发出警告]
E3[转储阈值: 80%<br/>触发转储]
E4[拒绝阈值: 95%<br/>拒绝新写入]
E5{文档数检查<br/>默认100万}
E6{时间检查<br/>默认5分钟}
D3 --> E1
E1 --> E2
E1 --> E3
E1 --> E4
E1 --> E5
E1 --> E6
end
subgraph Dump["转储触发"]
F1[返回NeedDump<br/>触发转储]
F2[异步转储<br/>不阻塞写入]
F3[释放MemSegment内存]
E3 --> F1
E5 -->|超过阈值| F1
E6 -->|超过阈值| F1
F1 --> F2
F2 --> F3
F3 --> D1
end
E1 -->|未超阈值| D1
E5 -->|未超阈值| D1
E6 -->|未超阈值| D1
style Estimate fill:#e3f2fd
style Check fill:#fff9c4
style Build fill:#fff3e0
style Monitor fill:#f3e5f5
style Dump fill:#e8f5e9
内存控制机制详解：
EstimateMemUsed() 估算构建所需内存
EvaluateCurrentMemUsed() 评估当前实际内存使用
内存控制策略：
Flush 阶段负责将内存数据刷新到磁盘，创建 DiskSegment。让我们先通过图来理解 Flush 流程：
flowchart TD
A[Flush调用<br/>或自动触发] --> B[检查转储条件<br/>NeedDump检查]
subgraph Conditions["转储条件判断"]
C1{内存使用检查<br/>默认阈值80%}
C2{文档数检查<br/>默认阈值100万}
C3{时间检查<br/>默认阈值5分钟}
C4[OR策略: 任一满足即触发]
C5[AND策略: 全部满足才触发]
C6[优先级策略: 内存优先]
B --> C1
B --> C2
B --> C3
C1 --> C4
C2 --> C4
C3 --> C4
C4 --> C5
C5 --> C6
end
subgraph Create["创建Dumper"]
D1[创建SegmentDumper<br/>CreateSegmentDumper]
D2[准备转储参数<br/>内存配额/IO配额]
D3[预留转储资源<br/>避免资源竞争]
D4[创建转储项列表<br/>索引文件/元数据文件]
C6 -->|满足条件| D1
D1 --> D2
D2 --> D3
D3 --> D4
end
subgraph Dump["执行转储"]
E1[设置Segment状态<br/>ST_BUILDING → ST_DUMPING]
E2[创建转储项<br/>CreateSegmentDumpItems]
E3[索引文件转储<br/>倒排/正排/主键索引]
E4[元数据文件转储<br/>SegmentInfo/SegmentMetrics]
E5[异步转储到磁盘<br/>Dump方法]
E6[文件组织<br/>Package/Archive格式]
D4 --> E1
E1 --> E2
E2 --> E3
E2 --> E4
E3 --> E5
E4 --> E5
E5 --> E6
end
subgraph CreateDisk["创建DiskSegment"]
F1[创建SegmentMeta<br/>元数据信息]
F2[创建DiskSegment<br/>从转储文件]
F3[初始化DiskSegment<br/>Open方法]
F4[根据OpenMode加载<br/>NORMAL/LAZY]
E6 --> F1
F1 --> F2
F2 --> F3
F3 --> F4
end
subgraph Update["更新TabletData"]
G1[Reopen TabletData<br/>更新版本]
G2[添加DiskSegment<br/>AddSegment]
G3[移除MemSegment<br/>RemoveSegment]
G4[释放MemSegment内存]
F4 --> G1
G1 --> G2
G2 --> G3
G3 --> G4
end
C6 -->|不满足| A
style Conditions fill:#e3f2fd
style Create fill:#fff9c4
style Dump fill:#fff3e0
style CreateDisk fill:#e8f5e9
style Update fill:#f3e5f5
Flush 流程包括以下步骤：
转储条件判断通过 MemSegment::NeedDump() 实现：
// framework/MemSegment.h
class MemSegment : public Segment
{
public:
// 是否需要转储：判断是否达到转储条件
virtual bool NeedDump() const = 0;
// 创建转储项：准备转储到磁盘
virtual std::pair<Status, std::vector<std::shared_ptr<SegmentDumpItem>>>
CreateSegmentDumpItems() = 0;
};
转储条件：
转储条件的判断是 Flush 阶段的关键，需要综合考虑多个因素。让我们通过流程图来理解转储条件的判断逻辑：
graph TD
A[检查转储条件] --> B{内存使用检查}
B -->|超过阈值| C[触发转储]
B -->|未超阈值| D{文档数检查}
D -->|超过阈值| C
D -->|未超阈值| E{时间检查}
E -->|超过阈值| C
E -->|未超阈值| F[继续构建]
C --> G[创建SegmentDumper]
G --> H[执行转储]
style B fill:#e3f2fd
style D fill:#fff3e0
style E fill:#f3e5f5
style C fill:#e8f5e9
转储条件详解：
转储条件组合策略：
SegmentDumper 负责将 MemSegment 转储到磁盘，定义在 framework/SegmentDumper.h 中：
// framework/SegmentDumper.h
class SegmentDumper : public SegmentDumpable
{
public:
SegmentDumper(const std::string& tabletName,
const std::shared_ptr<MemSegment>& segment,
int64_t dumpExpandMemSize,
std::shared_ptr<kmonitor::MetricsReporter> metricsReporter)
: _tabletName(tabletName)
, _dumpingSegment(segment)
, _dumpExpandMemSize(dumpExpandMemSize)
{
// 设置 Segment 状态为 DUMPING
_dumpingSegment->SetSegmentStatus(Segment::SegmentStatus::ST_DUMPING);
}
// 执行转储
virtual Status Dump() = 0;
// 获取转储的 SegmentMeta
virtual std::pair<Status, SegmentMeta> GetDumpedSegmentMeta() = 0;
};
转储流程：
flowchart TD
Start[CreateSegmentDumper创建转储器] --> InitStart[初始化阶段]
InitStart --> B1[设置Segment状态ST_BUILDING到ST_DUMPING]
B1 --> B2[准备转储参数dumpExpandMemSize]
B2 --> B3[创建MetricsReporter监控转储进度]
B3 --> CreateStart[创建转储项]
CreateStart --> C1[调用CreateSegmentDumpItems MemSegment方法]
C1 --> C2[创建索引文件转储项倒排正排主键索引]
C1 --> C3[创建元数据文件转储项SegmentInfo SegmentMetrics]
C1 --> C4[创建摘要文件转储项SummaryIndex]
C2 --> C5[转储项列表DumpItems]
C3 --> C5
C4 --> C5
C5 --> DumpStart[执行转储]
DumpStart --> D1[调用Dump方法SegmentDumper.Dump]
D1 --> D2[遍历每个DumpItem]
D2 --> D3[写入索引文件磁盘IO操作]
D2 --> D4[写入元数据文件SegmentInfo等]
D3 --> D5[文件组织Package Archive格式]
D4 --> D5
D5 --> D6[原子性保证要么全部成功要么全部失败]
D6 --> DiskStart[创建DiskSegment]
DiskStart --> E1[获取转储的SegmentMeta GetDumpedSegmentMeta]
E1 --> E2[创建DiskSegment从转储文件]
E2 --> E3[初始化DiskSegment Open方法]
E3 --> E4[根据OpenMode加载NORMAL或LAZY]
E4 --> UpdateStart[更新状态]
UpdateStart --> F1[Segment状态更新ST_DUMPING到ST_BUILT]
F1 --> F2[更新TabletData添加DiskSegment]
F2 --> F3[移除MemSegment释放内存]
F3 --> End[转储完成]
subgraph InitGroup["1. 初始化阶段"]
B1
B2
B3
end
subgraph CreateGroup["2. 创建转储项"]
C1
C2
C3
C4
C5
end
subgraph DumpGroup["3. 执行转储"]
D1
D2
D3
D4
D5
D6
end
subgraph DiskGroup["4. 创建DiskSegment"]
E1
E2
E3
E4
end
subgraph UpdateGroup["5. 更新状态"]
F1
F2
F3
end
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style InitStart fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style InitGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CreateStart fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style CreateGroup fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style DumpStart fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DumpGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DiskStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style DiskGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style UpdateStart fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style UpdateGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
CreateSegmentDumper() 创建转储器ST_DUMPINGDump() 将内存数据写入磁盘ST_BUILT（实际已被 DiskSegment 替代）转储是异步的，不会阻塞新的写入。关键设计：
flowchart TD
A[MemSegment1达到转储条件<br/>NeedDump返回true] --> B[创建SegmentDumper<br/>CreateSegmentDumper]
B --> C[加入转储队列<br/>DumpQueue.Enqueue]
subgraph Async["异步转储机制"]
D1[转储线程池<br/>DumpThreadPool]
D2[从队列取出Dumper<br/>Dequeue]
D3[执行转储<br/>Dumper.Dump]
D4[写入磁盘<br/>异步IO操作]
D5[创建DiskSegment<br/>转储完成]
C --> D1
D1 --> D2
D2 --> D3
D3 --> D4
D4 --> D5
end
subgraph Continue["继续写入"]
E1[创建新MemSegment2<br/>CreateNewMemSegment]
E2[设置状态ST_BUILDING<br/>开始接收新文档]
E3[继续Build操作<br/>不阻塞写入]
E4[写入新文档批次<br/>IDocumentBatch]
B --> E1
E1 --> E2
E2 --> E3
E3 --> E4
end
subgraph Control["资源控制"]
F1[DumpControl<br/>转储任务控制]
F2[并发度限制<br/>限制同时转储任务数]
F3[优先级调度<br/>重要任务优先]
F4[资源监控<br/>内存/IO使用监控]
D1 -.-> F1
F1 --> F2
F1 --> F3
F1 --> F4
end
subgraph Advantages["异步优势"]
G1[不阻塞写入<br/>写入延迟低]
G2[提高吞吐量<br/>写入和转储并行]
G3[资源控制<br/>避免资源竞争]
G4[用户体验好<br/>请求立即返回]
E3 -.-> G1
D3 -.-> G2
F1 -.-> G3
E3 -.-> G4
end
style Async fill:#e3f2fd
style Continue fill:#fff3e0
style Control fill:#e8f5e9
style Advantages fill:#f3e5f5
异步转储的优势：
异步转储是 IndexLib 高性能写入的关键设计。让我们通过序列图来理解异步转储的完整机制：
sequenceDiagram
participant Writer as TabletWriter
participant MemSeg1 as MemSegment1
participant Dumper as SegmentDumper
participant DumpQueue as DumpQueue
participant DumpThread as DumpThread
participant MemSeg2 as MemSegment2
participant DiskSeg as DiskSegment
Writer->>MemSeg1: NeedDump()?
MemSeg1-->>Writer: true
Writer->>Writer: CreateSegmentDumper()
Writer->>Dumper: SegmentDumper(MemSeg1)
Writer->>DumpQueue: Enqueue(Dumper)
Writer->>MemSeg2: CreateNewMemSegment()
Writer->>MemSeg2: Build(newBatch)
DumpThread->>DumpQueue: Dequeue()
DumpQueue-->>DumpThread: Dumper
DumpThread->>Dumper: Dump()
Dumper->>DiskSeg: CreateDiskSegment()
DiskSeg-->>Dumper: Success
Dumper-->>DumpThread: Success
异步转储的优势详解：
DumpControl 控制转储任务的并发度
异步转储的性能优化：
转储需要额外的内存空间，通过 DumpExpandMemSize 控制：
flowchart TD
Start[转储内存成本管理] --> Estimate[估算转储内存<br/>EstimateDumpMemUsed]
Estimate --> CheckQuota[检查内存配额<br/>MemoryQuotaController]
CheckQuota --> QuotaCheck{配额充足?}
QuotaCheck -->|是| Allocate[分配转储内存<br/>从MemoryQuotaController分配]
QuotaCheck -->|否| Wait[等待内存释放<br/>或拒绝转储]
Allocate --> DumpControl[控制转储并发<br/>DumpControl限制并发度]
DumpControl --> Dump[执行转储<br/>使用分配的内存]
Dump --> Monitor[监控内存使用<br/>实时监控转储内存]
Monitor --> Release[释放转储内存<br/>转储完成后释放]
Release --> End[转储完成]
Wait --> Retry{重试?}
Retry -->|是| CheckQuota
Retry -->|否| Reject[拒绝转储<br/>返回错误]
subgraph Config["配置参数"]
direction TB
Config1[DumpExpandMemSize<br/>控制转储内存上限]
Config2[避免内存溢出<br/>限制单次转储内存]
Config1 --> Config2
end
Config2 -.->|配置| Allocate
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Estimate fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CheckQuota fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style QuotaCheck fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Allocate fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style DumpControl fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Dump fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Monitor fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Release fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Wait fill:#ffebee,stroke:#c62828,stroke-width:2px
style Retry fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Reject fill:#ffebee,stroke:#c62828,stroke-width:2px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Config fill:#f5f5f5,stroke:#757575,stroke-width:1px
内存成本控制：
EstimateDumpMemUsed() 估算转储所需内存Seal 阶段负责封存 Segment，标记为只读，不再接收新文档。让我们先通过图来理解 Seal 流程：
flowchart TD
A[Seal调用<br/>MemSegment.Seal] --> B[检查Segment状态<br/>ST_BUILDING]
subgraph Seal["封存操作"]
C1[标记为只读<br/>不再接收新文档]
C2[设置状态标志<br/>_sealed = true]
C3[检查Segment数据<br/>docCount > 0?]
B --> C1
C1 --> C2
C2 --> C3
end
subgraph Dump["有数据时转储"]
D1{有数据?<br/>docCount > 0}
D2[触发转储<br/>Flush操作]
D3[创建SegmentDumper<br/>CreateSegmentDumper]
D4[执行转储<br/>Dump方法]
D5[等待转储完成<br/>同步等待]
D6[创建DiskSegment<br/>从转储文件]
D7[更新状态<br/>ST_BUILT]
C3 --> D1
D1 -->|是| D2
D2 --> D3
D3 --> D4
D4 --> D5
D5 --> D6
D6 --> D7
end
subgraph Empty["无数据时直接完成"]
E1[无数据<br/>docCount == 0]
E2[直接完成<br/>无需转储]
E3[更新状态<br/>ST_BUILT]
D1 -->|否| E1
E1 --> E2
E2 --> E3
end
subgraph Purpose["Seal的作用"]
P1[不再接收新文档<br/>写入保护]
P2[准备合并<br/>可以参与合并操作]
P3[保证一致性<br/>Segment内容不再变化]
P4[版本提交前置条件<br/>Commit前必须Seal]
C1 -.-> P1
D7 -.-> P2
E3 -.-> P2
D7 -.-> P3
E3 -.-> P3
D7 -.-> P4
E3 -.-> P4
end
subgraph Scenarios["使用场景"]
S1[合并前<br/>封存待合并Segment]
S2[版本提交前<br/>封存所有Segment]
S3[Schema变更前<br/>封存当前Segment]
P4 -.-> S1
P4 -.-> S2
P4 -.-> S3
end
D7 --> F[完成Seal]
E3 --> F
style Seal fill:#e3f2fd
style Dump fill:#fff3e0
style Empty fill:#e8f5e9
style Purpose fill:#f3e5f5
style Scenarios fill:#f5f5f5
Seal 流程包括以下步骤：
MemSegment::Seal() 封存当前构建中的 SegmentST_BUILTMemSegment::Seal() 的实现：
// framework/MemSegment.h
class MemSegment : public Segment
{
public:
// 封存：标记为只读，不再接收新文档
virtual void Seal() = 0;
};
Seal 的作用：
Seal 通常在以下场景使用：
flowchart TD
Start[Seal使用场景] --> Scenario1[场景1: 合并前]
Start --> Scenario2[场景2: 版本提交前]
Start --> Scenario3[场景3: Schema变更前]
subgraph MergeScenario["场景1: 合并前"]
direction TB
M1[触发合并操作]
M2[封存待合并Segment<br/>标记为只读]
M3[准备合并数据<br/>Segment内容不再变化]
M4[执行合并操作]
Scenario1 --> M1
M1 --> M2
M2 --> M3
M3 --> M4
end
subgraph CommitScenario["场景2: 版本提交前"]
direction TB
C1[触发版本提交]
C2[封存所有Segment<br/>确保版本一致性]
C3[准备新Version<br/>收集Segment列表]
C4[提交新版本]
Scenario2 --> C1
C1 --> C2
C2 --> C3
C3 --> C4
end
subgraph SchemaScenario["场景3: Schema变更前"]
direction TB
S1[检测Schema变更]
S2[封存当前Segment<br/>使用旧Schema]
S3[创建新Segment<br/>使用新Schema]
S4[继续构建新Segment]
Scenario3 --> S1
S1 --> S2
S2 --> S3
S3 --> S4
end
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Scenario1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Scenario2 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Scenario3 fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style MergeScenario fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CommitScenario fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style SchemaScenario fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
使用场景：
Commit 阶段负责提交新版本，更新 Version，持久化到磁盘。让我们先通过图来理解 Commit 流程：
flowchart TD
Start[Commit调用VersionCommitter.Commit] --> Check[检查提交条件NeedCommit检查]
Check --> ConditionStart[提交条件判断]
ConditionStart --> C1{有新Segment?<br/>有新增的DiskSegment}
ConditionStart --> C2{有数据变更?<br/>Locator更新}
ConditionStart --> C3{强制提交?<br/>forceCommit=true}
C1 --> C4[OR策略任一满足即提交]
C2 --> C4
C3 --> C4
C4 --> ConditionCheck{满足提交条件?}
ConditionCheck -->|否| Start
ConditionCheck -->|是| PrepareStart[准备版本信息]
PrepareStart --> D1[收集所有已构建Segment CreateSlice ST_BUILT]
D1 --> D2[准备Segment列表SegmentInVersion]
D2 --> D3[准备Locator最新数据处理位置]
D3 --> D4[准备时间戳当前时间]
D4 --> D5[计算新VersionId当前VersionId加1]
D5 --> FenceStart[Fence机制原子性保证]
FenceStart --> E1[创建Fence目录临时目录]
E1 --> E2[写入Version文件版本信息]
E2 --> E3[写入Segment列表SegmentInVersion]
E3 --> E4[写入Locator位置信息]
E4 --> E5[原子切换重命名为正式版本目录]
E5 --> UpdateStart[更新TabletData]
UpdateStart --> F1[更新Version _onDiskVersion]
F1 --> F2[更新Segment列表 _segments]
F2 --> F3[更新Locator最新位置信息]
F3 --> CleanupStart[清理旧版本]
CleanupStart --> G1[检查保留版本列表reservedVersions]
G1 --> G2[删除不再需要的版本cleanVersion=true]
G2 --> G3[清理旧Segment文件释放磁盘空间]
G3 --> End[Commit完成返回VersionMeta]
subgraph ConditionGroup["1. 提交条件判断"]
C1
C2
C3
C4
end
subgraph PrepareGroup["2. 准备版本信息"]
D1
D2
D3
D4
D5
end
subgraph FenceGroup["3. Fence机制原子性保证"]
E1
E2
E3
E4
E5
end
subgraph UpdateGroup["4. 更新TabletData"]
F1
F2
F3
end
subgraph CleanupGroup["5. 清理旧版本"]
G1
G2
G3
end
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Check fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style ConditionStart fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ConditionGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ConditionCheck fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style PrepareStart fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style PrepareGroup fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style FenceStart fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style FenceGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style UpdateStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style UpdateGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CleanupStart fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style CleanupGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
Commit 流程包括以下步骤：
VersionCommitter 负责版本提交，定义在 framework/VersionCommitter.h 中：
// framework/VersionCommitter.h
class VersionCommitter
{
public:
// 提交版本
static std::pair<Status, VersionMeta> Commit(
const std::shared_ptr<TabletData>& tabletData,
const std::shared_ptr<config::ITabletSchema>& schema,
const CommitOptions& commitOptions);
};
Commit 的关键步骤：
flowchart TB
Start([Commit开始<br/>Commit Start]) --> PrepareLayer[准备阶段<br/>Preparation Phase]
subgraph PrepareGroup["准备版本信息 Prepare Version Information"]
direction TB
P1[准备版本信息<br/>Prepare Version Information]
P2[收集Segment列表<br/>Collect Segment List<br/>CreateSlice ST_BUILT]
P3[准备Locator<br/>Prepare Locator<br/>最新数据处理位置]
P1 --> P2
P2 --> P3
end
PrepareLayer --> FenceLayer[Fence机制阶段<br/>Fence Mechanism Phase]
subgraph FenceGroup["Fence机制原子性保证 Fence Mechanism Atomicity"]
direction TB
F1[创建Fence目录<br/>Create Fence Directory<br/>临时目录 version.fence]
F2[写入所有文件<br/>Write All Files<br/>Version Segment列表]
F3[原子重命名<br/>Atomic Rename<br/>rename操作]
F4[保证原子性<br/>Guarantee Atomicity<br/>要么全部成功要么全部失败]
F1 --> F2
F2 --> F3
F3 --> F4
end
FenceLayer --> WriteLayer[写入阶段<br/>Write Phase]
subgraph WriteGroup["写入Version文件 Write Version File"]
direction TB
W1[写入Version文件<br/>Write Version File<br/>版本信息 Segment列表 Locator]
end
WriteLayer --> AtomicLayer[原子切换阶段<br/>Atomic Switch Phase]
subgraph AtomicGroup["原子切换 Atomic Switch"]
direction TB
A1[原子切换<br/>Atomic Switch<br/>重命名为正式版本目录]
end
AtomicLayer --> UpdateLayer[更新阶段<br/>Update Phase]
subgraph UpdateGroup["更新TabletData Update TabletData"]
direction TB
U1[更新TabletData<br/>Update TabletData<br/>_onDiskVersion _segments]
end
UpdateLayer --> End([Commit完成<br/>Commit Complete])
PrepareLayer -.->|包含| PrepareGroup
FenceLayer -.->|包含| FenceGroup
WriteLayer -.->|包含| WriteGroup
AtomicLayer -.->|包含| AtomicGroup
UpdateLayer -.->|包含| UpdateGroup
P3 --> F1
F4 --> W1
W1 --> A1
A1 --> U1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style PrepareLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style FenceLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style WriteLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style AtomicLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style UpdateLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style PrepareGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style P1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style FenceGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style F1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style F4 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style WriteGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style W1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style AtomicGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style A1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style UpdateGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style U1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
Fence 机制保证版本提交的原子性：
flowchart TB
Start([开始提交<br/>Start Commit]) --> CreateLayer[创建Fence目录阶段<br/>Create Fence Directory Phase]
subgraph CreateGroup["创建Fence目录 Create Fence Directory"]
direction TB
C1[创建Fence目录<br/>Create Fence Directory<br/>临时目录version.fence]
end
CreateLayer --> WriteLayer[写入阶段<br/>Write Phase]
subgraph WriteGroup["写入Version文件 Write Version File"]
direction TB
W1[写入Version文件<br/>Write Version File<br/>版本信息 Segment列表 Locator]
end
WriteLayer --> SwitchLayer[原子切换阶段<br/>Atomic Switch Phase]
subgraph SwitchGroup["原子切换 Atomic Switch"]
direction TB
S1[原子切换<br/>Atomic Switch<br/>rename操作]
S2[重命名为正式版本<br/>Rename to Official Version<br/>version.fence → version_N]
S1 --> S2
end
SwitchLayer --> UpdateLayer[更新阶段<br/>Update Phase]
subgraph UpdateGroup["更新TabletData Update TabletData"]
direction TB
U1[更新TabletData<br/>Update TabletData<br/>切换到新版本]
end
UpdateLayer --> AtomicLayer[原子性保证阶段<br/>Atomicity Guarantee Phase]
subgraph AtomicGroup["原子性保证 Atomicity Guarantee"]
direction TB
A1[临时目录<br/>Temporary Directory<br/>version.fence]
A2[写入所有文件<br/>Write All Files<br/>Version Segment列表]
A3[原子重命名<br/>Atomic Rename<br/>rename操作]
A4[要么全部成功<br/>要么全部失败<br/>All or Nothing]
A1 --> A2
A2 --> A3
A3 --> A4
end
AtomicLayer --> End([提交完成<br/>Commit Complete])
CreateLayer -.->|包含| CreateGroup
WriteLayer -.->|包含| WriteGroup
SwitchLayer -.->|包含| SwitchGroup
UpdateLayer -.->|包含| UpdateGroup
AtomicLayer -.->|包含| AtomicGroup
C1 --> W1
W1 --> S1
S2 --> U1
U1 --> A1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style CreateLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style WriteLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style SwitchLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style UpdateLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style AtomicLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style CreateGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style C1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style WriteGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style W1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style SwitchGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style S1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style S2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style UpdateGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style U1 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style AtomicGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style A1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style A4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
Create -.->|使用| Atomic
Switch -.->|完成| Atomic
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Create fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Write fill:#fff3e0,stroke:#f57c00,stroke-width:1px
style Switch fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Rename fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
style Update fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Atomic fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px ```
Fence 机制：
CommitOptions 控制提交行为，定义在 framework/CommitOptions.h 中：
// framework/CommitOptions.h
struct CommitOptions
{
// 是否强制提交（即使没有数据变更）
bool forceCommit = false;
// 提交的描述信息
std::string commitMessage;
// 是否等待转储完成
bool waitDumpFinish = true;
// 是否清理旧版本
bool cleanVersion = false;
// 保留的版本列表
std::vector<versionid_t> reservedVersions;
};
提交选项的作用：
每次 Commit 都会创建新版本，版本号递增：
flowchart TB
Start([版本演进流程<br/>Version Evolution Flow]) --> V1Layer[Version 1 层<br/>Version 1 Layer]
subgraph V1Group["Version 1 版本信息"]
direction TB
V1_ID[versionId: 1<br/>版本号1]
V1_SEG[Segment 1,2<br/>索引段1和2]
V1_LOC[Locator timestamp=100<br/>处理位置时间戳100]
V1_ID --> V1_SEG
V1_SEG --> V1_LOC
end
V1Layer --> Commit1Layer[Commit 操作层<br/>Commit Operation Layer]
subgraph Commit1Group["Commit 操作 Commit Operation"]
direction TB
C1[Commit操作<br/>Commit Operation<br/>提交新版本]
end
Commit1Layer --> V2Layer[Version 2 层<br/>Version 2 Layer]
subgraph V2Group["Version 2 版本信息"]
direction TB
V2_ID[versionId: 2<br/>版本号2]
V2_SEG[Segment 1,2,3<br/>新增Segment 3]
V2_LOC[Locator timestamp=200<br/>处理位置时间戳200]
V2_ID --> V2_SEG
V2_SEG --> V2_LOC
end
V2Layer --> Commit2Layer[Commit 操作层<br/>Commit Operation Layer]
subgraph Commit2Group["Commit 操作 Commit Operation"]
direction TB
C2[Commit操作<br/>Commit Operation<br/>提交新版本]
end
Commit2Layer --> V3Layer[Version 3 层<br/>Version 3 Layer]
subgraph V3Group["Version 3 版本信息"]
direction TB
V3_ID[versionId: 3<br/>版本号3]
V3_SEG[Segment 4<br/>合并后的Segment 4]
V3_LOC[Locator timestamp=300<br/>处理位置时间戳300]
V3_ID --> V3_SEG
V3_SEG --> V3_LOC
end
V3Layer --> EvolutionLayer[版本演进特点层<br/>Version Evolution Features Layer]
subgraph EvolutionGroup["版本演进特点 Version Evolution Features"]
direction TB
E1[版本号递增<br/>VersionId Monotonic Increase<br/>versionId单调递增]
E2[Segment列表变化<br/>Segment List Changes<br/>新增或合并Segment]
E3[Locator更新<br/>Locator Update<br/>记录最新处理位置]
E1 --> E2
E2 --> E3
end
EvolutionLayer --> End([版本演进完成<br/>Version Evolution Complete])
V1Layer -.->|包含| V1Group
Commit1Layer -.->|包含| Commit1Group
V2Layer -.->|包含| V2Group
Commit2Layer -.->|包含| Commit2Group
V3Layer -.->|包含| V3Group
EvolutionLayer -.->|包含| EvolutionGroup
V1Group -.->|提交| Commit1Group
Commit1Group -.->|创建| V2Group
V2Group -.->|提交| Commit2Group
Commit2Group -.->|创建| V3Group
V3Group -.->|展示| EvolutionGroup
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style V1Layer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style Commit1Layer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style V2Layer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style Commit2Layer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style V3Layer fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style EvolutionLayer fill:#fff9c4,stroke:#f9a825,stroke-width:3px
style V1Group fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style V1_ID fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V1_SEG fill:#90caf9,stroke:#1976d2,stroke-width:2px
style V1_LOC fill:#90caf9,stroke:#1976d2,stroke-width:2px
style Commit1Group fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V2Group fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style V2_ID fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style V2_SEG fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style V2_LOC fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Commit2Group fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style V3Group fill:#e8f5e9,stroke:#2e7d32,stroke-width:3px
style V3_ID fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style V3_SEG fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style V3_LOC fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style EvolutionGroup fill:#fff9c4,stroke:#f9a825,stroke-width:3px
style E1 fill:#ffe082,stroke:#f9a825,stroke-width:2px
style E2 fill:#ffe082,stroke:#f9a825,stroke-width:2px
style E3 fill:#ffe082,stroke:#f9a825,stroke-width:2px
V3Content -.->|演进特点| Evolution
style V1 fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style V1Content fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style Commit1 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style V2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style V2Content fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style Commit2 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style V3 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style V3Content fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Evolution fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px ```
版本演进示例：
在实时写入场景中，完整的构建流程：
graph LR
A[持续Build] --> B[文档写入MemSegment]
B --> C{达到阈值?}
C -->|是| D[定期Flush]
C -->|否| A
D --> E[转储为DiskSegment]
E --> F[创建新MemSegment]
F --> A
E --> G[定期Seal]
G --> H[定期Commit]
H --> I[更新Version]
style A fill:#e3f2fd
style D fill:#fff3e0
style G fill:#e8f5e9
style H fill:#f3e5f5
流程示例：
在批量构建场景中，完整的构建流程：
flowchart TD
Start[批量构建场景] --> ProcessLayer[构建流程层]
ProcessLayer --> CharacterLayer[场景特点层]
subgraph ProcessGroup["批量构建流程：一次性完成所有操作"]
direction TB
P1[批量Build<br/>一次性构建大量文档<br/>接收所有文档批次]
P2[Flush转储<br/>构建完成后触发Flush<br/>转储MemSegment]
P3[转储为DiskSegment<br/>创建DiskSegment<br/>加载转储后的Segment]
P4[Seal所有Segment<br/>封存所有Segment<br/>标记为只读]
P5[Commit最终版本<br/>提交最终版本<br/>更新Version到磁盘]
P1 --> P2
P2 --> P3
P3 --> P4
P4 --> P5
end
ProcessLayer --> ProcessGroup
ProcessGroup --> CharacterLayer
subgraph CharacterGroup["批量场景特点：一次性处理"]
direction TB
C1[一次性构建<br/>所有文档一次性构建完成<br/>不进行增量构建]
C2[完成后转储<br/>构建完成后统一转储<br/>不进行中间转储]
C3[一次性提交<br/>所有操作完成后提交<br/>不进行多次提交]
C4[适合离线场景<br/>批量导入数据<br/>全量索引构建]
end
CharacterLayer --> CharacterGroup
CharacterGroup --> End[批量构建完成]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ProcessLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ProcessGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style P1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style P2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style P4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style P5 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style CharacterLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style CharacterGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style C1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style C2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style C3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style C4 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
流程示例：
IndexLib 的构建流程支持异步和并发：
flowchart TD
Start[异步与并发设计] --> AsyncLayer[异步处理层]
Start --> ConcurrentLayer[并发处理层]
subgraph AsyncGroup["异步处理层：异步转储机制"]
direction TB
A1[触发转储<br/>创建SegmentDumper]
A2[提交转储任务<br/>提交到后台线程池]
A3[立即返回<br/>不阻塞写入操作]
A4[后台线程执行转储<br/>异步转储到磁盘]
A5[转储完成回调<br/>更新Segment状态]
A1 --> A2
A2 --> A3
A3 --> A4
A4 --> A5
end
AsyncLayer --> AsyncGroup
subgraph ConcurrentBuildGroup["并发处理层：并发构建"]
direction TB
CB1[接收文档批次<br/>IDocumentBatch]
CB2[创建并行构建器<br/>NormalTabletParallelBuilder]
CB3[多线程并行构建<br/>线程池处理文档]
CB4[并行写入Indexer<br/>倒排/正排/主键索引]
CB5[合并构建结果<br/>汇总各线程结果]
CB1 --> CB2
CB2 --> CB3
CB3 --> CB4
CB4 --> CB5
end
subgraph ConcurrentDumpGroup["并发处理层：并发转储"]
direction TB
CD1[多个Segment待转储<br/>收集需要转储的Segment]
CD2[DumpControl控制并发度<br/>限制同时转储的Segment数]
CD3[并发转储任务<br/>多个Segment并行转储]
CD4[充分利用IO资源<br/>磁盘IO并行处理]
CD5[转储完成<br/>所有Segment转储完成]
CD1 --> CD2
CD2 --> CD3
CD3 --> CD4
CD4 --> CD5
end
ConcurrentLayer --> ConcurrentBuildGroup
ConcurrentLayer --> ConcurrentDumpGroup
AsyncGroup --> Performance[性能优势]
ConcurrentBuildGroup --> Performance
ConcurrentDumpGroup --> Performance
subgraph PerformanceGroup["性能优势：提升整体性能"]
direction TB
P1[提高吞吐量<br/>并行处理提高整体吞吐<br/>充分利用多核CPU]
P2[充分利用资源<br/>CPU和IO并行利用<br/>避免资源闲置]
P3[降低延迟<br/>异步转储不阻塞写入<br/>提升写入响应速度]
end
Performance --> PerformanceGroup
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style AsyncLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style AsyncGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style A2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style A5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ConcurrentLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ConcurrentBuildGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CB1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style CB2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CB3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style CB4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style CB5 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style ConcurrentDumpGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CD1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style CD2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style CD3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style CD4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style CD5 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Performance fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style PerformanceGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style P1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style P2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style P3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
异步与并发设计：
构建流程需要严格控制内存使用：
flowchart TD
Start[内存管理机制] --> MonitorLayer[内存监控层]
MonitorLayer --> ControlLayer[内存控制层]
ControlLayer --> ReleaseLayer[内存释放层]
subgraph MonitorGroup["内存监控层：实时监控内存使用"]
direction TB
M1[构建前估算<br/>EstimateMemUsed方法]
M2[根据Schema估算<br/>索引字段类型和数量]
M3[根据文档数估算<br/>预期文档数量]
M4[构建中评估<br/>EvaluateCurrentMemUsed方法]
M5[实时监控内存使用<br/>统计实际内存占用]
M1 --> M2
M2 --> M3
M3 --> M4
M4 --> M5
end
MonitorLayer --> MonitorGroup
subgraph ControlGroup["内存控制层：MemoryQuotaController"]
direction TB
C1[检查内存配额<br/>查询可用内存配额]
C2{配额是否充足?}
C3[分配内存配额<br/>从MemoryQuotaController分配]
C4[拒绝分配<br/>等待或拒绝写入]
C5[控制内存上限<br/>设置总内存配额]
C1 --> C2
C2 -->|充足| C3
C2 -->|不足| C4
C3 --> C5
C4 --> C1
end
MonitorGroup --> ControlGroup
subgraph TriggerGroup["触发转储：达到阈值时释放内存"]
direction TB
T1[检查转储条件<br/>内存使用/文档数/时间间隔]
T2{是否达到阈值?}
T3[内存超阈值<br/>当前内存使用超过限制]
T4[文档数超阈值<br/>文档数量超过限制]
T5[时间间隔达到<br/>达到转储时间间隔]
T1 --> T2
T2 -->|是| T3
T2 -->|是| T4
T2 -->|是| T5
T2 -->|否| T1
end
ControlGroup --> TriggerGroup
TriggerGroup --> ReleaseLayer
subgraph ReleaseGroup["内存释放层：转储释放内存"]
direction TB
R1[触发转储操作<br/>创建SegmentDumper]
R2[转储MemSegment<br/>异步转储到磁盘]
R3[释放内存配额<br/>释放MemSegment内存]
R4[创建新MemSegment<br/>继续构建]
R1 --> R2
R2 --> R3
R3 --> R4
R4 --> MonitorGroup
end
ReleaseLayer --> ReleaseGroup
ReleaseGroup --> Result[内存管理目标<br/>保证系统稳定性<br/>避免内存溢出<br/>及时释放内存]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style MonitorLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style MonitorGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style M1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style M2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style M3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style M4 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style M5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ControlLayer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style ControlGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#ffebee,stroke:#c62828,stroke-width:1px
style C5 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style TriggerGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style T1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style T2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style T3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style T4 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style T5 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ReleaseLayer fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style ReleaseGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style R1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style R2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style R3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style R4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Result fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
内存管理机制：
MemoryQuotaController 控制内存上限构建流程需要完善的错误处理：
flowchart TD
Start[错误处理机制] --> ErrorDetection[错误检测层]
ErrorDetection --> ErrorHandling[错误处理层]
ErrorHandling --> ErrorRecovery[错误恢复层]
subgraph ErrorDetectionGroup["错误检测层：及时发现错误"]
direction TB
ED1[构建错误检测<br/>检测构建过程中的异常]
ED2[转储错误检测<br/>检测转储过程中的异常]
ED3[版本提交错误检测<br/>检测版本提交过程中的异常]
end
ErrorDetection --> ErrorDetectionGroup
subgraph RetryGroup["1. 重试机制：构建失败处理"]
direction TB
R1[检测构建错误<br/>捕获异常和错误码]
R2[判断是否可重试<br/>检查错误类型和重试次数]
R3[自动重试构建<br/>重新执行构建操作]
R4[记录重试信息<br/>记录重试次数和错误详情]
R1 --> R2
R2 -->|可重试| R3
R2 -->|不可重试| R5[抛出错误]
R3 --> R4
R4 --> R2
end
subgraph RollbackGroup["2. 回滚机制：转储失败处理"]
direction TB
RB1[检测转储错误<br/>捕获转储异常]
RB2[保存当前状态<br/>记录转储前的状态]
RB3[回滚到稳定状态<br/>恢复到上一个成功版本]
RB4[清理失败文件<br/>删除失败的转储文件]
RB1 --> RB2
RB2 --> RB3
RB3 --> RB4
end
subgraph AtomicityGroup["3. 原子性保证：版本提交处理"]
direction TB
A1[创建Fence临时目录<br/>version.fence]
A2[写入版本文件<br/>Version和Segment信息]
A3[原子重命名操作<br/>rename临时目录为正式版本]
A4[验证提交结果<br/>检查是否全部成功]
A1 --> A2
A2 --> A3
A3 --> A4
A4 -->|失败| A5[清理临时目录<br/>保证原子性]
A4 -->|成功| A6[提交完成]
end
ErrorDetectionGroup --> RetryGroup
ErrorDetectionGroup --> RollbackGroup
ErrorDetectionGroup --> AtomicityGroup
RetryGroup --> ErrorHandling
RollbackGroup --> ErrorHandling
AtomicityGroup --> ErrorHandling
ErrorHandling --> ErrorRecovery
subgraph RecoveryGroup["错误恢复层：保证系统稳定"]
direction TB
Recovery1[数据一致性保证<br/>保证数据完整性<br/>避免部分写入<br/>保证版本一致性]
Recovery2[系统稳定性保证<br/>快速恢复服务<br/>避免数据丢失<br/>保证服务可用性]
end
ErrorRecovery --> RecoveryGroup
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ErrorDetection fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ErrorDetectionGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ED1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ED2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ED3 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ErrorHandling fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style RetryGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style R1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style R2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style R3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style R4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style R5 fill:#ffebee,stroke:#c62828,stroke-width:1px
style RollbackGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style RB1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style RB2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style RB3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style RB4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style AtomicityGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style A1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style A2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style A3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style A4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style A5 fill:#ffebee,stroke:#c62828,stroke-width:1px
style A6 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style ErrorRecovery fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style RecoveryGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Recovery1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style Recovery2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
错误处理机制：
构建性能优化的关键点：
flowchart TD
Start[构建性能优化] --> StrategyLayer[优化策略层]
StrategyLayer --> EffectLayer[优化效果层]
subgraph BatchGroup["1. 批量写入优化"]
direction TB
B1[批量接收文档<br/>IDocumentBatch]
B2[批量处理文档<br/>减少函数调用次数]
B3[批量写入Indexer<br/>减少索引更新开销]
B4[减少调用开销<br/>降低系统调用成本]
B1 --> B2
B2 --> B3
B3 --> B4
end
subgraph ParallelGroup["2. 并行构建优化"]
direction TB
P1[多线程并行构建<br/>NormalTabletParallelBuilder]
P2[并行处理文档批次<br/>充分利用多核CPU]
P3[并行写入索引<br/>倒排/正排/主键索引]
P4[提高构建速度<br/>缩短构建时间]
P1 --> P2
P2 --> P3
P3 --> P4
end
subgraph MemoryGroup["3. 内存优化"]
direction TB
M1[优化内存分配<br/>减少内存分配次数]
M2[内存池管理<br/>复用内存对象]
M3[减少内存拷贝<br/>使用移动语义]
M4[减少内存分配开销<br/>降低GC压力]
M1 --> M2
M2 --> M3
M3 --> M4
end
StrategyLayer --> BatchGroup
StrategyLayer --> ParallelGroup
StrategyLayer --> MemoryGroup
BatchGroup --> EffectLayer
ParallelGroup --> EffectLayer
MemoryGroup --> EffectLayer
subgraph EffectGroup["优化效果：提升整体性能"]
direction TB
E1[提高吞吐量<br/>单位时间处理更多文档<br/>提升整体处理能力]
E2[降低延迟<br/>减少单次操作耗时<br/>提升响应速度]
E3[提高资源利用率<br/>充分利用CPU和内存<br/>提升系统效率]
end
EffectLayer --> EffectGroup
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style BatchGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style B1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style B2 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style B3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style B4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ParallelGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style P1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style P2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style P4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style MemoryGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style M1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style M2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style M3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style M4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style EffectLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style EffectGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style E1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style E2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style E3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
优化策略：
转储性能优化的关键点：
flowchart TD
Start[转储性能优化] --> StrategyLayer[优化策略层]
StrategyLayer --> EffectLayer[优化效果层]
subgraph AsyncGroup["1. 异步转储优化"]
direction TB
A1[触发转储操作<br/>创建SegmentDumper]
A2[提交转储任务<br/>提交到后台线程池]
A3[立即返回<br/>不阻塞写入操作]
A4[后台线程执行转储<br/>异步转储到磁盘]
A5[不阻塞写入<br/>写入和转储并行进行]
A1 --> A2
A2 --> A3
A3 --> A4
A4 --> A5
end
subgraph ConcurrentGroup["2. 并发转储优化"]
direction TB
C1[收集待转储Segment<br/>多个Segment需要转储]
C2[DumpControl控制并发度<br/>限制同时转储的Segment数]
C3[并发转储任务<br/>多个Segment并行转储]
C4[多个Segment并发<br/>充分利用IO资源]
C5[转储完成<br/>所有Segment转储完成]
C1 --> C2
C2 --> C3
C3 --> C4
C4 --> C5
end
subgraph IOGroup["3. IO优化"]
direction TB
IO1[批量IO操作<br/>减少系统调用次数]
IO2[顺序写入优化<br/>减少磁盘寻道时间]
IO3[压缩优化<br/>减少IO数据量]
IO4[减少IO开销<br/>提高IO效率]
IO1 --> IO2
IO2 --> IO3
IO3 --> IO4
end
StrategyLayer --> AsyncGroup
StrategyLayer --> ConcurrentGroup
StrategyLayer --> IOGroup
AsyncGroup --> EffectLayer
ConcurrentGroup --> EffectLayer
IOGroup --> EffectLayer
subgraph EffectGroup["优化效果：提升转储性能"]
direction TB
E1[提高吞吐量<br/>单位时间转储更多数据<br/>提升整体转储能力]
E2[降低延迟<br/>减少转储对写入的影响<br/>提升写入响应速度]
E3[提高IO效率<br/>充分利用磁盘IO资源<br/>提升转储速度]
end
EffectLayer --> EffectGroup
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style StrategyLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style AsyncGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style A1 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style A2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A3 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style A4 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style A5 fill:#c5e1f5,stroke:#1976d2,stroke-width:1px
style ConcurrentGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style C1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C4 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style C5 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style IOGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style IO1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style IO2 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style IO3 fill:#a5d6a7,stroke:#2e7d32,stroke-width:2px
style IO4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style EffectLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style EffectGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style E1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style E2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style E3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
优化策略：
优化策略：
优化策略：
优化策略：
索引构建流程是 IndexLib 的核心功能，包括 Build、Flush、Seal、Commit 四个阶段。通过本文的深入解析，我们了解到：
核心流程：
设计亮点：
性能优化：
理解索引构建流程，是掌握 IndexLib 索引机制的关键。在下一篇文章中，我们将深入介绍查询流程的实现细节，包括 TabletReader、IndexReader、查询解析、结果合并等各个组件的实现原理和性能优化策略。
]]>Tablet 和 Segment 的组织关系是 IndexLib 索引机制的核心。让我们通过类图来理解它们的关系：
classDiagram
class Tablet {
- TabletData _tabletData
- TabletSchema _schema
- TabletOptions _options
+ Open(string) Status
+ Build(IDocumentBatch) Status
+ Flush() Status
+ Seal() void
+ Commit() Status
+ GetTabletReader() TabletReader
+ Reopen() Status
}
class TabletData {
- Version _onDiskVersion
- vector~shared_ptr~Segment~~ _segments
- shared_ptr~ResourceMap~ _resourceMap
+ CreateSlice(SegmentStatus) vector~Segment~
+ GetSegment(segmentid_t) Segment
+ GetSegmentWithBaseDocid(docid_t) Segment
+ UpdateVersion(Version) void
+ GetSegmentCount() size_t
}
class Segment {
<<abstract>>
# segmentid_t _segmentId
# SegmentStatus _status
+ GetSegmentId() segmentid_t
+ GetDocCount() uint32_t
+ GetSegmentStatus() SegmentStatus
+ GetIndexer(string) IIndexer
+ GetBaseDocId() docid_t
}
class MemSegment {
- map~string,IIndexer~ _indexers
+ Build(IDocumentBatch) Status
+ NeedDump() bool
+ CreateSegmentDumpItems() vector~SegmentDumpItem~
+ Seal() void
+ EvaluateCurrentMemUsed() size_t
}
class DiskSegment {
- map~string,IIndexer~ _indexers
+ Open(string) Status
+ Reopen() Status
+ GetIndexer(string) IIndexer
}
Tablet "1" --> "1" TabletData : 管理
TabletData "1" *-- "many" Segment : 包含多个有序Segment
Segment <|-- MemSegment : 继承
Segment <|-- DiskSegment : 继承
note for Tablet "索引表的完整抽象<br/>管理索引的构建和查询"
note for TabletData "管理Segment列表和版本<br/>提供Segment访问接口"
note for Segment "抽象基类<br/>定义Segment通用接口"
note for MemSegment "内存段<br/>实时写入和构建"
note for DiskSegment "磁盘段<br/>持久化存储和查询"
组织关系：
Tablet 是索引表的完整抽象，而 Segment 是索引的基本存储单元。一个 Tablet 包含多个 Segment，这些 Segment 按照时间顺序组织，共同构成完整的索引。
通过阅读源码，我们可以看到 Tablet 和 Segment 的关系定义在 framework/TabletData.h 中：
// framework/TabletData.h
class TabletData : private autil::NoCopyable
{
private:
Version _onDiskVersion; // 磁盘版本
std::vector<std::shared_ptr<Segment>> _segments; // Segment 列表（有序）
std::shared_ptr<ResourceMap> _resourceMap; // 共享资源
};
关键设计：
_segments 是一个有序的 Segment 列表，按照 SegmentId 排序_onDiskVersion 记录哪些 Segment 已持久化ResourceMap（内存池、缓存等）Segment 的 ID 分配有特殊的规则，定义在 framework/Segment.h 中：
// framework/Segment.h
class Segment {
public:
// Segment ID 的掩码定义
static constexpr segmentid_t RT_SEGMENT_ID_MASK = (segmentid_t)0x1 << 30; // 实时 Segment
static constexpr segmentid_t MERGED_SEGMENT_ID_MASK = (segmentid_t)0x0; // 合并 Segment
static constexpr segmentid_t PUBLIC_SEGMENT_ID_MASK = (segmentid_t)0x1 << 29; // 公共 Segment
static constexpr segmentid_t PRIVATE_SEGMENT_ID_MASK = (segmentid_t)0x1 << 30; // 私有 Segment
// 判断 Segment 类型
static bool IsRtSegmentId(segmentid_t segId) {
return (segId & RT_SEGMENT_ID_MASK) > 0;
}
static bool IsMergedSegmentId(segmentid_t segId) {
return segId != INVALID_SEGMENTID &&
(segId & (PUBLIC_SEGMENT_ID_MASK | PRIVATE_SEGMENT_ID_MASK)) == 0;
}
};
Segment ID 的分类：
Segment ID 采用位掩码机制，通过不同的位来区分 Segment 的类型和属性。这种设计使得 ID 分配和类型判断都非常高效：
graph TD
A[Segment ID: 32位整数] --> B{检查第30位}
B -->|第30位=1| C[RT Segment<br/>实时Segment]
B -->|第30位=0| D{检查第29位}
D -->|第29位=1| E[Public Segment<br/>公共Segment]
D -->|第29位=0| F[Merged Segment<br/>合并Segment]
C --> G[用于实时写入]
E --> H[用于公共数据]
F --> I[用于合并后的数据]
style C fill:#e3f2fd
style E fill:#fff3e0
style F fill:#e8f5e9
Segment ID 分配规则：
0x40000000），用于实时写入
0x20000000），用于公共数据设计优势：
SegmentMeta 记录 Segment 的元数据信息，定义在 framework/SegmentMeta.h 中：
// framework/SegmentMeta.h
struct SegmentMeta {
segmentid_t segmentId; // Segment ID
std::shared_ptr<indexlib::file_system::Directory> segmentDir; // Segment 目录
std::shared_ptr<SegmentInfo> segmentInfo; // Segment 信息
std::shared_ptr<indexlib::framework::SegmentMetrics> segmentMetrics; // Segment 指标
std::shared_ptr<config::ITabletSchema> schema; // Schema
std::string lifecycle; // 生命周期标签
};
SegmentMeta 的组成：
SegmentMeta 是 Segment 的元数据容器，包含了 Segment 的所有元信息。让我们通过类图来理解其结构：
classDiagram
class SegmentMeta {
+ segmentid_t segmentId
+ Directory segmentDir
+ SegmentInfo segmentInfo
+ SegmentMetrics segmentMetrics
+ ITabletSchema schema
+ string lifecycle
}
class SegmentInfo {
+ uint64_t docCount
+ int64_t timestamp
+ schemaid_t schemaId
+ Locator locator
+ uint32_t shardId
+ bool mergedSegment
}
class Directory {
+ CreateFileReader()
+ CreateFileWriter()
+ ListDir()
}
class SegmentMetrics {
+ map_string_double metrics
+ GetMetric()
}
SegmentMeta --> SegmentInfo : 包含
SegmentMeta --> Directory : 使用
SegmentMeta --> SegmentMetrics : 包含
字段详解：
设计原理：
SegmentInfo 记录 Segment 的详细信息，定义在 framework/SegmentInfo.h 中：
// framework/SegmentInfo.h
class SegmentInfo : public autil::legacy::Jsonizable
{
public:
// 基本信息
volatile uint64_t docCount = 0; // 文档数量
int64_t timestamp = INVALID_TIMESTAMP; // 时间戳
schemaid_t schemaId = DEFAULT_SCHEMAID; // Schema ID
// Locator 信息
Locator GetLocator() const;
void SetLocator(const Locator& locator);
// 分片信息
uint32_t shardId = INVALID_SHARDING_ID; // 分片 ID
uint32_t shardCount = 1; // 分片数量
// 其他信息
bool mergedSegment = false; // 是否合并 Segment
uint32_t maxTTL = 0; // 最大 TTL
std::map<std::string, std::string> descriptions; // 描述信息
};
SegmentInfo 的关键字段：
flowchart TD
A[SegmentInfo<br/>Segment元数据信息] --> B[基础信息]
A --> C[位置信息]
A --> D[分片信息]
A --> E[状态信息]
subgraph Basic["基础信息"]
B1[segmentId<br/>Segment唯一标识]
B2[directory<br/>目录路径]
B3[schemaId<br/>Schema版本]
B4[docCount<br/>文档数量<br/>用于DocId映射]
B --> B1
B --> B2
B --> B3
B --> B4
end
subgraph Location["位置信息"]
C1[Locator<br/>数据位置信息<br/>用于增量更新]
C2[timestamp<br/>时间戳]
C3[concurrentIdx<br/>并发索引]
C --> C1
C1 --> C2
C1 --> C3
end
subgraph Shard["分片信息"]
D1[shardId<br/>当前分片ID]
D2[shardCount<br/>总分片数]
D3[支持分片存储<br/>水平扩展]
D --> D1
D --> D2
D1 --> D3
D2 --> D3
end
subgraph Status["状态信息"]
E1[mergedSegment<br/>合并标识<br/>是否为合并Segment]
E2[segmentStatus<br/>Segment状态<br/>ST_BUILT/ST_BUILDING等]
E --> E1
E --> E2
end
style Basic fill:#e3f2fd
style Location fill:#fff3e0
style Shard fill:#f3e5f5
style Status fill:#e8f5e9
IndexLib 使用两级 DocId 机制：
DocId 映射关系：
IndexLib 使用两级 DocId 机制，这是理解索引查询和构建的关键。让我们通过流程图来理解 DocId 的映射关系：
flowchart TD
subgraph Write["写入路径：分配DocId"]
W1[文档写入<br/>IDocumentBatch]
W2[获取当前MemSegment<br/>_normalBuildingSegment]
W3[获取BaseDocId<br/>前面所有Segment的docCount之和]
W4[分配LocalDocId<br/>从0开始递增]
W5[计算GlobalDocId<br/>GlobalDocId = BaseDocId + LocalDocId]
W6[写入Indexer<br/>使用GlobalDocId]
W1 --> W2
W2 --> W3
W3 --> W4
W4 --> W5
W5 --> W6
end
subgraph Query["查询路径：转换DocId"]
Q1[查询请求<br/>GlobalDocId]
Q2[遍历TabletData中的Segment]
Q3[计算每个Segment的BaseDocId<br/>累加前面Segment的docCount]
Q4{GlobalDocId在范围内?<br/>BaseDocId <= GlobalDocId < BaseDocId + docCount}
Q5[计算LocalDocId<br/>LocalDocId = GlobalDocId - BaseDocId]
Q6[在Segment内查询<br/>使用LocalDocId]
Q7[返回查询结果]
Q1 --> Q2
Q2 --> Q3
Q3 --> Q4
Q4 -->|是| Q5
Q4 -->|否| Q2
Q5 --> Q6
Q6 --> Q7
end
subgraph Example["示例：3个Segment"]
E1[Segment1: docCount=1000<br/>BaseDocId=0<br/>GlobalDocId范围: 0-999]
E2[Segment2: docCount=2000<br/>BaseDocId=1000<br/>GlobalDocId范围: 1000-2999]
E3[Segment3: docCount=1500<br/>BaseDocId=3000<br/>GlobalDocId范围: 3000-4499]
E1 --> E2
E2 --> E3
end
style Write fill:#e3f2fd
style Query fill:#fff3e0
style Example fill:#f5f5f5
DocId 映射示例：
假设有 3 个 Segment：
那么：
从代码中可以看到，TabletData 提供了获取 Segment 及其基础 DocId 的方法：
// framework/TabletData.h
class TabletData {
public:
// 获取 Segment 及其基础 DocId
// 返回：(Segment 指针, 基础 DocId)
std::pair<SegmentPtr, docid64_t> GetSegmentWithBaseDocid(segmentid_t segmentId);
};
DocId 计算逻辑（通过代码分析）：
// 伪代码：计算全局 DocId
docid64_t globalDocId = baseDocId + localDocId;
// 其中：
// - baseDocId：前面所有 Segment 的文档数之和
// - localDocId：当前 Segment 内的局部 DocId
BaseDocId 是 Segment 的全局 DocId 起始值，等于前面所有 Segment 的文档数之和：
BaseDocId 计算流程：
sequenceDiagram
participant Writer as TabletWriter
participant TabletData as TabletData
participant Seg1 as Segment 1
participant Seg2 as Segment 2
participant Seg3 as Segment 3
Writer->>TabletData: GetSegmentWithBaseDocid(segId=1)
TabletData->>Seg1: GetDocCount()
Seg1-->>TabletData: 1000
TabletData-->>Writer: (Segment1, baseDocId=0)
Writer->>TabletData: GetSegmentWithBaseDocid(segId=2)
TabletData->>Seg1: GetDocCount()
Seg1-->>TabletData: 1000
TabletData->>Seg2: GetDocCount()
Seg2-->>TabletData: 2000
TabletData-->>Writer: (Segment2, baseDocId=1000)
Writer->>TabletData: GetSegmentWithBaseDocid(segId=3)
TabletData->>Seg1: GetDocCount()
Seg1-->>TabletData: 1000
TabletData->>Seg2: GetDocCount()
Seg2-->>TabletData: 2000
TabletData->>Seg3: GetDocCount()
Seg3-->>TabletData: 1500
TabletData-->>Writer: (Segment3, baseDocId=3000)
计算示例：
代码实现逻辑（通过阅读源码理解）：
// TabletData 内部维护 Segment 列表
// 计算 baseDocId 时，遍历前面的 Segment，累加 docCount
docid64_t baseDocId = 0;
for (auto& seg : _segments) {
if (seg->GetSegmentId() == segmentId) {
break;
}
baseDocId += seg->GetDocCount();
}
TabletData 通过 Init() 方法初始化 Segment 列表：
// framework/TabletData.h
class TabletData {
public:
// 初始化：设置版本和 Segment 列表
Status Init(Version onDiskVersion,
std::vector<SegmentPtr> segments,
const std::shared_ptr<ResourceMap>& resourceMap);
};
Segment 列表的维护：
flowchart TD
Start[TabletData<br/>管理Segment列表] --> Operations{操作类型}
Operations --> Init[Init<br/>初始化]
Operations --> Add[AddSegment<br/>添加Segment]
Operations --> Remove[RemoveSegment<br/>移除Segment]
Operations --> Reopen[Reopen<br/>更新Segment列表]
Init --> InitDetail[设置初始Segment列表<br/>设置Version和ResourceMap<br/>建立Segment有序列表]
Add --> AddDetail[新Segment通过TabletWriter创建<br/>添加到_segments列表末尾<br/>保持SegmentId有序]
Remove --> RemoveDetail[合并后移除旧Segment<br/>从_segments列表中删除<br/>释放Segment资源]
Reopen --> ReopenDetail[加载新Version<br/>更新Segment列表<br/>重新建立Segment视图]
InitDetail --> End[Segment列表已更新]
AddDetail --> End
RemoveDetail --> End
ReopenDetail --> End
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Operations fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Init fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Add fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Remove fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Reopen fill:#fce4ec,stroke:#c2185b,stroke-width:2px
style InitDetail fill:#fff9c4,stroke:#f57f17,stroke-width:1px
style AddDetail fill:#c8e6c9,stroke:#388e3c,stroke-width:1px
style RemoveDetail fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style ReopenDetail fill:#f8bbd0,stroke:#c2185b,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
Init() 设置初始 Segment 列表TabletWriter 创建后添加到列表Reopen() 时更新 Segment 列表Slice 是 TabletData 提供的 Segment 视图机制，可以按状态筛选 Segment：
// framework/TabletData.h
class TabletData {
public:
class Slice {
// 提供迭代器，可以遍历筛选后的 Segment
auto begin() { return _cBegin; }
auto end() { return _cEnd; }
auto rbegin() { return _cRbegin; }
auto rend() { return _cRend; }
};
// 创建 Slice：按状态筛选
Slice CreateSlice(Segment::SegmentStatus segmentStatus) const;
};
Slice 的使用场景：
Slice 机制是 TabletData 的核心设计，提供了灵活的 Segment 筛选能力。让我们通过流程图来理解不同场景下的使用：
graph TD
A[TabletData] --> B[CreateSlice]
B --> C{使用场景}
C -->|查询| D[ST_BUILT]
C -->|写入| E[ST_BUILDING]
C -->|合并| F[ST_BUILT]
C -->|监控| G[ST_DUMPING]
C -->|全部| H[无筛选]
D --> I[获取所有已构建的Segment<br/>用于查询]
E --> J[获取构建中的Segment<br/>用于写入]
F --> K[获取需要合并的Segment<br/>用于合并]
G --> L[获取转储中的Segment<br/>用于监控]
H --> M[获取所有Segment<br/>用于管理]
style D fill:#e3f2fd
style E fill:#fff3e0
style F fill:#f3e5f5
style G fill:#e8f5e9
使用场景详解：
CreateSlice(ST_BUILT) 获取所有已构建的 Segment
CreateSlice(ST_BUILDING) 获取构建中的 Segment
CreateSlice(ST_BUILT) 获取需要合并的 Segment
CreateSlice(ST_DUMPING) 获取转储中的 Segment
设计优势：
通过阅读 table/normal_table/NormalMemSegment.h，我们可以看到 NormalMemSegment 的实现：
// table/normal_table/NormalMemSegment.h
class NormalMemSegment : public plain::PlainMemSegment
{
public:
NormalMemSegment(const config::TabletOptions* options,
const std::shared_ptr<config::ITabletSchema>& schema,
const framework::SegmentMeta& segmentMeta);
protected:
// 创建转储参数
std::pair<Status, std::shared_ptr<framework::DumpParams>> CreateDumpParams() override;
// 计算转储内存成本
void CalcMemCostInCreateDumpParams() override;
};
MemSegment 的构建流程：
MemSegment 的构建是索引写入的核心流程。让我们通过序列图来理解完整的构建过程：
sequenceDiagram
participant Writer as TabletWriter
participant MemSeg as MemSegment
participant Indexer1 as InvertedIndexer
participant Indexer2 as AttributeIndexer
participant MemCtrl as MemoryQuotaController
Writer->>MemSeg: Open(SegmentMeta, BuildResource)
MemSeg->>Indexer1: CreateIndexer(indexConfig)
MemSeg->>Indexer2: CreateIndexer(indexConfig)
MemSeg-->>Writer: Success
Writer->>MemSeg: Build(documentBatch)
MemSeg->>MemSeg: DispatchDocIds(batch)
MemSeg->>Indexer1: BuildDocument(doc, docId)
MemSeg->>Indexer2: BuildDocument(doc, docId)
Indexer1-->>MemSeg: Success
Indexer2-->>MemSeg: Success
MemSeg->>MemSeg: UpdateSegmentInfo()
MemSeg-->>Writer: Success
Writer->>MemSeg: NeedDump()?
MemSeg->>MemCtrl: GetUsedQuota()
MemCtrl-->>MemSeg: usedQuota
MemSeg->>MemSeg: CheckThreshold(usedQuota)
MemSeg-->>Writer: true/false
alt NeedDump == true
Writer->>MemSeg: CreateDumpParams()
MemSeg->>MemSeg: CalcMemCost()
MemSeg->>MemSeg: PrepareDumpItems()
MemSeg-->>Writer: DumpParams
end
构建流程详解：
ST_BUILDINGMemSegment 在内存中构建索引，需要严格控制内存使用。关键代码（table/plain/PlainMemSegment.h）：
class PlainMemSegment : public MemSegment {
public:
// 估算内存使用
std::pair<Status, size_t> EstimateMemUsed(
const std::shared_ptr<config::ITabletSchema>& schema) override;
// 评估当前内存使用
size_t EvaluateCurrentMemUsed() override;
};
内存管理机制：
MemSegment 的内存管理是保证系统稳定性的关键。让我们通过流程图来理解内存管理的完整机制：
flowchart TD
Start[开始构建] --> Estimate[EstimateMemUsed<br/>估算内存需求]
Estimate --> QuotaCheck{内存配额检查<br/>检查可用配额}
QuotaCheck -->|配额不足| WaitOrReject[等待或拒绝<br/>等待配额释放或拒绝构建]
QuotaCheck -->|配额充足| Allocate[分配内存<br/>从MemoryQuotaController分配]
Allocate --> BuildLoop[构建循环]
subgraph BuildLoop["构建循环"]
direction TB
Build[Build文档<br/>写入MemSegment]
Evaluate[EvaluateCurrentMemUsed<br/>评估当前内存使用]
MemCheck{内存使用检查<br/>是否超过阈值?}
Build --> Evaluate
Evaluate --> MemCheck
MemCheck -->|未超阈值<br/>继续构建| Build
MemCheck -->|超过阈值<br/>触发转储| Dump[触发转储<br/>NeedDump返回true]
end
Dump --> DumpProcess[转储处理]
subgraph DumpProcess["转储处理"]
direction TB
CreateDump[CreateSegmentDumpItems<br/>创建转储项]
AsyncDump[异步转储到磁盘<br/>不阻塞写入]
ReleaseMem[释放内存<br/>释放MemSegment内存]
CreateNew[创建新MemSegment<br/>继续构建]
CreateDump --> AsyncDump
AsyncDump --> ReleaseMem
ReleaseMem --> CreateNew
end
CreateNew --> BuildLoop
WaitOrReject --> End[结束]
BuildLoop -.->|构建完成| End
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Estimate fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style QuotaCheck fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style WaitOrReject fill:#ffebee,stroke:#c62828,stroke-width:2px
style Allocate fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style BuildLoop fill:#f5f5f5,stroke:#757575,stroke-width:2px
style Build fill:#fff3e0,stroke:#f57c00,stroke-width:1px
style Evaluate fill:#fff3e0,stroke:#f57c00,stroke-width:1px
style MemCheck fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Dump fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style DumpProcess fill:#f5f5f5,stroke:#757575,stroke-width:2px
style CreateDump fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1px
style AsyncDump fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1px
style ReleaseMem fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
style CreateNew fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
内存管理策略：
EstimateMemUsed() 估算构建所需内存
EvaluateCurrentMemUsed() 评估当前实际内存使用
MemoryQuotaController 控制内存上限
性能优化：
通过阅读 table/normal_table/NormalDiskSegment.h，我们可以看到 NormalDiskSegment 的实现：
// table/normal_table/NormalDiskSegment.h
class NormalDiskSegment : public plain::PlainDiskSegment
{
public:
NormalDiskSegment(const std::shared_ptr<config::ITabletSchema>& schema,
const framework::SegmentMeta& segmentMeta,
const framework::BuildResource& buildResource);
// 估算内存使用
std::pair<Status, size_t> EstimateMemUsed(
const std::shared_ptr<config::ITabletSchema>& schema) override;
private:
// 打开 Indexer
std::pair<Status, std::vector<plain::DiskIndexerItem>>
OpenIndexer(const std::shared_ptr<config::IIndexConfig>& indexConfig) override;
};
DiskSegment 的加载流程：
flowchart TD
Start[DiskSegment.Open<br/>打开磁盘段] --> ReadInfo[读取SegmentInfo<br/>从磁盘加载元数据]
ReadInfo --> ModeSelect{OpenMode选择}
subgraph Normal["NORMAL 模式：立即加载"]
direction TB
N1[遍历所有IndexConfig] --> N2[打开所有Indexer<br/>并行加载]
N2 --> N3[InvertedIndexer<br/>倒排索引]
N2 --> N4[AttributeIndexer<br/>正排索引]
N2 --> N5[PrimaryKeyIndexer<br/>主键索引]
N2 --> N6[SummaryIndexer<br/>摘要索引]
N3 --> N7[所有Indexer在内存<br/>查询延迟低]
N4 --> N7
N5 --> N7
N6 --> N7
end
subgraph Lazy["LAZY 模式：按需加载"]
direction TB
L1[只读取SegmentInfo<br/>不加载Indexer] --> L2[等待查询请求]
L2 --> L3[GetIndexer调用<br/>type, indexName]
L3 --> L4{Indexer已加载?}
L4 -->|否| L5[按需打开Indexer<br/>OpenIndexer]
L4 -->|是| L8[返回缓存的Indexer]
L5 --> L6[加载索引数据到内存]
L6 --> L7[缓存Indexer]
L7 --> L8
end
subgraph Reopen["Reopen操作：Schema变更"]
direction TB
R1[Schema变更检测] --> R2[调用Reopen<br/>重新打开Segment]
R2 --> R3[使用新Schema<br/>重新加载Indexer]
R3 -.->|重新打开| Start
end
subgraph Memory["内存管理"]
direction TB
M1[MemoryQuotaController<br/>内存配额控制]
M2[估算内存使用<br/>EstimateMemUsed]
M3[检查内存配额]
M4[分配内存]
M1 --> M2
M2 --> M3
M3 --> M4
end
ModeSelect -->|NORMAL| Normal
ModeSelect -->|LAZY| Lazy
N2 -.->|内存分配| Memory
L5 -.->|内存分配| Memory
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style ReadInfo fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style ModeSelect fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Normal fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Lazy fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Reopen fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Memory fill:#f5f5f5,stroke:#757575,stroke-width:2px
style N7 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style L8 fill:#fff9c4,stroke:#f57f17,stroke-width:2px
DiskSegment 支持按需加载，通过 GetIndexer() 方法实现：
// framework/Segment.h
class Segment {
public:
// 获取 Indexer（LAZY 模式下按需加载）
virtual std::pair<Status, std::shared_ptr<indexlibv2::index::IIndexer>>
GetIndexer(const std::string& type, const std::string& indexName) {
return std::make_pair(Status::NotFound(), nullptr);
}
};
按需加载的优势：
flowchart TD
A[DiskSegment<br/>LAZY模式] --> B[Open调用<br/>只读取SegmentInfo]
B --> C[不加载任何Indexer<br/>快速启动]
subgraph Query["查询时按需加载"]
Q1[查询请求到达]
Q2[GetIndexer调用<br/>指定type和indexName]
Q3{Indexer缓存中?}
Q4[从缓存返回]
Q5[按需打开Indexer<br/>OpenIndexer]
Q6[读取索引文件<br/>从磁盘加载]
Q7[解析索引数据]
Q8[缓存Indexer<br/>避免重复加载]
Q9[返回Indexer]
C --> Q1
Q1 --> Q2
Q2 --> Q3
Q3 -->|是| Q4
Q3 -->|否| Q5
Q5 --> Q6
Q6 --> Q7
Q7 --> Q8
Q8 --> Q9
Q4 --> Q9
end
subgraph Advantages["LAZY模式优势"]
A1[减少内存占用<br/>只加载查询需要的索引]
A2[提高启动速度<br/>不需要等待所有索引加载]
A3[灵活查询<br/>支持部分索引查询场景]
A4[节省资源<br/>适合离线场景]
A5[动态加载<br/>根据查询模式优化]
C -.-> A1
C -.-> A2
Q9 -.-> A3
Q9 -.-> A4
Q9 -.-> A5
end
subgraph Comparison["对比NORMAL模式"]
C1[NORMAL模式<br/>启动时加载所有索引]
C2[内存占用大<br/>但查询延迟低]
C3[适合在线查询场景]
A1 -.-> C1
A2 -.-> C2
A3 -.-> C3
end
style Query fill:#e3f2fd
style Advantages fill:#fff3e0
style Comparison fill:#f5f5f5
通过阅读 table/normal_table/NormalTabletWriter.h，我们可以看到 TabletWriter 的实现：
// table/normal_table/NormalTabletWriter.h
class NormalTabletWriter : public table::CommonTabletWriter
{
public:
// 打开：初始化 TabletData 和构建资源
Status Open(const std::shared_ptr<framework::TabletData>& tabletData,
const framework::BuildResource& buildResource,
const framework::OpenOptions& openOptions) override;
// 构建：接收文档批次并写入
Status Build(const std::shared_ptr<document::IDocumentBatch>& batch) override;
// 创建 SegmentDumper：准备转储
std::unique_ptr<framework::SegmentDumper> CreateSegmentDumper() override;
private:
std::shared_ptr<NormalMemSegment> _normalBuildingSegment; // 当前构建中的 Segment
docid_t _buildingSegmentBaseDocId; // 构建 Segment 的基础 DocId
};
TabletWriter 与 Segment 的交互流程：
flowchart TD
subgraph Open["Open阶段"]
O1[TabletWriter.Open<br/>初始化]
O2[保存TabletData引用]
O3[保存BuildResource<br/>内存配额/IO配额]
O4{当前有MemSegment?}
O5[获取现有MemSegment]
O6[创建新MemSegment<br/>CreateMemSegment]
O7[初始化MemSegment<br/>设置状态ST_BUILDING]
O1 --> O2
O2 --> O3
O3 --> O4
O4 -->|是| O5
O4 -->|否| O6
O6 --> O7
O5 --> B1
O7 --> B1
end
subgraph Build["Build阶段"]
B1[接收文档批次<br/>IDocumentBatch]
B2[文档验证<br/>格式/Schema验证]
B3[分配DocId<br/>DispatchDocIds]
B4[写入MemSegment<br/>Build方法]
B5[写入倒排索引<br/>InvertedIndexer]
B6[写入正排索引<br/>AttributeIndexer]
B7[更新SegmentInfo<br/>docCount/Locator]
B8[评估内存使用<br/>EvaluateCurrentMemUsed]
B9{NeedDump检查<br/>转储条件}
B1 --> B2
B2 --> B3
B3 --> B4
B4 --> B5
B4 --> B6
B5 --> B7
B6 --> B7
B7 --> B8
B8 --> B9
B9 -->|否| B1
end
subgraph Dump["Dump阶段"]
D1[创建SegmentDumper<br/>CreateSegmentDumper]
D2[设置状态<br/>ST_BUILDING → ST_DUMPING]
D3[创建转储项<br/>CreateSegmentDumpItems]
D4[索引文件转储]
D5[元数据文件转储]
D6[异步转储到磁盘<br/>Dump方法]
D7[创建DiskSegment<br/>从转储文件]
D8[初始化DiskSegment<br/>Open方法]
B9 -->|是| D1
D1 --> D2
D2 --> D3
D3 --> D4
D3 --> D5
D4 --> D6
D5 --> D6
D6 --> D7
D7 --> D8
end
subgraph Update["更新阶段"]
U1[Reopen TabletData<br/>更新版本]
U2[添加DiskSegment<br/>AddSegment]
U3[移除MemSegment<br/>RemoveSegment]
U4[更新Version<br/>新增Segment]
U5[释放MemSegment内存]
D8 --> U1
U1 --> U2
U2 --> U3
U3 --> U4
U4 --> U5
end
subgraph Conditions["转储条件"]
C1[内存使用 > 阈值<br/>默认80%]
C2[文档数 > 阈值<br/>默认100万]
C3[时间间隔 > 阈值<br/>默认5分钟]
B9 -.-> C1
B9 -.-> C2
B9 -.-> C3
end
style Open fill:#e3f2fd
style Build fill:#fff3e0
style Dump fill:#f3e5f5
style Update fill:#e8f5e9
style Conditions fill:#f5f5f5
_normalBuildingSegmentTabletWriter 在构建时需要为文档分配 DocId。关键代码：
// table/normal_table/NormalTabletWriter.h
class NormalTabletWriter {
private:
// 分发 DocId：为文档分配 DocId
void DispatchDocIds(document::IDocumentBatch* batch);
docid_t _buildingSegmentBaseDocId; // 当前构建 Segment 的基础 DocId
};
DocId 分配机制：
flowchart TD
Start[文档写入] --> GetBase[获取BaseDocId<br/>_buildingSegmentBaseDocId]
GetBase --> AllocLocal[分配LocalDocId<br/>从0开始递增]
AllocLocal --> Increment[LocalDocId递增<br/>localDocId++]
Increment --> CalcGlobal[计算GlobalDocId<br/>GlobalDocId = BaseDocId + LocalDocId]
CalcGlobal --> Assign[为文档分配DocId<br/>设置到Document对象]
subgraph Concepts["概念说明"]
direction TB
BaseConcept[BaseDocId<br/>基础文档ID] --> BaseDesc[前面所有Segment的<br/>docCount之和]
LocalConcept[LocalDocId<br/>局部文档ID] --> LocalDesc[在Segment内<br/>从0开始递增]
GlobalConcept[GlobalDocId<br/>全局文档ID] --> GlobalDesc[全局唯一<br/>BaseDocId + LocalDocId]
end
subgraph Example["示例计算"]
direction TB
E1[Segment1: docCount=100<br/>BaseDocId=0]
E2[Segment2: docCount=200<br/>BaseDocId=100]
E3[Segment3: 第1个文档<br/>BaseDocId=300, LocalDocId=0<br/>GlobalDocId=300]
E4[Segment3: 第2个文档<br/>BaseDocId=300, LocalDocId=1<br/>GlobalDocId=301]
E1 --> E2
E2 --> E3
E3 --> E4
end
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetBase fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style AllocLocal fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Increment fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1px
style CalcGlobal fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style Assign fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Concepts fill:#f5f5f5,stroke:#757575,stroke-width:1px
style BaseConcept fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
style LocalConcept fill:#f3e5f5,stroke:#7b1fa2,stroke-width:1px
style GlobalConcept fill:#fce4ec,stroke:#c2185b,stroke-width:1px
style Example fill:#fff9c4,stroke:#f57f17,stroke-width:1px
baseDocId + localDocIdSegmentDumper 负责将 MemSegment 转储到磁盘，定义在 framework/SegmentDumper.h 中：
// framework/SegmentDumper.h
class SegmentDumper : public SegmentDumpable
{
public:
SegmentDumper(const std::string& tabletName,
const std::shared_ptr<MemSegment>& segment,
int64_t dumpExpandMemSize,
std::shared_ptr<kmonitor::MetricsReporter> metricsReporter)
: _tabletName(tabletName)
, _dumpingSegment(segment)
, _dumpExpandMemSize(dumpExpandMemSize)
{
// 设置 Segment 状态为 DUMPING
_dumpingSegment->SetSegmentStatus(Segment::SegmentStatus::ST_DUMPING);
}
// 执行转储
virtual Status Dump() = 0;
// 获取转储的 SegmentMeta
virtual std::pair<Status, SegmentMeta> GetDumpedSegmentMeta() = 0;
};
转储流程：
转储是将 MemSegment 持久化为 DiskSegment 的关键步骤。让我们通过序列图来理解完整的转储流程：
sequenceDiagram
participant Writer as TabletWriter
participant MemSeg as MemSegment
participant Dumper as SegmentDumper
participant DiskSeg as DiskSegment
participant TabletData as TabletData
participant FileSys as FileSystem
Writer->>MemSeg: NeedDump()?
MemSeg-->>Writer: true
Writer->>Writer: CreateSegmentDumper()
Writer->>Dumper: SegmentDumper(MemSeg)
Dumper->>MemSeg: SetStatus(ST_DUMPING)
Dumper-->>Writer: Dumper
Writer->>Dumper: Dump()
Dumper->>MemSeg: CreateDumpItems()
MemSeg-->>Dumper: DumpItems
loop 遍历每个DumpItem
Dumper->>FileSys: WriteFile(dumpItem)
FileSys-->>Dumper: Success
end
Dumper->>DiskSeg: CreateDiskSegment(SegmentMeta)
DiskSeg->>DiskSeg: Open(OpenMode)
DiskSeg-->>Dumper: Success
Dumper-->>Writer: Success
Writer->>TabletData: AddSegment(DiskSeg)
Writer->>TabletData: RemoveSegment(MemSeg)
TabletData-->>Writer: Success
转储流程详解：
CreateSegmentDumper() 创建转储器
ST_DUMPING
ST_BUILDING 转换为 ST_DUMPINGDump() 将内存数据写入磁盘
Open() 初始化 DiskSegmentST_BUILT（实际已被 DiskSegment 替代）
转储是异步的，不会阻塞新的写入。关键设计：
// framework/SegmentDumper.h
class DumpControl {
public:
// 控制转储任务的执行
std::tuple<uint32_t, uint32_t> StartTask();
std::tuple<uint32_t, uint32_t> Iterate(Status& taskStatus);
uint32_t ExitTask(const bool isCoordinator);
private:
std::atomic<uint32_t> _finishCount = 0; // 完成的任务数
uint32_t _totalCount; // 总任务数
std::mutex _dumpMutex; // 转储互斥锁
std::condition_variable _dumpCv; // 转储条件变量
};
异步转储的优势：
异步转储是 IndexLib 高性能写入的关键设计。让我们通过流程图来理解异步转储的机制：
graph TD
A[MemSegment达到转储条件] --> B[创建转储任务]
B --> C[提交到转储队列]
C --> D[创建新MemSegment]
D --> E[继续接收写入]
C --> F[转储线程池]
F --> G[执行转储任务]
G --> H[写入磁盘]
H --> I[创建DiskSegment]
I --> J[更新TabletData]
K[转储控制] --> F
K --> L{检查并发度}
L -->|未超限| G
L -->|超限| M[等待]
M --> L
style A fill:#e3f2fd
style D fill:#fff3e0
style G fill:#f3e5f5
style J fill:#e8f5e9
异步转储的优势：
DumpControl 控制转储任务的并发度
性能优化：
查询时需要遍历多个 Segment，可以并行查询以提高性能：
flowchart TD
A[查询请求<br/>Query对象] --> B[TabletData.CreateSlice<br/>ST_BUILT]
B --> C[获取Segment列表<br/>已构建的Segment]
subgraph Segments["Segment列表"]
S1[Segment1<br/>docCount=1000<br/>BaseDocId=0]
S2[Segment2<br/>docCount=2000<br/>BaseDocId=1000]
S3[Segment3<br/>docCount=1500<br/>BaseDocId=3000]
C --> S1
C --> S2
C --> S3
end
subgraph Parallel["并行查询执行"]
P1[Segment1查询<br/>IndexReader.Search]
P2[Segment2查询<br/>IndexReader.Search]
P3[Segment3查询<br/>IndexReader.Search]
P4[线程池执行<br/>并发查询]
P5[收集查询结果<br/>Result1, Result2, Result3]
S1 --> P1
S2 --> P2
S3 --> P3
P1 --> P4
P2 --> P4
P3 --> P4
P4 --> P5
end
subgraph Merge["结果合并"]
M1[DocId去重<br/>避免重复文档]
M2[按相关性分数排序<br/>或按指定字段排序]
M3[分页处理<br/>offset/limit]
M4[聚合统计<br/>总数/平均值等]
P5 --> M1
M1 --> M2
M2 --> M3
M3 --> M4
end
subgraph Performance["性能优化"]
PF1[并行度控制<br/>线程池大小]
PF2[结果流式合并<br/>边查询边合并]
PF3[索引剪枝<br/>跳过不相关Segment]
P4 -.-> PF1
M1 -.-> PF2
C -.-> PF3
end
M4 --> R[返回结果<br/>QueryResult]
style Segments fill:#e3f2fd
style Parallel fill:#fff3e0
style Merge fill:#f3e5f5
style Performance fill:#f5f5f5
style R fill:#e8f5e9
查询流程：
TabletData->CreateSlice(ST_BUILT) 获取所有已构建的 Segment查询时需要将全局 DocId 转换为局部 DocId：
// 伪代码：全局 DocId 转局部 DocId
for (auto& seg : segments) {
docid64_t baseDocId = GetBaseDocId(seg);
if (globalDocId >= baseDocId && globalDocId < baseDocId + seg->GetDocCount()) {
docid_t localDocId = globalDocId - baseDocId;
// 在 Segment 内查询
return seg->GetIndexer()->Get(localDocId);
}
}
DocId 转换流程：
flowchart TB
Start([查询请求<br/>Query Request<br/>GlobalDocId]) --> LocateLayer[定位Segment阶段<br/>Locate Segment Phase]
subgraph LocateGroup["1. 定位Segment Locate Segment"]
direction TB
L1[遍历Segment列表<br/>Traverse Segment List]
L2[计算BaseDocId<br/>Calculate BaseDocId<br/>累加前面Segment的docCount]
L3{GlobalDocId在范围内?<br/>In Range?<br/>BaseDocId <= GlobalDocId<br/>< BaseDocId + docCount}
L4[找到对应Segment<br/>Found Target Segment]
L1 --> L2
L2 --> L3
L3 -->|否| L1
L3 -->|是| L4
end
LocateLayer --> ConvertLayer[DocId转换阶段<br/>DocId Conversion Phase]
subgraph ConvertGroup["2. DocId转换 DocId Conversion"]
direction TB
C1[获取BaseDocId<br/>Get BaseDocId]
C2[计算LocalDocId<br/>Calculate LocalDocId<br/>LocalDocId = GlobalDocId - BaseDocId]
C3[验证有效性<br/>Validate<br/>0 <= LocalDocId < docCount]
C1 --> C2
C2 --> C3
end
ConvertLayer --> QueryLayer[Segment内查询阶段<br/>Segment Query Phase]
subgraph QueryGroup["3. Segment内查询 Segment Query"]
direction TB
Q1[使用LocalDocId查询<br/>Query with LocalDocId<br/>IndexReader.Get]
Q2[InvertedIndexer<br/>倒排索引<br/>Inverted Index]
Q3[AttributeIndexer<br/>正排索引<br/>Attribute Index]
Q4[返回文档数据<br/>Return Document Data]
Q1 --> Q2
Q1 --> Q3
Q2 --> Q4
Q3 --> Q4
end
QueryLayer --> ExampleLayer[转换示例<br/>Conversion Example]
subgraph ExampleGroup["转换示例 Conversion Example"]
direction TB
E1[GlobalDocId = 1500]
E2[Segment1: BaseDocId=0, docCount=1000<br/>范围: 0-999 不在范围内]
E3[Segment2: BaseDocId=1000, docCount=2000<br/>范围: 1000-2999 在范围内]
E4[LocalDocId = 1500 - 1000 = 500]
E5[在Segment2内查询<br/>Query in Segment2<br/>LocalDocId=500]
E1 --> E2
E2 --> E3
E3 --> E4
E4 --> E5
end
ExampleLayer --> End([返回查询结果<br/>Return Query Result])
LocateLayer -.->|包含| LocateGroup
ConvertLayer -.->|包含| ConvertGroup
QueryLayer -.->|包含| QueryGroup
ExampleLayer -.->|包含| ExampleGroup
L4 --> C1
C3 --> Q1
Q4 --> E1
style Start fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style End fill:#c8e6c9,stroke:#388e3c,stroke-width:3px
style LocateLayer fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style ConvertLayer fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style QueryLayer fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style ExampleLayer fill:#fff9c4,stroke:#f57f17,stroke-width:3px
style LocateGroup fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
style L1 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L2 fill:#90caf9,stroke:#1976d2,stroke-width:2px
style L3 fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style L4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
style ConvertGroup fill:#fff3e0,stroke:#f57c00,stroke-width:3px
style C1 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C2 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style C3 fill:#ffcc80,stroke:#f57c00,stroke-width:2px
style QueryGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:3px
style Q1 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Q2 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Q3 fill:#ce93d8,stroke:#7b1fa2,stroke-width:2px
style Q4 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style ExampleGroup fill:#fff9c4,stroke:#f57f17,stroke-width:3px
style E1 fill:#ffe082,stroke:#f57f17,stroke-width:2px
style E2 fill:#ffe082,stroke:#f57f17,stroke-width:2px
style E3 fill:#ffe082,stroke:#f57f17,stroke-width:2px
style E4 fill:#ffe082,stroke:#f57f17,stroke-width:2px
style E5 fill:#ffe082,stroke:#f57f17,stroke-width:2px
localDocId = globalDocId - baseDocIdSegment 的创建通过 ITabletFactory 实现：
// framework/ITabletFactory.h
class ITabletFactory {
public:
// 创建 MemSegment
virtual std::unique_ptr<MemSegment> CreateMemSegment(
const SegmentMeta& segmentMeta) = 0;
// 创建 DiskSegment
virtual std::unique_ptr<DiskSegment> CreateDiskSegment(
const SegmentMeta& segmentMeta,
const framework::BuildResource& buildResource) = 0;
};
Segment 创建流程：
flowchart TD
Start[开始创建Segment] --> CreateMeta[创建SegmentMeta<br/>设置元数据]
CreateMeta --> SetMeta[设置SegmentMeta属性<br/>SegmentId/Directory/Schema<br/>SegmentStatus等]
SetMeta --> CallFactory[调用ITabletFactory<br/>根据类型创建Segment]
CallFactory --> TypeSelect{Segment类型选择}
TypeSelect -->|MemSegment| CreateMem[CreateMemSegment<br/>创建内存段<br/>传入SegmentMeta]
TypeSelect -->|DiskSegment| CreateDisk[CreateDiskSegment<br/>创建磁盘段<br/>传入SegmentMeta和BuildResource]
CreateMem --> Init[调用Open初始化<br/>加载Schema和配置<br/>初始化Indexer]
CreateDisk --> Init
Init --> AddToTablet[添加到TabletData<br/>TabletData.AddSegment]
AddToTablet --> UpdateList[Segment列表更新<br/>_segments列表添加新Segment<br/>保持SegmentId有序]
UpdateList --> End[Segment创建完成]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style CreateMeta fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style SetMeta fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style CallFactory fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style TypeSelect fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style CreateMem fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style CreateDisk fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Init fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style AddToTablet fill:#fce4ec,stroke:#c2185b,stroke-width:2px
style UpdateList fill:#fce4ec,stroke:#c2185b,stroke-width:1px
style End fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
ITabletFactory 创建 SegmentOpen() 初始化Segment 的销毁通过智能指针自动管理：
// Segment 使用 shared_ptr 管理
using SegmentPtr = std::shared_ptr<Segment>;
// 当 Segment 不再被引用时，自动析构
// 析构时会：
// 1. 释放内存资源（MemSegment）
// 2. 关闭文件句柄（DiskSegment）
// 3. 清理 Indexer
Segment 销毁时机：
graph LR
A[Segment销毁触发] --> B{触发条件}
B -->|合并后| C[旧Segment不再被引用]
B -->|版本清理| D[清理旧版本]
B -->|资源回收| E[ReclaimSegmentResource]
C --> F[自动析构]
D --> F
E --> F
F --> G[释放内存资源]
F --> H[关闭文件句柄]
F --> I[清理Indexer]
style A fill:#e3f2fd
style B fill:#fff3e0
style F fill:#e8f5e9
style G fill:#f3e5f5
ReclaimSegmentResource() 主动回收资源在实时写入场景中，Tablet 和 Segment 的组织方式：
graph LR
A[文档持续写入] --> B[MemSegment]
B --> C{达到阈值?}
C -->|是| D[转储为DiskSegment]
C -->|否| A
D --> E[创建新MemSegment]
E --> A
D --> F[定期Commit]
F --> G[更新Version]
H[Tablet] --> B
H --> I[DiskSegment1]
H --> J[DiskSegment2]
H --> K[DiskSegment3]
style A fill:#e3f2fd
style B fill:#fff3e0
style D fill:#e8f5e9
style F fill:#f3e5f5
style H fill:#fce4ec
在查询场景中，需要遍历多个 Segment：
flowchart TD
Start[查询请求] --> GetTabletData[TabletData.GetSegment<br/>获取Segment列表]
GetTabletData --> GetList[获取所有已构建的Segment<br/>ST_BUILT状态的Segment]
GetList --> ParallelQueryStart[并行查询各Segment]
subgraph ParallelQueryGroup["并行查询阶段"]
direction LR
Q1[Segment1查询<br/>使用LocalDocId]
Q2[Segment2查询<br/>使用LocalDocId]
Q3[Segment3查询<br/>使用LocalDocId]
Q4[更多Segment...]
end
ParallelQueryStart --> Q1
ParallelQueryStart --> Q2
ParallelQueryStart --> Q3
ParallelQueryStart --> Q4
Q1 --> MergeStart[结果合并]
Q2 --> MergeStart
Q3 --> MergeStart
Q4 --> MergeStart
subgraph MergeGroup["结果合并阶段"]
direction TB
M1[收集各Segment结果<br/>Result1, Result2, Result3...]
M2[DocId去重<br/>避免重复文档]
M3[按相关性分数排序<br/>或按指定字段排序]
M4[分页处理<br/>offset/limit]
M1 --> M2
M2 --> M3
M3 --> M4
end
MergeStart --> M1
M4 --> DocIdConvertStart[DocId转换]
subgraph DocIdConvertGroup["DocId转换阶段"]
direction TB
D1[获取每个Segment的BaseDocId<br/>前面所有Segment的docCount之和]
D2[转换LocalDocId为GlobalDocId<br/>GlobalDocId = BaseDocId + LocalDocId]
D3[验证GlobalDocId有效性]
D1 --> D2
D2 --> D3
end
DocIdConvertStart --> D1
D3 --> Return[返回查询结果<br/>包含GlobalDocId和文档数据]
style Start fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style GetTabletData fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style GetList fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
style ParallelQueryStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style ParallelQueryGroup fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Q1 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Q2 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Q3 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Q4 fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style MergeStart fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style MergeGroup fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style M1 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style M2 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style M3 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style M4 fill:#e1bee7,stroke:#7b1fa2,stroke-width:1px
style DocIdConvertStart fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DocIdConvertGroup fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style D1 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style D2 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style D3 fill:#ffe0b2,stroke:#f57c00,stroke-width:1px
style Return fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
Segment 大小的影响：
最佳实践：
优化策略：
优化策略：
Tablet 和 Segment 的组织方式是 IndexLib 索引机制的核心。通过本文的深入解析，我们了解到：
核心概念：
设计亮点：
性能优化：
理解 Tablet 和 Segment 的组织方式，是掌握 IndexLib 索引构建和查询机制的基础。在下一篇文章中，我们将深入介绍索引构建的完整流程，包括 Build、Flush、Seal、Commit 等各个阶段的实现细节和性能优化策略。
]]>