Skip to content

feat:add page source overlap when use graph search#606

Open
FeiZhou02 wants to merge 1 commit into
nashsu:mainfrom
FeiZhou02:feat/jaccard-source-overlap
Open

feat:add page source overlap when use graph search#606
FeiZhou02 wants to merge 1 commit into
nashsu:mainfrom
FeiZhou02:feat/jaccard-source-overlap

Conversation

@FeiZhou02

Copy link
Copy Markdown

Summary

为 Wiki 搜索引入基于 source overlap(来源重叠) 的图扩展能力:从 frontmatter 中提取 sources 字段,用 Jaccard 相似度阈值判定页面间的关联强弱,将共享参考文献的页面作为图邻居返回给搜索结果。

Motivation

之前搜索的图扩展只支持 wikilink([[...]])连接。但实际上很多页面即使没有互相引用,也会因为使用了相同的来源而高度相关。

同时,简单的"共享任意一个来源即连边"会导致弱关联泛滥:两个各引用 10 篇文献、只共享其中 1 篇的页面,不应该和两个只引用同一篇文献的页面对等看待。

Changes

1. 数据层:extract_sources() 解析 frontmatter

  • GraphPage 新增 sources: Vec<String> 字段
  • 新增 extract_sources() 函数,支持两种 YAML 格式:
    • 行内数组:sources: ["a.pdf", "b.pdf"]
    • 多行列表:
      sources:
        - "a.pdf"
        - "b.pdf"
  • 新增 frontmatter_block()fm_match() 辅助函数

2. 图边判定:Jaccard 相似度阈值

  • 新增常量 SOURCE_JACCARD_THRESHOLD = 0.3
  • 构建每个页面的规范化 source 集合(BTreeSet,自动去重+规范化)
  • 两两计算 |A ∩ B| / |A ∪ B|,仅当 > 0.3 时才建立图邻接边
  • 通过 seen_pairs 确保每对页面只计算一次,避免重复
  • source overlap 边和 wikilink 边共用同一邻接表,BTreeSet 自动去重

3. 测试

  • extract_sources_parses_inline_and_multiline_yaml:覆盖行内、多行、空列表、无 frontmatter 等场景
  • graph_search_blends_source_overlap_neighbors:验证通过 Jaccard 阈值发现的图邻居
  • source_overlap_and_wikilinks_combine_in_graph:验证 wikilink 和 source overlap 同时存在时去重

Example

# page-a.md
sources: ["doc-X.pdf", "doc-Y.pdf"]

# page-b.md  
sources: ["doc-X.pdf"]

A ↔ B 的 Jaccard = 1/2 = 0.5 > 0.3 → 建立图边 ✅

# page-a.md
sources: ["a.pdf", "b.pdf", "c.pdf", "d.pdf", "e.pdf",
          "f.pdf", "g.pdf", "h.pdf", "i.pdf", "j.pdf"]

# page-b.md
sources: ["j.pdf", "k.pdf", "l.pdf", "m.pdf", "n.pdf",
          "o.pdf", "p.pdf", "q.pdf", "r.pdf", "s.pdf"]

A ↔ B 的 Jaccard = 1/19 ≈ 0.05 ≤ 0.3 → 不建边 ❌(弱关联被过滤)

Testing

全部 30 个 search 模块测试通过:

test result: ok. 30 passed; 0 failed

Copilot AI review requested due to automatic review settings July 22, 2026 15:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enhances wiki search’s graph expansion by adding “source overlap” edges derived from each page’s frontmatter sources field, using a Jaccard similarity threshold to avoid weak/accidental connections.

Changes:

  • Add sources extraction from YAML frontmatter (inline array and multiline list) and store it on GraphPage.
  • Add source-overlap graph neighbors using Jaccard similarity (SOURCE_JACCARD_THRESHOLD = 0.3) and merge them with existing wikilink edges.
  • Add tests covering YAML parsing and graph blending behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +579 to +584
let source_set: BTreeSet<String> = page
.sources
.iter()
.filter(|s| !s.is_empty())
.map(|s| normalize_graph_alias(s))
.collect();
Comment on lines +599 to +603
for page_set in source_to_pages.values() {
if page_set.len() < 2 {
continue;
}
for a in page_set {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants