feat:add page source overlap when use graph search#606
Open
FeiZhou02 wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR enhances wiki search’s graph expansion by adding “source overlap” edges derived from each page’s frontmatter sources field, using a Jaccard similarity threshold to avoid weak/accidental connections.
Changes:
- Add
sourcesextraction from YAML frontmatter (inline array and multiline list) and store it onGraphPage. - Add source-overlap graph neighbors using Jaccard similarity (
SOURCE_JACCARD_THRESHOLD = 0.3) and merge them with existing wikilink edges. - Add tests covering YAML parsing and graph blending behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+579
to
+584
| let source_set: BTreeSet<String> = page | ||
| .sources | ||
| .iter() | ||
| .filter(|s| !s.is_empty()) | ||
| .map(|s| normalize_graph_alias(s)) | ||
| .collect(); |
Comment on lines
+599
to
+603
| for page_set in source_to_pages.values() { | ||
| if page_set.len() < 2 { | ||
| continue; | ||
| } | ||
| for a in page_set { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
为 Wiki 搜索引入基于 source overlap(来源重叠) 的图扩展能力:从 frontmatter 中提取
sources字段,用 Jaccard 相似度阈值判定页面间的关联强弱,将共享参考文献的页面作为图邻居返回给搜索结果。Motivation
之前搜索的图扩展只支持 wikilink(
[[...]])连接。但实际上很多页面即使没有互相引用,也会因为使用了相同的来源而高度相关。同时,简单的"共享任意一个来源即连边"会导致弱关联泛滥:两个各引用 10 篇文献、只共享其中 1 篇的页面,不应该和两个只引用同一篇文献的页面对等看待。
Changes
1. 数据层:
extract_sources()解析 frontmatterGraphPage新增sources: Vec<String>字段extract_sources()函数,支持两种 YAML 格式:sources: ["a.pdf", "b.pdf"]frontmatter_block()和fm_match()辅助函数2. 图边判定:Jaccard 相似度阈值
SOURCE_JACCARD_THRESHOLD = 0.3BTreeSet,自动去重+规范化)|A ∩ B| / |A ∪ B|,仅当 > 0.3 时才建立图邻接边seen_pairs确保每对页面只计算一次,避免重复BTreeSet自动去重3. 测试
extract_sources_parses_inline_and_multiline_yaml:覆盖行内、多行、空列表、无 frontmatter 等场景graph_search_blends_source_overlap_neighbors:验证通过 Jaccard 阈值发现的图邻居source_overlap_and_wikilinks_combine_in_graph:验证 wikilink 和 source overlap 同时存在时去重Example
A ↔ B 的 Jaccard = 1/2 = 0.5 > 0.3 → 建立图边 ✅
A ↔ B 的 Jaccard = 1/19 ≈ 0.05 ≤ 0.3 → 不建边 ❌(弱关联被过滤)
Testing
全部 30 个 search 模块测试通过: