Add normalization to minhash

[ayushdg]

Description

Closes #2246.

Adds the option to normalize text (lowercase + whitespace only) to catch more duplicates that differ because of these changes. Will share more stats on dedup rates

Usage

# Add snippet demonstrating usage

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR adds an optional normalize_text flag to MinHashStage that lowercases text and collapses all whitespace (tabs, newlines, multiple spaces) into a single space before computing minhash signatures, catching duplicates that differ only in casing or whitespace formatting.

• MinHashStage gains a normalize_text: bool = False parameter; when set, _normalize_text applies str.lower().str.normalize_spaces() inside a timed metric block before hashing.
• A new GPU test covers multilingual case/whitespace variants (English, French, Spanish, Greek, Russian, Arabic, Japanese) and asserts that each pair produces identical signatures after normalization.
• FuzzyDeduplicationWorkflow — the main high-level entry point — does not accept or forward normalize_text when constructing MinHashStage, so the feature is unreachable through the workflow API without using MinHashStage directly.

Confidence Score: 3/5

The core normalization logic in MinHashStage is correct and well-tested, but the feature is currently inaccessible to users of FuzzyDeduplicationWorkflow, the main API for running fuzzy dedup end-to-end.

FuzzyDeduplicationWorkflow._create_minhash_pipeline constructs MinHashStage without normalize_text, leaving the new option dead code for the majority of users who rely on the workflow API. This should be resolved before merging to avoid shipping a feature that appears available but is silently ignored in the typical usage path.

nemo_curator/stages/deduplication/fuzzy/workflow.py needs normalize_text added to FuzzyDeduplicationWorkflow.__init__ and forwarded in _create_minhash_pipeline.

Important Files Changed

Filename	Overview
nemo_curator/stages/deduplication/fuzzy/minhash.py	Adds normalize_text parameter to MinHashStage that lowercases and collapses whitespace before computing minhashes; implementation is correct but docstrings misdescribe the normalization scope.
tests/stages/deduplication/fuzzy/test_minhash_stage.py	Adds multilingual fixture and an end-to-end GPU test verifying case+whitespace variants hash identically after normalization; test logic and ID-ordering assumption are sound.
nemo_curator/stages/deduplication/fuzzy/workflow.py	FuzzyDeduplicationWorkflow constructs MinHashStage without normalize_text, so the new feature is unreachable through the high-level workflow API.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[MinHashStage.process] --> B{normalize_text?}
    B -- Yes --> C["_normalize_text(text_series)\nstr.lower() + str.normalize_spaces()"]
    C --> D[compute_minhashes]
    B -- No --> D
    D --> E[write_parquet output]

    F[FuzzyDeduplicationWorkflow] --> G["_create_minhash_pipeline()"]
    G --> H["MinHashStage(... normalize_text NOT passed ...)"]
    H -.->|feature unreachable| I["normalize_text=False (hardcoded default)"]

Loading

_{Reviews (1): Last reviewed commit: "Merge branch 'main' into minhash-normali..." | Re-trigger Greptile}

[greptile-apps]

The docstring says "trim whitespace" but str.normalize_spaces() also collapses all interior whitespace sequences (tabs, newlines, multiple spaces) into a single space — not just leading/trailing trimming. The misleading description may cause users to set normalize_text=True expecting only edge-trimming while silently changing their shingled text more aggressively than expected.

Suggested change

	normalize_text : bool, default=False
	Whether to normalize text before computing minhashes
	Current normalization is limited to lowercase and trim whitespace
	normalize_text : bool, default=False
	Whether to normalize text before computing minhashes
	Current normalization lowercases text and collapses all whitespace
	sequences (tabs, newlines, leading/trailing spaces) to a single space

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Same imprecision in the _normalize_text docstring: "trim whitespace" only describes stripping, while normalize_spaces() also collapses interior whitespace.

Suggested change

	def _normalize_text(self, text_series: cudf.Series) -> cudf.Series:
	"""Normalize text to lowercase and trim whitespace."""
	return text_series.str.lower().str.normalize_spaces()
	def _normalize_text(self, text_series: cudf.Series) -> cudf.Series:
	"""Normalize text to lowercase and collapse all whitespace sequences to a single space."""
	return text_series.str.lower().str.normalize_spaces()

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!