Skip to content

fix(prompt): allow ZWJ emoji in context files#18616

Closed
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/issue-18581-zwj-emoji-allowed
Closed

fix(prompt): allow ZWJ emoji in context files#18616
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/issue-18581-zwj-emoji-allowed

Conversation

@liuhao1024

@liuhao1024 liuhao1024 commented May 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What does this PR do?

  • Remove U+200D (Zero Width Joiner) from _CONTEXT_INVISIBLE_CHARS so that standard Unicode emoji sequences no longer silently block entire context files
  • Add regression tests: ZWJ emoji passes through, other invisible chars still blocked

Related Issue

N/A

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • See commit messages for detailed changes

How to Test

  1. Run pytest tests/ -q — all tests should pass
  2. Verify the specific scenario described above is resolved

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 26.4.1

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture and workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A

@liuhao1024
fix(prompt): allow ZWJ emoji in context files
1fdb5b5 
U+200D (Zero Width Joiner) is the standard Unicode mechanism for
gendered and compound emoji sequences.  The invisible-char scanner
blocked entire context files (SOUL.md, AGENTS.md, etc.) when any
ZWJ was present, silently censoring legitimate emoji.

Remove U+200D from _CONTEXT_INVISIBLE_CHARS while keeping all other
truly suspicious invisible characters (zero-width space, directional
overrides, BOM).  Add regression tests for both the allowed ZWJ path
and the continued blocking of the remaining invisible characters.

Closes NousResearch#18581
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 2, 2026
@Kailigithub Kailigithub mentioned this pull request May 2, 2026
Closed
Cyrene963 pushed a commit to Cyrene963/hermes-agent that referenced this pull request May 3, 2026
Nitrogen Cyrene963
Apply 14 community PRs + fix session_search ordering
a6e0ad9 
Community PRs applied:
- NousResearch#18596: Enable secret redaction by default (SECURITY)
- NousResearch#18650: Sanitize malformed tool messages + auto-recover on API 400
- NousResearch#18607: Emergency compression before max_iterations exhaustion
- NousResearch#18603: Compression fallback to main model on 413 rate limit
- NousResearch#18638: Pass threshold_percent on model switch
- NousResearch#18663: Strip extra_content from tool_calls for strict APIs
- NousResearch#18618: Forward explicit_api_key to OpenRouter
- NousResearch#18632: Show cache tokens in /insights breakdown
- NousResearch#18614: Add idempotency guard for patch duplicate loops
- NousResearch#18600: Raise ValueError when HERMES_HOME unset in profile mode
- NousResearch#18616: Allow ZWJ emoji in context files
- NousResearch#18582: Reload .env on /restart
- NousResearch#18547: Stabilize system prompt prefix for KV cache reuse
- NousResearch#18692: Strip FTS5 operators from session search truncation terms

Fix: Add order_by_last_active=True to list_sessions_rich call
(pre-existing commit 142b4bf code sync)
@alt-glitch alt-glitch mentioned this pull request May 18, 2026
Closed
@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jun 29, 2026
@teknium1

teknium1 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for this, @liuhao1024 — the diagnosis is correct and the regression tests are well thought out. Closing without merging for two reasons:

1. The target has been refactored away. The _CONTEXT_INVISIBLE_CHARS set this PR edits no longer exists in agent/prompt_builder.py on main. The invisible-char blocklist was consolidated into tools/threat_patterns.py as INVISIBLE_CHARS, now shared by three scanners (context files, the memory tool, and skills_guard.py), so the diff no longer applies.

2. Blanket removal is the wrong fix — and the right one already landed. Dropping U+200D from the set entirely would also stop blocking ZWJ hidden between plain-text characters, which is a real steganographic injection vector — that's why the isolated ZWJ block is intentional (see the discussion on #18581).

The accepted approach is emoji-aware: allow ZWJ when it sits inside an emoji grapheme cluster, still block it in plain text. That's already implemented for cron prompts in tools/cronjob_tools.py (_zwj_has_emoji_neighbour / _strip_legitimate_emoji_zwj, ~L141–168).

Issue #18581 was closed as completed. Appreciate the contribution and the clear repro — credit to you for surfacing the emoji case.

@teknium1 teknium1 closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liuhao1024 @teknium1 @alt-glitch