fix(dataprep): smart_chunk_text single-chunk path leaks internal tensor type when eos_token_id is None

[chuenchen309]

Problem

RawTextDataLoader.smart_chunk_text() (unsloth/dataprep/raw_text.py) has
two branches for building the input_ids output, and they disagree on
output type:

Single-chunk branch (when the whole text fits in chunk_size):

if return_tokenized:
    eos_token_id = getattr(self.tokenizer, "eos_token_id", None)
    if eos_token_id is not None:
        tokens = tokens.tolist() if hasattr(tokens, "tolist") else list(tokens)
        tokens.append(eos_token_id)

    attention_mask = [1] * len(tokens)
    return [{"input_ids": tokens, "attention_mask": attention_mask}]

The tolist()/list() conversion only happens inside the
if eos_token_id is not None: guard. When the tokenizer has no EOS token
configured, that branch never runs, and tokens — whatever
tensor-like object came out of the tokenizer normalization step a few
lines up — is returned as input_ids verbatim.

Multi-chunk branch (a few lines below), for comparison:

chunk_tokens_list = (
    chunk_tokens.tolist() if hasattr(chunk_tokens, "tolist") else list(chunk_tokens)
)
if end_idx == len(tokens) or len(chunk_tokens_list) == chunk_size:
    eos_token_id = getattr(self.tokenizer, "eos_token_id", None)
    if eos_token_id is not None:
        chunk_tokens_list.append(eos_token_id)

Here the conversion runs unconditionally, before checking
eos_token_id — the correct pattern.

Impact

create_causal_dataset() does labels = [list(ids) for ids in input_ids].
list()-ing a tensor produces a list of 0-d tensor elements rather than
plain Python ints, inconsistent with every multi-chunk sample and liable to
break type inference in Dataset.from_dict() / downstream
tokenization/collation code that expects plain ints.

Repro (bare, before fix — via a minimal mock tokenizer, no torch/GPU needed)

loader = RawTextDataLoader(MockTokenizerWithNoEos(), chunk_size=2048, stride=512)
result = loader.smart_chunk_text("hello world short text", chunk_size=2048, stride=512, return_tokenized=True)
type(result[0]["input_ids"])   # MockTensor (or torch.Tensor in practice), not list

Fix

Move the list conversion out of the eos_token_id guard, matching the
multi-chunk branch's existing pattern (2-line diff, single-chunk branch
only).

Testing

Added test_smart_chunk_text_single_chunk_no_eos_returns_plain_list to
tests/test_raw_text.py (reusing the file's existing MockTensor/mock-
tokenizer pattern, no heavy deps). Confirmed:

• red against unfixed code (assertion failure: input_ids was the mock
  tensor type, not a list)
• green after the fix
• full tests/test_raw_text.py (both test functions) passes:
  python3 tests/test_raw_text.py → ✅ All tests passed! /
  ✅ test_smart_chunk_text_single_chunk_no_eos_returns_plain_list passed!

ruff check and the repo's scripts/run_ruff_format.py (kwarg-spacing
formatter): clean.

Note: tests/test_raw_text.py doesn't appear to be wired into any
.github/workflows/*.yml job as far as I could find — a pre-existing repo
characteristic, not something this PR changes. Verified locally instead.

Duplicate-work check

Searched merged/closed PRs and issues for smart_chunk_text /
raw_text.py — found #7126 ("guard smart_chunk_text against stride >=
chunk_size", merged), a different bug in the same function (stride
validation, not the EOS-branch type leak this PR fixes). No open PR
touches this file.

---

Disclosure: I used AI assistance (Claude) to find this bug and draft the
fix and test. I independently traced both branches, reproduced the type
mismatch against unfixed code with a bare mock-tokenizer repro, verified
the fix red→green, and ran the full local test file + ruff before
submitting.

[gemini-code-assist]

Code Review

This pull request updates the smart_chunk_text method in unsloth/dataprep/raw_text.py to ensure that tokens are unconditionally converted to a plain list when returning tokenized single-chunk text, even if the tokenizer does not have an eos_token_id. Additionally, a unit test has been added to tests/test_raw_text.py to verify this behavior. There are no review comments to address, and we have no further feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Etherll]

LGTM!

[Etherll]

Thanks for the PR!