How is the tokenizer supposed to be used?

I am developing an app that could probably use jscpd.  I want to find out the most efficient way to use it.  I see that there is a tokenizer available.  What use would there be for a tokenizer?  Would the search go faster with tokens pre-generated?  If so, is this because the tokens are shorter than the original text when doing the Rabin-Karp search?
