Text replacement took 20GB RAM and more <- because of changed "max_line_len_for_undo"

[Hexaae]

Today I was trying to replace \r\n occurrences with a LF (0x0A, I pressed CTRL+RETURN in the multiline search gadget) in a huge one line json file to make it more readable, and noticed cudatext took more than 50secs and then showed me this warning message:

Then I noticed my used RAM went from 10GB to 29GB (!!), and when I Aborted the process clicking on the button 19GB were finally release after cudatext got force-closed automatically.

Is it ok it takes so much time and memory? Can this be optimized in some way?
Here is the file:

SkaldProject-resources.assets-2261.gz

And my user settings:

{
  "ui_toolbar_show": true,
  "ui_theme": "syn",
  "ui_theme_syntax": "syn",
  "ui_toolbar_theme": "default_16x16",
  "micromap_show": true,
  "minimap_show": true,
  "ui_statusbar_no_sel": "{_ln} {y}/{count}, {_col} {xx}",
  "ui_statusbar_small_sel": "{_ln} {y}/{count}, {_col} {xx}, {_sel} {selchars}",
  "ui_statusbar_str_sel": "{_ln} {y}/{count}, {_col} {xx}, {selchars}:{sel} {_linesel}",
  "ui_statusbar_panels": "caret,C,220|enc,C,120|ends,A,45|lexer,C,140|tabsize,A,75|selmode,A,15|msg,L,4000",
  "ui_max_history_menu": 25,
  "ui_auto_save_session": false,
  "ui_one_instance": true,
  "ruler_show": true,
  "wrap_enabled_max_lines": 360000,
  "unprinted_show": false,
  "unprinted_content": "sedp",
  "font_name": "Consolas",
  "font_size": 10,
  "margin_right": 90,
  "margin_left_1": 0,
  "margin_left": 0,
  "max_line_len_for_undo": 99902400,
  "undo_limit": 9000,
  "find_hi_max_lines": 10000000,
  "find_hi_max_columns": 6000,
  "find_hi_max_line_len": 10000000,
  "find_suggest_sel": true,
  "find_wrapped_blinking": "EdMarkedRangeBg"
}

[Alexey-T]

Without testing, first thinking: undo data takes this memory. You have huge value of the option

max_line_len_for_undo

What if you set it to 10000?

[Alexey-T]

with default settings, Cud makes this replacement without taking lot of mem (tested mem by htop linux app). yes, very long, 2-3 minutes already and not finished.
guess, culprit of lot of memory is that mentioned huge value of the option.

[Hexaae]

Yeah must be that. Tried with 100000 and cudatext warned me (Line) Undo was beyond limit, and continued... memory is stable around 10->~14GB but after 1-2mins indeed is still there...

... It's a borderline case anyway: probably better to use a hex-editor for these cases, f.e. 010Editor took ~20secs. Just a heads up on slow text replacement for huge operations.

[Alexey-T]

maybe it can be optimized in the future (not sure which future). when

• 'ask to confirm' is Off (or all confirms are OK)
• whole-words option is Off
• 'in selection' is Off
• syntax-elements-option is Off

app can do very fast replacement via StringReplace pascal function (works very fast for any string len).

[Hexaae]

Mmhh... that's strange though: I just stopped it clicking on the small button in the bottom-right "Cancel" (was 3%!), and it stopped the operation.
Surprise: it seems to me it already replaced ALL strings actually (!) and was just stuck in some strange internal loop... mmh

EDIT: uh, no I was wrong, indeed there's still a long line not LFed 29404878 Col (!)

[Alexey-T]

I did not confirm it: stopped at 10%...20% and not done.

[Hexaae]

Correct. I added a EDIT in the previous message in the meanwhile.

[Hexaae]

... a question: would be possible to define a custom end-line in cudatext? To parse "\n" or user selected end-of-line string (\n, \r\n, \r....) acting as a end of line for better readability? (<--- advanced feature request)

[Alexey-T]

as I know none apps allow to specify custom substring for EOL. my app also don't allow this. I don't think it's really needed for real world.

[Hexaae]

Uh, there seems to be a small bug with text line "[si] UndoStopped" in localization: %s (line number) shown as ? here:

Translation in it_IT:
UndoStopped=Controllo annullamento interrotto alla linea %s, vedere l'opzione "max_line_len_for_undo": %d (clicca per nascondere)

[Alexey-T]

it is a feature: when line-index is not known by the core (it is 2 for this case, but for some reason code cannot know it), its shown as '?'.

[Alexey-T]

Solution of that replace-problem by Python.

1. open that textfile, badly formatted line with len=30M has index=2
2. in python console of Cud:
   >>> s=ed.get_text_all()
>>> ss=s.replace(r'\r\n', '\n').replace(r'\"', '"')
3. open new tab. and in console again:
   >>> ed.insert(0,0,ss)

[Hexaae]

Good to know, thank you!