Why does appending to TextBox.Text during a loop take up more memory with each iteration?

Question

Short Question

I have a loop that runs 180,000 times. At the end of each iteration it is supposed to append the results to a TextBox, which is updated real-time.

Using MyTextBox.Text += someValue is causing the application to eat huge amounts of memory, and it runs out of available memory after a few thousand records.

Is there a more efficient way of appending text to a TextBox.Text 180,000 times?

Edit I really don't care about the result of this specific case, however I want to know why this seems to be a memory hog, and if there is a more efficient way to append text to a TextBox.

---

Long (Original) Question

I have a small app which reads a list of ID numbers in a CSV file and generates a PDF report for each one. After each pdf file is generated, the ResultsTextBox.Text gets appended with the ID Number of the report that got processed and that it was successfully processed. The process runs on a background thread, so the ResultsTextBox gets updated real-time as items get processed

I am currently running the app against 180,000 ID numbers, however the memory the application is taking up is growing exponentially as time goes by. It starts by around 90K, but by about 3000 records it is taking up roughly 250MB and by 4000 records the application is taking up about 500 MB of memory.

If I comment out the update to the Results TextBox, the memory stays relatively stationary at roughly 90K, so I can assume that writing ResultsText.Text += someValue is what is causing it to eat memory.

My question is, why is this? What is a better way of appending data to a TextBox.Text that doesn't eat memory?

My code looks like this:

try
{
    report.SetParameterValue("Id", id);

    report.ExportToDisk(ExportFormatType.PortableDocFormat,
        string.Format(@"{0}\{1}.pdf", new object[] { outputLocation, id}));

    // ResultsText.Text += string.Format("Exported {0}\r\n", id);
}
catch (Exception ex)
{
    ErrorsText.Text += string.Format("Failed to export {0}: {1}\r\n", 
        new object[] { id, ex.Message });
}

It should also be worth mentioning that the app is a one-time thing and it doesn't matter that it is going to take a few hours (or days :)) to generate all the reports. My main concern is that if it hits the system memory limit, it will stop running.

I'm fine with leaving the line updating the Results TextBox commented out to run this thing, but I would like to know if there is a more memory efficient way of appending data to a TextBox.Text for future projects.

Answer 1

I suspect the reason the memory usage is so large is because textboxes maintain a stack so that the user can undo/redo text. That feature doesn't seem to be required in your case, so try setting IsUndoEnabled to false.

Answer 2

Use TextBox.AppendText(someValue) instead of TextBox.Text += someValue. It's easy to miss since it's on TextBox, not TextBox.Text. Like StringBuilder, this will avoid creating copies of the entire text each time you add something.

It would be interesting to see how this compares to the IsUndoEnabled flag from keyboardP's answer.

Answer 3

Don't append directly to the text property. Use a StringBuilder for the appending, then when done, set the .text to the finished string from the stringbuilder

Answer 4

Instead of using a text box I would do the following:

1. Open up a text file and stream the errors to a log file just in case.
2. Use a list box control to represent the errors to avoid copying potentially massive strings.

Answer 5

Personally, I always use string.Concat* . I remember reading a question here on Stack Overflow years ago that had profiling statistics comparing the commonly-used methods, and (seem) to recall that string.Concat won out.

Nonetheless, the best I can find is this reference question and this specific String.Format vs. StringBuilder question, which mentions that String.Format uses a StringBuilder internally. This makes me wonder if your memory hog lies elsewhere.

**based on James' comment, I should mention that I never do heavy string formatting, as I focus on web-based development.*

Answer 6

Maybe reconsider the TextBox? A ListBox holding string Items will probably perform better.

But the main problem seem to be the requirements, Showing 180,000 items cannot be aimed at a (human) user, neither is changing it in "Real Time".

The preferable way would be to show a sample of the data or a progress indicator.

When you do want to dump it at the poor User, batch string updates. No user could descern more than 2 or 3 changes per second. So if you produce 100/second, make groups of 50.

Answer 7

Some responses have alluded to it, but nobody has outright stated it which is surprising. Strings are immutable which means a String cannot be modified after it is created. Therefore, every time you concatenate to an existing String, a new String Object needs to be created. The memory associated with that String Object also obviously needs to be created, which can get expensive as your Strings become larger and larger. In college, I once made the amateur mistake of concatenating Strings in a Java program that did Huffman coding compression. When you're concatenating extremely large amounts of text, String concatenation can really hurt you when you could have simply used StringBuilder, as some in here have mentioned.

Answer 8

Use the StringBuilder as suggested. Try to estimate the final string size then use that number when instantiating the StringBuilder. StringBuilder sb = new StringBuilder(estSize);

When updating the TextBox just use assignment eg: textbox.text = sb.ToString();

Watch for cross-thread operations as above. However use BeginInvoke. No need to block the background thread while the UI updates.

Answer 9

A) Intro: already mentioned, use StringBuilder

B) Point: don't update too frequently, i.e.

DateTime dtLastUpdate = DateTime.MinValue;

while (condition)
{
    DoSomeWork();
    if (DateTime.Now - dtLastUpdate > TimeSpan.FromSeconds(2))
    {
        _form.Invoke(() => {textBox.Text = myStringBuilder.ToString()});
        dtLastUpdate = DateTime.Now;
    }
}

C) If that's one-time job, use x64 architecture to stay within 2Gb limit.

Answer 10

StringBuilder in ViewModel will avoid string rebindings mess and bind it to MyTextBox.Text. This scenario will increase performance many times over and decrease memory usage.

Answer 11

Something that has not been mentioned is that even if you're performing the operation in the background thread, the update of the UI element itself HAS to happen on the main thread itself (in WinForms anyway).

When updating your textbox, do you have any code that looks like

if(textbox.dispatcher.checkAccess()){
    textbox.text += "whatever";
}else{
    textbox.dispatcher.invoke(...);
}

If so, then your background op is definitely being bottlenecked by the UI Update.

I would suggest that your background op use StringBuilder as noted above, but instead of updating the textbox every cycle, try updating it at regular intervals to see if it increases performance for you.

EDIT NOTE:have not used WPF.

Answer 12

You say memory grows exponentially. No, it is a quadratic growth, i.e. a polynomial growth, which is not as dramatic as an exponential growth.

You are creating strings holding the following number of items:

1 + 2 + 3 + 4 + 5 ... + n = (n^2 + n) /2.

With n = 180,000 you get total memory allocation for 16,200,090,000 items, i.e. 16.2 billion items! This memory will not be allocated at once, but it is a lot of cleanup work for the GC (garbage collector)!

Also, bear in mind, that the previous string (which is growing) must be copied into the new string 179,999 times. The total number of copied bytes goes with n^2 as well!

As others have suggested, use a ListBox instead. Here you can append new strings without creating a huge string. A StringBuild does not help, since you want to display the intermediate results as well.