Extreme memory usage when opening a 36k file in viewer/editor

[johann@…]

When i open the attached README.txt (its the one from the game Empire Earth 2) the used memory of the muc-vm gets incredible high (see attached screenshots) and never falls back to a "healthy" value.

It seems that opening *any* file in the viewer/editor causes more memory usage, wich will never get released.

Maybe this is a 64bit-specific behaviour, since i never had this problem on 32bit.

[johann@…]

Memory usage after opening the file

[johann@…]

Memory usage before opening the file

[johann@…]

Thread dump after opening the file

[johann@…]

The README.txt causing the problem

[nicolas]

Hey there,

We've changed a few things in the text editor and the issue you describe appears to have vanished. On the other hand, I've never been able to reproduce it myself. Would you mind grabbing the latest nightly build and letting me know whether the issue still stands?

Nicolas

[Johann]

Hi Nicolas,
i'm sorry, but the problem still stands with the current nightly build.

Currently, the first time i open the file works. But a second attempt causes the JVM to crash (!) with an:

Exception in thread "AWT-EventQueue?-0" java.lang.OutOfMemoryError?: Java heap space

at com.sun.java.swing.plaf.gtk.GTKNativeEngine.finishPainting(GTKNativeEngine.java:493)
at com.sun.java.swing.plaf.gtk.GTKPainter.paintViewportBorder(GTKPainter.java:597)
at javax.swing.plaf.synth.SynthScrollPaneUI$ViewportBorder?.paintBorder(SynthScrollPaneUI.java:196)
at javax.swing.plaf.synth.SynthScrollPaneUI.paint(SynthScrollPaneUI.java:67)
at javax.swing.plaf.synth.SynthScrollPaneUI.update(SynthScrollPaneUI.java:51)
at javax.swing.JComponent.paintComponent(JComponent.java:763)
at javax.swing.JComponent.paint(JComponent.java:1027)
at javax.swing.JComponent.paintChildren(JComponent.java:864)
at javax.swing.JComponent.paint(JComponent.java:1036)
at javax.swing.JComponent.paintChildren(JComponent.java:864)
at javax.swing.JComponent.paint(JComponent.java:1036)
at javax.swing.JComponent.paintChildren(JComponent.java:864)
at javax.swing.JComponent.paint(JComponent.java:1036)
at javax.swing.JLayeredPane.paint(JLayeredPane.java:564)
at javax.swing.JComponent.paintChildren(JComponent.java:864)
at javax.swing.JComponent.paintToOffscreen(JComponent.java:5129)
at javax.swing.BufferStrategyPaintManager?.paint(BufferStrategyPaintManager?.java:277)
at javax.swing.RepaintManager?.paint(RepaintManager?.java:1217)
at javax.swing.JComponent.paint(JComponent.java:1013)
at java.awt.GraphicsCallback?$PaintCallback?.run(GraphicsCallback?.java:21)
at sun.awt.SunGraphicsCallback?.runOneComponent(SunGraphicsCallback?.java:60)
at sun.awt.SunGraphicsCallback?.runComponents(SunGraphicsCallback?.java:97)
at java.awt.Container.paint(Container.java:1780)
at javax.swing.RepaintManager?.paintDirtyRegions(RepaintManager?.java:814)
at javax.swing.RepaintManager?.paintDirtyRegions(RepaintManager?.java:714)
at javax.swing.RepaintManager?.seqPaintDirtyRegions(RepaintManager?.java:694)
at javax.swing.SystemEventQueueUtilities?$ComponentWorkRequest?.run(SystemEventQueueUtilities?.java:128)
at java.awt.event.InvocationEvent?.dispatch(InvocationEvent?.java:209)
at java.awt.EventQueue?.dispatchEvent(EventQueue?.java:597)
at java.awt.EventDispatchThread?.pumpOneEventForFilters(EventDispatchThread?.java:269)
at java.awt.EventDispatchThread?.pumpEventsForFilter(EventDispatchThread?.java:184)
at java.awt.EventDispatchThread?.pumpEventsForHierarchy(EventDispatchThread?.java:174)

(<unknown>:3603): GdkPixbuf?-CRITICAL **: gdk_pixbuf_get_pixels: assertion `GDK_IS_PIXBUF (pixbuf)' failed

(<unknown>:3603): GdkPixbuf?-CRITICAL **: gdk_pixbuf_get_pixels: assertion `GDK_IS_PIXBUF (pixbuf)' failed

(<unknown>:3603): GdkPixbuf?-CRITICAL **: gdk_pixbuf_get_rowstride: assertion `GDK_IS_PIXBUF (pixbuf)' failed
#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f1bdf6b9840, pid=3603, tid=139757469120848
#
# Java VM: Java HotSpot?(TM) 64-Bit Server VM (11.3-b02 mixed mode linux-amd64)
# Problematic frame:
# C [libmawt.so+0x35840]
#
# An error report file with more information is saved as:
# /data/downloads/muCommander-0_8_4/hs_err_pid3603.log
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

Sometimes even the first attempt to open the file fails :(

[Johann]

Ok, ive tried to track down the problem:

On 32bit:
- Opening various files (20k to 80k) causes a OutOfMemoryError? but the VM is still running. (closing the viewer works, so muc is still alive).
- Opening the attached README.txt: OOMError, but the viewer shows the file

On 64bit:
- Opening muc readme.txt/license.txt -> no prob
- Opening a file containing 500x1500 chars -> high mem usage but works
- Opening the attached README.txt -> OOM + JVM crash (see above)
- Opening files with different encodings: works
- Opening html files: ok

(both Gentoo Linux / Gnome / Java 1.6)

Conclusion:
- Line length is not the problem
- Encoding is not the problem
- Filesize is not the primary problem
- 32bit is more stable than 64bit
- The memory used to draw/load/whatever is never released

see it yourself:

open a ~1 meg file and see on your memory usage
close the viewer
take another look on the memory usage:)

opening a 700k (text-)file increases the resident memory from 90m to 1000m!

[Johann]

Last one for today:

Ok, the bug seems to be related to Linux (?) and JDK 1.6:

Using JDK 1.6: fails (On Gentoo Linux / Gnome 2.26 and Suse / KDE 4.2)
Using JDK 1.5: works partially (high memory usage; both on gentoo and suse)

[nicolas]

Cheers for that in-depth analysis. To be honest, this appears to be an issue with the Linux VM: impossible to reproduce it under OS X or Windows, and the fact that your stack trace includes a segfault is a dead giveaway.

Obviously we'll try to find a workaround, but since this occurs within native code outside of our control, I don't have high hopes that we'll find something, or any time soon at any rate.

In the meantime, might I suggest that you consider using an external text viewer / editor ?
Check the FAQ for instructions on how to replace the internal text editor by one of your choice. I'll let you know if we do find a workaround, but I'd rather you didn't have to work with a crippled muCommander in the meantime.

Cheers,
Nicolas

[Johann]

Ok, that might be a workaround.

I've played a litte bit: A simple jframe form with a textarea loads the readme.txt without problems and much faster than mucommander.
Further i've hacked a litte bit in the TextEditorImpl?: changing the startEditing() method to use a stupid java.io.FileInputStream? is much faster, but after 3 or 4 times opening the file, the memory usage increases very fast (not earlier). This does not happen with my test form..
So the code which triggers the bug is in muc, but i have no clue where. Maybe it is in the Theme management (changing the laf does not help), or somewhere else :(

[nicolas]

That's interesting. How exactly did you change the text reading to a file input stream? Did you skip the encoding detection mechanism, do you ignore encoding issues altogether?
If you wouldn't mind, I'm quite keen on seeing your code and analyse what you did that increased performances. Even if we can't use it live, at least it'll point me in the direction of the culprit.

[Johann]

Ok, some dirty test cases:

First the TestForm?.java: Its a simple JFrame with a JTextArea and a JButton. It loads the file via an FileInputStream?. Testfile: ~45ms, README.txt: 86ms - no problem, no crash.

Current SVN HEAD: Loading testfile: 130ms to 160ms

Test1 - Using a simple fis:
Loading testfile: ~39ms
Loading README.txt: Crash / high memory

Test2 - Disabling the encoding detection and fallback to utf-8
Loading testfile: ~70ms
Loading README.txt: Crash / high memory

Test3 - Disabling the encoding detection and no use of the BOM stream
Loading testfile: ~72ms
Loading README.txt: Crash / high memory

Test4 - Disabling the encoding detection and no use of the BOM stream, and use a FileReader? instead of a BufferedReader?:
Loading testfile: ~77ms
Loading README.txt: First time WORKS, ~60ms, Second time: oom error and high mem but editor visible + closable (no crash)

Sure, the time may vary due to the hdd cache. :(

I've marked the changes in the attached TextEditorImpl?.java. Im sure you know what you have to do to follow the test cases. Sorry, but i'm too lazy to provide a diff for each different case ;)

I used the org.apache.commons.lang.time.StopWatch? class to measure the time. See the attached ViewFrame?.java for some points.
Loading the file seems to be the fastest part. The resizing progress / decorating or whatever takes much time too, especially wrapping the viewer in the scrollpane.

Oh, "testfile" means the loremipsum.txt :)

[Johann]

Testfile