Nasty memory leak in redraws of Chaco + Qt

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Nasty memory leak in redraws of Chaco + Qt

Brennan Sellner
Hi all,

There appears to be a memory leak in Chaco or its supporting libraries
when redrawing using the Qt backend.  I originally noticed this in my
application, but it's reproducible using qt_example.py from python-chaco
[1].  It does not appear to occur when using non-Qt demos (e.g. Chaco's
quiver.py example).

I'm using PySide in my code.  Both PySide and PyQt4 are installed on my
system; it looks like pyface is pulling in PySide for qt_example.py as well.

To reproduce:
1. Run qt_example.py.
2. Run top (-p <PID> is handy here).
3. Pan the plot around wildly with the mouse.
4. Watch the resident size of the program grow by ~27 MB per second.

Eeek!

Zooming in and out exhibits the same behavior as panning.

Simply resizing the window (via the window manager) also has the same
effect, although the rate of growth is necessarily slower, since it
takes longer for me to resize the window.

I'm guessing it's a PySide C++-side issue, since the output of objgraph
and pympler is essentially unchanged between the start and end of the
run.  However, I'm new to hunting Python memory leaks, so it's possible
I'm missing something.

Before I dive down this rabbit hole, is this a known issue, or does
anyone have any suggestions?  I haven't found anything while searching
around, but that doesn't prove the negative.

Also, is there a way to tell traits/enable/chaco to use PyQt, instead of
PySide?  That might demonstrate that it's a PySide-side issue...

Thanks,

-Brennan

------------

OS: Ubuntu 12.04.1

Package versions:
python-chaco, et. al.: 4.0.0
PyQt4: 4.9.1
PySide: 1.1.1

[1] Full disclosure: I had to make two changes to make qt_example run on
my machine:
    1. Change the QApplication constructor to a QApplication.instance()
       call.
    1. Move the app = QtGui.QApplication line to just below the pyface
       import.
    2. Replace the linspace call with a literal list (?!).

My apologies for the disclaimer below; our mail gateway adds it
automatically.

Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Christoph Gohlke
On 2/15/2013 10:47 AM, Brennan Sellner wrote:

> Hi all,
>
> There appears to be a memory leak in Chaco or its supporting libraries
> when redrawing using the Qt backend.  I originally noticed this in my
> application, but it's reproducible using qt_example.py from python-chaco
> [1].  It does not appear to occur when using non-Qt demos (e.g. Chaco's
> quiver.py example).
>
> I'm using PySide in my code.  Both PySide and PyQt4 are installed on my
> system; it looks like pyface is pulling in PySide for qt_example.py as well.
>
> To reproduce:
> 1. Run qt_example.py.
> 2. Run top (-p <PID> is handy here).
> 3. Pan the plot around wildly with the mouse.
> 4. Watch the resident size of the program grow by ~27 MB per second.
>
> Eeek!
>
> Zooming in and out exhibits the same behavior as panning.
>
> Simply resizing the window (via the window manager) also has the same
> effect, although the rate of growth is necessarily slower, since it
> takes longer for me to resize the window.
>
> I'm guessing it's a PySide C++-side issue, since the output of objgraph
> and pympler is essentially unchanged between the start and end of the
> run.  However, I'm new to hunting Python memory leaks, so it's possible
> I'm missing something.
>
> Before I dive down this rabbit hole, is this a known issue, or does
> anyone have any suggestions?  I haven't found anything while searching
> around, but that doesn't prove the negative.
>
> Also, is there a way to tell traits/enable/chaco to use PyQt, instead of
> PySide?  That might demonstrate that it's a PySide-side issue...
>
> Thanks,
>
> -Brennan
>

Maybe related: there was a huge memory leak in matplotlib when used with
PySide. PySide's QImage increases, but never decreases, the reference
count of a string buffer passed to QImage.

<https://github.com/matplotlib/matplotlib/pull/1323>

Christoph
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

John Wiggins
I concur. This sounds like the code which copies the kiva offscreen buffer into a QImage is leaking.
See enable/qt4/image.py

-- John

On Feb 15, 2013, at 12:53 PM, Christoph Gohlke wrote:

> Maybe related: there was a huge memory leak in matplotlib when used with
> PySide. PySide's QImage increases, but never decreases, the reference
> count of a string buffer passed to QImage.
>
> <https://github.com/matplotlib/matplotlib/pull/1323>
>
> Christoph
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Brennan Sellner
In reply to this post by Christoph Gohlke
On 02/15/2013 01:53 PM, Christoph Gohlke wrote:
>
> Maybe related: there was a huge memory leak in matplotlib when used with
> PySide. PySide's QImage increases, but never decreases, the reference
> count of a string buffer passed to QImage.
>
> <https://github.com/matplotlib/matplotlib/pull/1323>
>
> Christoph

Thanks, Christoph.  I'll take a swing at finding something similar in
enable/qt4/image.py, per John's suggestion.  For what it's worth, I'm
running Python 2.7.3: from the comments on that pull request, it wasn't
reproducible with 2.7, but was with 3.0.

-Brennan

My apologies for the disclaimer below; our mail gateway adds it
automatically.
Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Brennan Sellner
In reply to this post by John Wiggins
On 02/15/2013 01:59 PM, John Wiggins wrote:
> I concur. This sounds like the code which copies the kiva offscreen buffer into a QImage is leaking.
> See enable/qt4/image.py
>
> -- John

Unfortunately, that wasn't it.  The refcount of neither data nor
byte_data in enable/qt4/image.py::Window::_window_paint() changes after
the construction (and subsequent deletion) of the QImage.  I confirmed
that this code path was executing; a direct port of Christoph's pull
request had no effect.  His changes may very well be required with
Python 3, but not with Python 2.7.

I mucked about with image.py's _window_paint for a bit, to try to
confirm that the leak was originating there.  Short version: the
QByteArray in _window_paint appears to be simultaneously the source of
the leak and unnecessary.

Long version:

For reference, the unmodified function is:

1  def _window_paint(self, event):
2      if self.control is None:
3         return
4
5      # self._gc is an image context
6      w = self._gc.width()
7      h = self._gc.height()
8      data = self._gc.pixel_map.convert_to_argb32string()
9      byte_data = QtCore.QByteArray(data)
10     image = QtGui.QImage(byte_data, w, h, QtGui.QImage.Format_RGB32)
11
12     rect = QtCore.QRect(0,0,w,h)
13     painter = QtGui.QPainter(self.control)
14     painter.drawImage(rect, image)

Replacing the QImage construction on line 10 with a simple black-filled
QImage had no effect on the leak, so it's not QImage holding onto a
reference:

  #image = QtGui.QImage(byte_data, w, h, QtGui.QImage.Format_RGB32)
  image = QtGui.QImage(w, h, QtGui.QImage.Format_RGB32)
  image.fill(0)

However, additionally commenting out the construction of the QByteArray
on line 9 eliminated the leak.  Progress!  It's not very useful to just
paint black images, though.

The Qt docs for QByteArray state that it makes a deep copy of its
argument.  Calling `byte_data.clear()` (to clear out its internal
storage) and `del byte_data` (just for the heck of it) at the end of the
function had no effect on the leak.

However, the QImage docs don't actually include a QByteArray
constructor.  In fact, removing line 9 and passing 'data' into QImage()
on line 10 instead of byte_data seems to work just fine.

Armed with new search terms, I quickly turned up an enable pull request
that was merged a year ago with exactly the same fix:

  https://github.com/enthought/enable/pull/42

Sigh.

So, I guess this all boils down to: how do I tell what the latest
released version of ETS/enable/chaco/kiva is, and how that maps against
github?  I thought 4.0.0 was pretty up-to-date, and I'm using the 4.1.0
packages on newer versions of Ubuntu, but they both have a bug that
appears to have been fixed a year ago.  If release version/date
information is on github or www.enthought.com, I'm completely missing it.

Thanks,

-Brennan

My apologies for the disclaimer below; our mail gateway adds it
automatically.

Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Robert Kern
On Fri, Feb 15, 2013 at 8:47 PM, Brennan Sellner <[hidden email]> wrote:

> So, I guess this all boils down to: how do I tell what the latest
> released version of ETS/enable/chaco/kiva is, and how that maps against
> github?  I thought 4.0.0 was pretty up-to-date, and I'm using the 4.1.0
> packages on newer versions of Ubuntu, but they both have a bug that
> appears to have been fixed a year ago.  If release version/date
> information is on github or www.enthought.com, I'm completely missing it.

PyPI always lists the latest release.

  http://pypi.python.org/pypi/enable

We always tag releases in git if you need to do historical diving:

  https://github.com/enthought/enable/tags

--
Robert Kern
Enthought
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Brennan Sellner
On 02/15/2013 03:51 PM, Robert Kern wrote:
> PyPI always lists the latest release.
>
>   http://pypi.python.org/pypi/enable
>
> We always tag releases in git if you need to do historical diving:
>
>   https://github.com/enthought/enable/tags

Thanks, Robert.  It wasn't clear (to me, at least) what the system of
record was in terms of official releases.  I use github rarely enough
that I didn't notice the 'tags' link. :-)

On a self-serving note, I don't suppose there's an Ubuntu PPA floating
around with 4.2 in it, is there?  I haven't been able to find one on
Launchpad, so I doubt it, but I figured I'd ask before I rolled my own
packages.

Thanks for the help everyone!

-Brennan

Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Chris Colbert
I seem to  recall that we fixed a bug due to a leak in in QByteArray last year. 

I've cc'ed Deepankar, whom I seem to recall was the one who fixed it. 


On Fri, Feb 15, 2013 at 4:36 PM, Brennan Sellner <[hidden email]> wrote:
On 02/15/2013 03:51 PM, Robert Kern wrote:
> PyPI always lists the latest release.
>
>   http://pypi.python.org/pypi/enable
>
> We always tag releases in git if you need to do historical diving:
>
>   https://github.com/enthought/enable/tags

Thanks, Robert.  It wasn't clear (to me, at least) what the system of
record was in terms of official releases.  I use github rarely enough
that I didn't notice the 'tags' link. :-)

On a self-serving note, I don't suppose there's an Ubuntu PPA floating
around with 4.2 in it, is there?  I haven't been able to find one on
Launchpad, so I doubt it, but I figured I'd ask before I rolled my own
packages.

Thanks for the help everyone!

-Brennan

Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev


_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Nasty memory leak in redraws of Chaco + Qt

Brennan Sellner
On 02/16/2013 01:32 AM, Chris Colbert wrote:
> I seem to  recall that we fixed a bug due to a leak in in QByteArray
> last year.
>
> I've cc'ed Deepankar, whom I seem to recall was the one who fixed it.

Yep, I found his pull request
(https://github.com/enthought/enable/pull/42), but only after I had
replicated his work and googled for QByteArray. :-)

Oh, and for those following along at home, pypi-install works swimmingly
to get the latest enable/chaco/etc. packages into Ubuntu.

Thanks,

-Brennan

My apologies for the disclaimer below; our email gateway adds it
automatically.

Email Confidentiality Notice

The information contained in this transmission is confidential, proprietary or privileged and may be subject to protection under the law. This message is intended for the sole use of the individual or entity to whom it's addressed. If you are not the intended recipient, you are notified that any use, distribution or copying of the message is strictly prohibited and may subject you to criminal or civil penalties. If you received this transmission in error, please contact the sender immediately by replying to this email and delete the material from any computer.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev