is chaco faster than matplotlib

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

is chaco faster than matplotlib

Jeremy Lewi

Hi,

 

We’re doing a lot of “interactive” plotting.  We basically have scatter plots and colormaps that we want to update 2-3 times a second. We are currently using matplotlib which is very slow. The chaco FAQ says chaco is faster and I was hoping people could confirm this either by confirming that they did some benchmarking or providing some technical explanation as to why chaco is faster for repeatedly updating a plot.

 

Thanks

 

J

 

Jeremy Lewi

Engineering Scientist

The Intellisis Corporation

[hidden email]

 


_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Gary Pajer
An existence theorem:  my data acquisition app updates  chaco scatter plots and color maps at about that rate ... 2-3 frames per second.  I don't think chaco was the rate limiting factor.  I think it was the data acquisition rate.   I'd go back and check, but the hardware is not available at the moment.  (I had three scatter plots and one color map going simultaneously on different tabs.  It's not clear to me how much plotting was going on in the tabs that were not current.  Could be that all four were updating at the same time.  But I don't know if that was true or not.)

-gary



On Fri, Oct 29, 2010 at 11:33 AM, Jeremy Lewi <[hidden email]> wrote:

Hi,

 

We’re doing a lot of “interactive” plotting.  We basically have scatter plots and colormaps that we want to update 2-3 times a second. We are currently using matplotlib which is very slow. The chaco FAQ says chaco is faster and I was hoping people could confirm this either by confirming that they did some benchmarking or providing some technical explanation as to why chaco is faster for repeatedly updating a plot.

 

Thanks

 

J

 

Jeremy Lewi

Engineering Scientist

The Intellisis Corporation

[hidden email]

 


_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev



_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Peter Wang
On Mon, Nov 1, 2010 at 11:58 AM, Gary Pajer <[hidden email]> wrote:
> An existence theorem:  my data acquisition app updates  chaco scatter plots
> and color maps at about that rate ... 2-3 frames per second.  I don't think
> chaco was the rate limiting factor.  I think it was the data acquisition
> rate.   I'd go back and check, but the hardware is not available at the
> moment.  (I had three scatter plots and one color map going simultaneously
> on different tabs.  It's not clear to me how much plotting was going on in
> the tabs that were not current.  Could be that all four were updating at the
> same time.  But I don't know if that was true or not.)

Thanks for the info, Gary!

Based on your description, I can pretty confidently say that the
background tabs were not rendering.  Depending on how you wired up
your event handlers, data may or may not have been getting pushed into
the data sources, but even then, those only trigger invalidation
events.  No actual rendering takes place unless the underlying
windowing system issues a Paint event to the widget containing the
Chaco plot.

-Peter
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Peter Wang
In reply to this post by Jeremy Lewi
On Fri, Oct 29, 2010 at 10:33 AM, Jeremy Lewi <[hidden email]> wrote:
> Hi,
> We’re doing a lot of “interactive” plotting.  We basically have scatter
> plots and colormaps that we want to update 2-3 times a second. We are
> currently using matplotlib which is very slow. The chaco FAQ says chaco is
> faster and I was hoping people could confirm this either by confirming that
> they did some benchmarking or providing some technical explanation as to why
> chaco is faster for repeatedly updating a plot.

Hi Jeremy,

I don't have actual benchmarks, but I can provide a technical
explanation of how Chaco works.  Essentially, when you build a Chaco
plot, you are wiring up objects and components to form a miniature
visualization pipeline.  When the windowing system wants to display
the Chaco plot, the pipeline goes into action, grabbing the latest
data that's available at the data sources, then transforming them into
screen space, and then calling on the underlying graphics library to
render.

Each of these steps is heavily vectorized with numpy, and the entire
library is written with interactive display in mind.  We clip and mask
out invisible data, so that we don't spend time trying to render lines
or points that are off the screen.  Many expensive computations are
cached whenever possible, and it is even possible to backbuffer a plot
so that overlays which need to render on top can be drawn very
quickly, and the underlying plot can repaint at interactive rates
regardless of the underlying data.

So, while it usually takes a little more code to get a Chaco plot set
up than a pylab script, you can instantly start pumping a tremendous
amount of data through it and still have interactivity.

HTH,
Peter
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Dav Clark
In reply to this post by Jeremy Lewi
<base href="x-msg://134/">
On Oct 29, 2010, at 8:33 AM, Jeremy Lewi wrote:

We’re doing a lot of “interactive” plotting.  We basically have scatter plots and colormaps that we want to update 2-3 times a second. We are currently using matplotlib which is very slow. The chaco FAQ says chaco is faster and I was hoping people could confirm this either by confirming that they did some benchmarking or providing some technical explanation as to why chaco is faster for repeatedly updating a plot.

I've not done an extensive comparison, but Chaco definitely allows a level of interactivity that I've never seen implemented in a Matplotlib app - e.g. dragging of various overlays, adjusting controls in real-time, etc. You might look at matplotlib-traits integration if you're interested:


I've had trouble with a GridContainer containing about 10 plots updating "slowly" (I wanted a smoothly scrolling 60 Hz update or so for a crisp animation - I could get about a 10Hz update), but the problem wasn't with the redraw - something odd in the event logic. I'm guessing your 2-3 second refresh will be a snap for Chaco, and the problem I had may well be addressed by now - that was in late 2008!

Cheers,
Dav


_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

bryce hendrix-2
On Mon, Nov 1, 2010 at 3:09 PM, Dav Clark <[hidden email]> wrote:
On Oct 29, 2010, at 8:33 AM, Jeremy Lewi wrote:

We’re doing a lot of “interactive” plotting.  We basically have scatter plots and colormaps that we want to update 2-3 times a second. We are currently using matplotlib which is very slow. The chaco FAQ says chaco is faster and I was hoping people could confirm this either by confirming that they did some benchmarking or providing some technical explanation as to why chaco is faster for repeatedly updating a plot.

I've not done an extensive comparison, but Chaco definitely allows a level of interactivity that I've never seen implemented in a Matplotlib app - e.g. dragging of various overlays, adjusting controls in real-time, etc. You might look at matplotlib-traits integration if you're interested:


I've had trouble with a GridContainer containing about 10 plots updating "slowly" (I wanted a smoothly scrolling 60 Hz update or so for a crisp animation - I could get about a 10Hz update), but the problem wasn't with the redraw - something odd in the event logic. I'm guessing your 2-3 second refresh will be a snap for Chaco, and the problem I had may well be addressed by now - that was in late 2008!


From my experience, the wx event loop won't process events faster than about once per 60ms. You may be able to get paint updates faster than that, but I wouldn't expect much more from wx. I haven't tried to push Qt yet, it wouldn't surprise me if it was faster.

Bryce

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Fredrik Östlund
In reply to this post by Jeremy Lewi
Hi,

about the discussion about whether Matplotlib or Chaco is faster. This is
interesting for me too. I am using chaco for plotting large amounts of data
(~50000 datapoints). The plots has to update at about 5 Hz. This is possible
but stresses the processor quite a lot. To get around the problem I started
to derive ArrayPlotData, overriding the get_data method returning a
downsampled array. The amount of downsampling was related to the current
range being shown. That worked, in principle, but I never really used it
because of lack of time. Now, when the discussion here started, I thought
maybe now is a good time to finish this and to get some help.

Does anyone know if there's a "prettier" way of doing this? Looking though
the LinePlot class a bit I came across a _downsample method. I never found
any evidence of this method getting called ever. Does the functionality
already exist or is this something else?

Fredrik

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Brad Buran
I recall reading somewhere that you are expected to implement the
_downsample method because the optimal algorithm depends on the type
of data you're collecting.  If you want to see an example of how I've
implemented it, you can see the code at
http://bitbucket.org/bburan/neurobehavior/src/tip/cns/widgets/views/decimated_plot.py
(TimeSeries._downsample).

Essentially I subclassed BaseXYPlot and added my own _downsample
function that factors in the number of screen pixels the plot covers.
Since my downsampling method extracts the maximum and minimum of every
N points, I overrode the _render function to plot a vertical line for
each time point that indicates the minimum and maximum extent of the N
samples that time point represents.

With the downsampling, it's fast enough to plot 16 channels of neural
data acquired at 25 kHz per channel (while allowing you to pan/scroll
and zoom around the plot).

Brad

On Thu, Nov 4, 2010 at 10:15 AM, Fredrik Östlund
<[hidden email]> wrote:

> Hi,
>
> about the discussion about whether Matplotlib or Chaco is faster. This is
> interesting for me too. I am using chaco for plotting large amounts of data
> (~50000 datapoints). The plots has to update at about 5 Hz. This is possible
> but stresses the processor quite a lot. To get around the problem I started
> to derive ArrayPlotData, overriding the get_data method returning a
> downsampled array. The amount of downsampling was related to the current
> range being shown. That worked, in principle, but I never really used it
> because of lack of time. Now, when the discussion here started, I thought
> maybe now is a good time to finish this and to get some help.
>
> Does anyone know if there's a "prettier" way of doing this? Looking though
> the LinePlot class a bit I came across a _downsample method. I never found
> any evidence of this method getting called ever. Does the functionality
> already exist or is this something else?
>
> Fredrik
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Fredrik Östlund
In reply to this post by Jeremy Lewi
Many thanks Brad,

I think that will help me a lot actually. My data are realtime mass spectra
and I'm also interested in seeing the noise baseline. If overriding the
_render method (which I'm pretty sure I will do after seeing your code), is
there really any advantage doing the downsampling in _downsample? Why not
directly in the _render method?

best regards

Fredrik




Date: Thu, 4 Nov 2010 10:42:29 -0400
From: Brad Buran <[hidden email]>
Subject: Re: [Enthought-Dev] is chaco faster than matplotlib
To: [hidden email]
Message-ID:
<[hidden email]>
Content-Type: text/plain; charset=UTF-8

I recall reading somewhere that you are expected to implement the
_downsample method because the optimal algorithm depends on the type
of data you're collecting.  If you want to see an example of how I've
implemented it, you can see the code at
http://bitbucket.org/bburan/neurobehavior/src/tip/cns/widgets/views/decimated_plot.py
(TimeSeries._downsample).

Essentially I subclassed BaseXYPlot and added my own _downsample
function that factors in the number of screen pixels the plot covers.
Since my downsampling method extracts the maximum and minimum of every
N points, I overrode the _render function to plot a vertical line for
each time point that indicates the minimum and maximum extent of the N
samples that time point represents.

With the downsampling, it's fast enough to plot 16 channels of neural
data acquired at 25 kHz per channel (while allowing you to pan/scroll
and zoom around the plot).

Brad

On Thu, Nov 4, 2010 at 10:15 AM, Fredrik ?stlund
<[hidden email]> wrote:

> Hi,
>
> about the discussion about whether Matplotlib or Chaco is faster. This is
> interesting for me too. I am using chaco for plotting large amounts of
> data
> (~50000 datapoints). The plots has to update at about 5 Hz. This is
> possible
> but stresses the processor quite a lot. To get around the problem I
> started
> to derive ArrayPlotData, overriding the get_data method returning a
> downsampled array. The amount of downsampling was related to the current
> range being shown. That worked, in principle, but I never really used it
> because of lack of time. Now, when the discussion here started, I thought
> maybe now is a good time to finish this and to get some help.
>
> Does anyone know if there's a "prettier" way of doing this? Looking though
> the LinePlot class a bit I came across a _downsample method. I never found
> any evidence of this method getting called ever. Does the functionality
> already exist or is this something else?
>
> Fredrik
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>


------------------------------

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Brad Buran
I believe that get_screen_points (which is required to gather up all
the data points to be passed to _render) is just a stub in BaseXYPlot
so you'd need to flesh that out as well.  If you start with a subclass
of BaseXYPlot, keep in mind that get_screen_points computes the data
to screen coordinate transform before passing the points to _render.
If you want this (potentially) expensive mapping operation only on the
downsampled data, you will need to implement _render or
get_screen_points.

The way I've set up my code:

1. Get the data and clip it to the region that will be visible on the
screen (_gather_points)
2. Downsample the data (_downsample)
3. Compute data to screen coordinates transform on the downsampled
data (also in _downsample)
4. Plot the downsampled screen coordinates (_render)

Attached is a screenshot of how the plots look.  It's very easy to see
the noise baseline and deviations from baseline.

Brad

On Thu, Nov 4, 2010 at 3:58 PM, Fredrik Östlund
<[hidden email]> wrote:

> Many thanks Brad,
>
> I think that will help me a lot actually. My data are realtime mass spectra
> and I'm also interested in seeing the noise baseline. If overriding the
> _render method (which I'm pretty sure I will do after seeing your code), is
> there really any advantage doing the downsampling in _downsample? Why not
> directly in the _render method?
>
> best regards
>
> Fredrik
>
>
>
>
> Date: Thu, 4 Nov 2010 10:42:29 -0400
> From: Brad Buran <[hidden email]>
> Subject: Re: [Enthought-Dev] is chaco faster than matplotlib
> To: [hidden email]
> Message-ID:
> <[hidden email]>
> Content-Type: text/plain; charset=UTF-8
>
> I recall reading somewhere that you are expected to implement the
> _downsample method because the optimal algorithm depends on the type
> of data you're collecting.  If you want to see an example of how I've
> implemented it, you can see the code at
> http://bitbucket.org/bburan/neurobehavior/src/tip/cns/widgets/views/decimated_plot.py
> (TimeSeries._downsample).
>
> Essentially I subclassed BaseXYPlot and added my own _downsample
> function that factors in the number of screen pixels the plot covers.
> Since my downsampling method extracts the maximum and minimum of every
> N points, I overrode the _render function to plot a vertical line for
> each time point that indicates the minimum and maximum extent of the N
> samples that time point represents.
>
> With the downsampling, it's fast enough to plot 16 channels of neural
> data acquired at 25 kHz per channel (while allowing you to pan/scroll
> and zoom around the plot).
>
> Brad
>
> On Thu, Nov 4, 2010 at 10:15 AM, Fredrik ?stlund
> <[hidden email]> wrote:
>> Hi,
>>
>> about the discussion about whether Matplotlib or Chaco is faster. This is
>> interesting for me too. I am using chaco for plotting large amounts of
>> data
>> (~50000 datapoints). The plots has to update at about 5 Hz. This is
>> possible
>> but stresses the processor quite a lot. To get around the problem I
>> started
>> to derive ArrayPlotData, overriding the get_data method returning a
>> downsampled array. The amount of downsampling was related to the current
>> range being shown. That worked, in principle, but I never really used it
>> because of lack of time. Now, when the discussion here started, I thought
>> maybe now is a good time to finish this and to get some help.
>>
>> Does anyone know if there's a "prettier" way of doing this? Looking though
>> the LinePlot class a bit I came across a _downsample method. I never found
>> any evidence of this method getting called ever. Does the functionality
>> already exist or is this something else?
>>
>> Fredrik
>>
>> _______________________________________________
>> Enthought-Dev mailing list
>> [hidden email]
>> https://mail.enthought.com/mailman/listinfo/enthought-dev
>>
>
>
> ------------------------------
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev

multichannel timeseries.png (217K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: is chaco faster than matplotlib

Fredrik Östlund
Hi Brad,

just wanted to report in. I couldn't use your code directly since my x-axis
is not linear (also providing a perfect example on why downsampling isn't
general).
I ended up doing points 1-3 in the _gather_points method and 4 pretty much
like you did. Works really well, plus that I now finally understand how the
chaco renderes are working.

thanks a bunch

Fredrik





-----Original Message-----
From: Brad Buran
Sent: Thursday, November 04, 2010 11:26 PM
To: [hidden email]
Subject: Re: [Enthought-Dev] is chaco faster than matplotlib

I believe that get_screen_points (which is required to gather up all
the data points to be passed to _render) is just a stub in BaseXYPlot
so you'd need to flesh that out as well.  If you start with a subclass
of BaseXYPlot, keep in mind that get_screen_points computes the data
to screen coordinate transform before passing the points to _render.
If you want this (potentially) expensive mapping operation only on the
downsampled data, you will need to implement _render or
get_screen_points.

The way I've set up my code:

1. Get the data and clip it to the region that will be visible on the
screen (_gather_points)
2. Downsample the data (_downsample)
3. Compute data to screen coordinates transform on the downsampled
data (also in _downsample)
4. Plot the downsampled screen coordinates (_render)

Attached is a screenshot of how the plots look.  It's very easy to see
the noise baseline and deviations from baseline.

Brad

On Thu, Nov 4, 2010 at 3:58 PM, Fredrik Östlund
<[hidden email]> wrote:

> Many thanks Brad,
>
> I think that will help me a lot actually. My data are realtime mass
> spectra
> and I'm also interested in seeing the noise baseline. If overriding the
> _render method (which I'm pretty sure I will do after seeing your code),
> is
> there really any advantage doing the downsampling in _downsample? Why not
> directly in the _render method?
>
> best regards
>
> Fredrik
>
>
>
>
> Date: Thu, 4 Nov 2010 10:42:29 -0400
> From: Brad Buran <[hidden email]>
> Subject: Re: [Enthought-Dev] is chaco faster than matplotlib
> To: [hidden email]
> Message-ID:
> <[hidden email]>
> Content-Type: text/plain; charset=UTF-8
>
> I recall reading somewhere that you are expected to implement the
> _downsample method because the optimal algorithm depends on the type
> of data you're collecting.  If you want to see an example of how I've
> implemented it, you can see the code at
> http://bitbucket.org/bburan/neurobehavior/src/tip/cns/widgets/views/decimated_plot.py
> (TimeSeries._downsample).
>
> Essentially I subclassed BaseXYPlot and added my own _downsample
> function that factors in the number of screen pixels the plot covers.
> Since my downsampling method extracts the maximum and minimum of every
> N points, I overrode the _render function to plot a vertical line for
> each time point that indicates the minimum and maximum extent of the N
> samples that time point represents.
>
> With the downsampling, it's fast enough to plot 16 channels of neural
> data acquired at 25 kHz per channel (while allowing you to pan/scroll
> and zoom around the plot).
>
> Brad
>
> On Thu, Nov 4, 2010 at 10:15 AM, Fredrik ?stlund
> <[hidden email]> wrote:
>> Hi,
>>
>> about the discussion about whether Matplotlib or Chaco is faster. This is
>> interesting for me too. I am using chaco for plotting large amounts of
>> data
>> (~50000 datapoints). The plots has to update at about 5 Hz. This is
>> possible
>> but stresses the processor quite a lot. To get around the problem I
>> started
>> to derive ArrayPlotData, overriding the get_data method returning a
>> downsampled array. The amount of downsampling was related to the current
>> range being shown. That worked, in principle, but I never really used it
>> because of lack of time. Now, when the discussion here started, I thought
>> maybe now is a good time to finish this and to get some help.
>>
>> Does anyone know if there's a "prettier" way of doing this? Looking
>> though
>> the LinePlot class a bit I came across a _downsample method. I never
>> found
>> any evidence of this method getting called ever. Does the functionality
>> already exist or is this something else?
>>
>> Fredrik
>>
>> _______________________________________________
>> Enthought-Dev mailing list
>> [hidden email]
>> https://mail.enthought.com/mailman/listinfo/enthought-dev
>>
>
>
> ------------------------------
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>






_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev 

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev