data structure for multiple lines?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

data structure for multiple lines?

Anne Archibald-2
Hi,

I'm trying to render a scene with many field lines. I've used an ODE
solver to map the field lines already (plus I've transformed them
after the fact), so I'm not using mayavi's streamlines object. Instead
I draw each one with a separate plot3d command. This leads to two
problems:

* The scene redraws after drawing every line, even if I simply run
"python myscript.py".

* Adding each line takes longer and longer. I think this is because
they are all added to the scene graph, which means the scene graph
gets large and ugly (~hundreds of lines). This ugliness is certainly
visible when I try running things in the mayavi2 application.

Is there an object appropriate for handling a collection of lines
(tubes)? This would certainly simplify the scene graph, and after all
I do want all the lines treated the same - I just don't want to
connect the end of one to the beginning of the next. Maybe this is
just a very difficult task for mayavi? My machine is not especially
fast, but I'm also not throwing all that many polygons (<100 segments
per plot3d, ~hundred plot3ds) at it. Here's a small script that
demonstrates the problem:

import enthought.mayavi.mlab as mlab
if __name__=='__main__':
   n = 20
   for i in range(n):
       for j in range(n):
           mlab.plot3d([i]*2,[j]*2,[0,1])

   mlab.show()

Thanks,
Anne
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Gael Varoquaux
Hi Anne,

On Sun, May 02, 2010 at 04:51:41PM -0400, Anne Archibald wrote:
> I'm trying to render a scene with many field lines. I've used an ODE
> solver to map the field lines already (plus I've transformed them
> after the fact), so I'm not using mayavi's streamlines object. Instead
> I draw each one with a separate plot3d command. This leads to two
> problems:

> * The scene redraws after drawing every line, even if I simply run
> "python myscript.py".

That's easy to avoid, and a good practice when scripting rather than
working interactively:

http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/tips.html#accelerating-a-mayavi-script

> * Adding each line takes longer and longer. I think this is because
> they are all added to the scene graph, which means the scene graph
> gets large and ugly (~hundreds of lines). This ugliness is certainly
> visible when I try running things in the mayavi2 application.

> Is there an object appropriate for handling a collection of lines
> (tubes)?

Yes. Actually the key to making this work efficiently is to understand
what is going on behind the scene: the mlab.plot3d creates a data source
(a polydata:
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/data.html#polydata)
on which a small set of filters are applied to make it look like tubes.

The polydata structure is a very general structure made of points that
can be linked together to form either lines or surfaces. That's all the
plot3d does: create a very specific polydata consisting of points linked
to form exactly one line.

Now, the trick to be more efficient, is to create only one polydata
structure that has several lines. The code to do this can be infered from
the various examples plotting graphes, eg
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/auto/example_flight_graph.html

To get you started quicker, I am copy-pasting some of my code that did
exactly this in my research:

################################################################################
for track, index, in zip(tracks, track_ids):#[::50]:
    this_length = np.isnan(track[:, 0]).argmax()
    track = track[:this_length]
    positions.append(track)
    connections.append(np.vstack(
                    [np.arange(length,   length + this_length - 1.5),
                     np.arange(length+1, length + this_length - .5)]
                         ).T)
    scalars.append(index*np.ones(this_length))
    length += this_length



positions   = np.vstack(positions)
connections = np.vstack(connections)
scalars     = np.hstack(scalars)

x, y, z = positions.T

src = mlab.pipeline.scalar_scatter(x, y, z, scalars)
src.mlab_source.dataset.lines = connections
tubes = mlab.pipeline.tube(
    mlab.pipeline.stripper(src), tube_radius=.8)
mlab.pipeline.surface(tubes, colormap='spectral')
################################################################################

If you know data that is easy to generate, or to download that provides a
good illustration for this problem, I'd love to add an example on it to
the official Mayavi documentation.

HTH,

Gaël
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Gael Varoquaux
In reply to this post by Anne Archibald-2
On Sun, May 02, 2010 at 04:51:41PM -0400, Anne Archibald wrote:
> * Adding each line takes longer and longer. I think this is because
> they are all added to the scene graph, which means the scene graph
> gets large and ugly (~hundreds of lines). This ugliness is certainly
> visible when I try running things in the mayavi2 application.

> Is there an object appropriate for handling a collection of lines
> (tubes)? This would certainly simplify the scene graph, and after all
> I do want all the lines treated the same - I just don't want to
> connect the end of one to the beginning of the next. Maybe this is
> just a very difficult task for mayavi?

In addition to my previous (short) answer, here is a fully-running
example that shows how to do this.

HTH,

Gaël

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev

plotting_many_lines.py (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Anne Archibald-2
On 4 May 2010 04:56, Gael Varoquaux <[hidden email]> wrote:

> On Sun, May 02, 2010 at 04:51:41PM -0400, Anne Archibald wrote:
>> * Adding each line takes longer and longer. I think this is because
>> they are all added to the scene graph, which means the scene graph
>> gets large and ugly (~hundreds of lines). This ugliness is certainly
>> visible when I try running things in the mayavi2 application.
>
>> Is there an object appropriate for handling a collection of lines
>> (tubes)? This would certainly simplify the scene graph, and after all
>> I do want all the lines treated the same - I just don't want to
>> connect the end of one to the beginning of the next. Maybe this is
>> just a very difficult task for mayavi?
>
> In addition to my previous (short) answer, here is a fully-running
> example that shows how to do this.

Thanks a lot, that's exactly what I was hoping for!

To avoid future questions like this (e.g., how do I make a colormap
given the sequence of RGB colors I want to be in it?), can you give me
some pointers on navigating the documentation? Once I reach the limit
of what's covered in the user manual - basically just elementary usage
of mlab - I get totally lost and don't have any idea where to look
next.

Thanks,
Anne

> HTH,
>
> Gaël
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
>
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Gael Varoquaux
On Thu, May 06, 2010 at 01:41:06AM -0400, Anne Archibald wrote:
> To avoid future questions like this (e.g., how do I make a colormap
> given the sequence of RGB colors I want to be in it?),

I added an example and an entry in the documentation about this only
recently. It's not yet on line, but you can find it here:

https://svn.enthought.com/enthought/browser/Mayavi/trunk/examples/mayavi/mlab/custom_colormap.py

> can you give me some pointers on navigating the documentation? Once I
> reach the limit of what's covered in the user manual - basically just
> elementary usage of mlab - I get totally lost and don't have any idea
> where to look next.

Yes. There is an interesting challenge here: with mlab and it's
documentation, we have more or less managed to steer the users to get the
most important parts done easily. The difficulty is to help them make
the step required to understand a bit more the inner workings of VTK and
Mayavi, so that they can more easily find the answers to their
complicated usecases by themselves. This is the goal of the more and more
complex examples that we add to the example list, as well as the chapter
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_use.html
However, there is still some simple documentation work to be done on
these texts, I believe.

If you find something fundemental that we could be doing better, I'd love
to hear and learn. It's probably a lot of small details that need to be
improved. We are working on it, but very slowly: writing docs does take a
lot of time, and neither Prabhu nor I are able to free enough time to
work on Mayavi (of course, we take patches :>).

On the bright side: it seems to me that users stumble on more and more
difficult problems, as time goes.

HTH,

Gaël
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Anne Archibald
On 6 May 2010 01:50, Gael Varoquaux <[hidden email]> wrote:
> On Thu, May 06, 2010 at 01:41:06AM -0400, Anne Archibald wrote:
>> To avoid future questions like this (e.g., how do I make a colormap
>> given the sequence of RGB colors I want to be in it?),
>
> I added an example and an entry in the documentation about this only
> recently. It's not yet on line, but you can find it here:
>
> https://svn.enthought.com/enthought/browser/Mayavi/trunk/examples/mayavi/mlab/custom_colormap.py

Ah, thanks, that helps.

>> can you give me some pointers on navigating the documentation? Once I
>> reach the limit of what's covered in the user manual - basically just
>> elementary usage of mlab - I get totally lost and don't have any idea
>> where to look next.
>
> Yes. There is an interesting challenge here: with mlab and it's
> documentation, we have more or less managed to steer the users to get the
> most important parts done easily. The difficulty is to help them make
> the step required to understand a bit more the inner workings of VTK and
> Mayavi, so that they can more easily find the answers to their
> complicated usecases by themselves. This is the goal of the more and more
> complex examples that we add to the example list, as well as the chapter
> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_use.html
> However, there is still some simple documentation work to be done on
> these texts, I believe.
>
> If you find something fundemental that we could be doing better, I'd love
> to hear and learn. It's probably a lot of small details that need to be
> improved. We are working on it, but very slowly: writing docs does take a
> lot of time, and neither Prabhu nor I are able to free enough time to
> work on Mayavi (of course, we take patches :>).
>
> On the bright side: it seems to me that users stumble on more and more
> difficult problems, as time goes.

I think the basic problem is that there are two different incompatible
ways to work with mayavi: using the mlab api to just plot things, and
constructing vtk pipelines. I find I stumble when I try to go from one
to the other. I think part of the problem is that there are no (or I
haven't found) examples of doing simple things with the pipeline
approach. Would it be reasonable to, for every example in the mlab
documentation, also offer a version, linked from the same page, that
offers "how to do this with pipelines"?

The second problem is that the documentation of the vtk objects you
have to use to work with pipelines is nonexistent, or separate from
the mlab documentation, or not indexed, or somehow otherwise hard to
find. How, for example, should I have discovered that I wanted the
module_manager.scalar_lut_manager attribute of the mesh I created?
What's a module manager, and when would I use it? I ask these
questions not so much for me, since I can look this stuff up (somehow)
now that I know it's needed, as to get them into the documentation
somewhere.

Thanks,
Anne

> HTH,
>
> Gaël
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Robert Cimrman
On 05/06/10 08:25, Anne Archibald wrote:

> On 6 May 2010 01:50, Gael Varoquaux<[hidden email]>  wrote:
>> On Thu, May 06, 2010 at 01:41:06AM -0400, Anne Archibald wrote:
>>> To avoid future questions like this (e.g., how do I make a colormap
>>> given the sequence of RGB colors I want to be in it?),
>>
>> I added an example and an entry in the documentation about this only
>> recently. It's not yet on line, but you can find it here:
>>
>> https://svn.enthought.com/enthought/browser/Mayavi/trunk/examples/mayavi/mlab/custom_colormap.py
>
> Ah, thanks, that helps.
>
>>> can you give me some pointers on navigating the documentation? Once I
>>> reach the limit of what's covered in the user manual - basically just
>>> elementary usage of mlab - I get totally lost and don't have any idea
>>> where to look next.
>>
>> Yes. There is an interesting challenge here: with mlab and it's
>> documentation, we have more or less managed to steer the users to get the
>> most important parts done easily. The difficulty is to help them make
>> the step required to understand a bit more the inner workings of VTK and
>> Mayavi, so that they can more easily find the answers to their
>> complicated usecases by themselves. This is the goal of the more and more
>> complex examples that we add to the example list, as well as the chapter
>> http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_use.html
>> However, there is still some simple documentation work to be done on
>> these texts, I believe.
>>
>> If you find something fundemental that we could be doing better, I'd love
>> to hear and learn. It's probably a lot of small details that need to be
>> improved. We are working on it, but very slowly: writing docs does take a
>> lot of time, and neither Prabhu nor I are able to free enough time to
>> work on Mayavi (of course, we take patches :>).
>>
>> On the bright side: it seems to me that users stumble on more and more
>> difficult problems, as time goes.
>
> I think the basic problem is that there are two different incompatible
> ways to work with mayavi: using the mlab api to just plot things, and
> constructing vtk pipelines. I find I stumble when I try to go from one
> to the other. I think part of the problem is that there are no (or I
> haven't found) examples of doing simple things with the pipeline
> approach. Would it be reasonable to, for every example in the mlab
> documentation, also offer a version, linked from the same page, that
> offers "how to do this with pipelines"?
>
> The second problem is that the documentation of the vtk objects you
> have to use to work with pipelines is nonexistent, or separate from
> the mlab documentation, or not indexed, or somehow otherwise hard to
> find. How, for example, should I have discovered that I wanted the
> module_manager.scalar_lut_manager attribute of the mesh I created?
> What's a module manager, and when would I use it? I ask these
> questions not so much for me, since I can look this stuff up (somehow)
> now that I know it's needed, as to get them into the documentation
> somewhere.
>
> Thanks,
> Anne

I guess you already know it, but nevertheless: whenever I want to do something
non-standard with mayavi (which is pretty often), there is the last-resort
power-combination of the red record button together with print_traits() method
of every pipeline object. Example: you want to change a colormap and do not
know which attribute of what/where is the one? Just do it in the GUI, record
all the steps, and save the generated Python script -
module_manager.scalar_lut_manager pops up there.

Otherwise, big +1 to your enhancement proposals and equally big thanks to Gael
and Prabhu for their never-ending work towards better(-documented) mayavi.

Best regards,
r.
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Gael Varoquaux
In reply to this post by Anne Archibald
Hi Anne,

On Thu, May 06, 2010 at 02:25:03AM -0400, Anne Archibald wrote:
> I think the basic problem is that there are two different incompatible
> ways to work with mayavi: using the mlab api to just plot things, and
> constructing vtk pipelines. I find I stumble when I try to go from one
> to the other. I think part of the problem is that there are no (or I
> haven't found) examples of doing simple things with the pipeline
> approach. Would it be reasonable to, for every example in the mlab
> documentation, also offer a version, linked from the same page, that
> offers "how to do this with pipelines"?

I think that your remark is interesting. I am not sure I understand what
you call 'VTK pipelines', and that certainly causes difficulties. Let me
go through a bit of background:

* You can use VTK pipelines with Mayavi, and that's a fairly different
  world: these pipelines have a different structure than the Mayavi
  pipeline, and they will not appear in the Mayavi pipeline view. Although
  they underlie everything that is done in Mayavi, I don't believe this is
  what you would want to do.

* Mayavi revolves around Mayavi pipelines, which is what you see in the
  pipeline view. And, as you point out, they are several ways of creating
  them. This might be part of the difficulties. One thing that does not
  seem clear to users, is that these several ways of creating the
  pipeline are completely equivalent, and that it should be fairly easy
  to go from one to another:

     1. mlab helper functions, such as 'mlab.plot3d': these create a
        pipeline for you without you having to worry or even know about
        the pipeline. You discover the pipeline only when you pop up the
        pipeline dialog in Mayavi, that looks alike the screenshot on:
        http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/mlab.html#changing-object-properties-interactively
        (that's an old screenshot, so it's a bit different nowadays).
   
     2. mlab.pipeline function, as detailed in:
        http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/mlab.html#assembling-pipelines-with-mlab
        To go from the step above to using mlab pipeline function should
        be really easy: the names of the nodes in the pipeline dialogs
        tell us what pipeline functions should be used.

     3. These functions are factories that avoid you to have to deal with
        creating pipeline objects and adding them yourself to the
        pipeline. But is is strictly equivalent to create and add the
        objects. For instance, it is shown on:
        http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_scripting.html#a-quick-example
        how to do it.

Now, I can see that the links between these different approaches, that
are fairly equivalent, might not be clear to people at all. I am not sure
that adding a 'how to do it with the pipeline' example for each mlab
example would add that much, because, apart from getting the big picture,
the only thing to know is that once you have an mlab example, you look at
the pipeline dialog, and you know you to reproduce it. There is one
notable exception, that is that the mlab.pipeline source functions do not
have a Mayavi pipeline object counterpart, the reason being that actually
perform some operations on the numpy arrays to create Mayavi pipeline
objects.


> The second problem is that the documentation of the vtk objects you
> have to use to work with pipelines is nonexistent, or separate from
> the mlab documentation, or not indexed, or somehow otherwise hard to
> find. How, for example, should I have discovered that I wanted the
> module_manager.scalar_lut_manager attribute of the mesh I created?
> What's a module manager, and when would I use it? I ask these
> questions not so much for me, since I can look this stuff up (somehow)
> now that I know it's needed, as to get them into the documentation
> somewhere.

I believe that you are mostly talking here about the Mayavi pipeline
object, and not really the underlying VTK objects, although the problems
really blend together, as the VTK objects are sub-objects of the Mayavi
objects and quite often one needs to modify them.

Anyhow, the problem you are mentioning is quite major. I think that they
are two aspects to it. The first challenge is that we are tapping in VTK,
which is a huge library with several hundreds classes. VTK has
automatically-generated documentation:
http://www.vtk.org/doc/nightly/html/annotated.html. In my experience, it
is fairly useless, unless you know exactly what you are looking for. The
second challenge is that, in an effort to adapt the VTK world to a
scripting workflow with numpy arrays, we add our set of objects on top of
that. This second problem can more easily be solved, and it might
actually solve a decent fraction of the usecases. It is still a fair
amount of work...

Action points
==============

My week end in the mountains just got canceled because of bad weather. My
partner will be off sailing. I think you just got me motivated to try and
address some issues in the docs this week end. I know I won't be able to
tackle much of the problems, but if I can spend a full day writing
documentation, I might get something done. Off course this will mean that
Ilan and VTK 5.6 might have to wait :(.

 1. I suggest adding a section to
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_use.html
    before 'Data representation in Mayavi' that explains what I have
    explained in the beginning of this mail on the different objects and ways
    of creating them in Mayavi.

 2. Systematic documentation of the Mayavi pipeline objects needs to be
    written, probably completing
    http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_scripting.html#objects-populating-the-mayavi-pipeline



However, even that done, I am not sure that it would address your
problems that were:

    a. I want to plot a bunch of lines efficiently
    b. I want to control the scalar-to-color relationship

The solution to problem a. was more or less in
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/data.html
but how could you know it was there?

And the solution to problem b. was hinted in the description of the
module manager on page:
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/advanced_scripting.html#objects-populating-the-mayavi-pipeline
although why you would end up on this page given your problem is not
clear.

However, given a user-problem, ie what you want to do, the challenge is:
how do you know where to look? Adding more detailed documentation will
help, because right now it is quite hard to read, but really what I would
like to find is a way to help people looking at the place.

Thanks for your feedback and your patience,

Gaël
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Anne Archibald-2
In reply to this post by Gael Varoquaux
Hi Gael,

What version of mayavi2 are you using for this? I get an error:

Traceback (most recent call last):
  File "gael_demo.py", line 79, in <module>
    lines = mlab.pipeline.stripper(src)
AttributeError: 'module' object has no attribute 'stripper'

using the Ubuntu packages 3.2.0 with VTK 5.2.1.

Thanks,
Anne

On 4 May 2010 04:56, Gael Varoquaux <[hidden email]> wrote:

> On Sun, May 02, 2010 at 04:51:41PM -0400, Anne Archibald wrote:
>> * Adding each line takes longer and longer. I think this is because
>> they are all added to the scene graph, which means the scene graph
>> gets large and ugly (~hundreds of lines). This ugliness is certainly
>> visible when I try running things in the mayavi2 application.
>
>> Is there an object appropriate for handling a collection of lines
>> (tubes)? This would certainly simplify the scene graph, and after all
>> I do want all the lines treated the same - I just don't want to
>> connect the end of one to the beginning of the next. Maybe this is
>> just a very difficult task for mayavi?
>
> In addition to my previous (short) answer, here is a fully-running
> example that shows how to do this.
>
> HTH,
>
> Gaël
>
> _______________________________________________
> Enthought-Dev mailing list
> [hidden email]
> https://mail.enthought.com/mailman/listinfo/enthought-dev
>
>
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: data structure for multiple lines?

Gael Varoquaux
On Fri, May 07, 2010 at 10:47:28AM -0400, Anne Archibald wrote:
> Hi Gael,

> What version of mayavi2 are you using for this?

SVN, off course :).

> I get an error:

> Traceback (most recent call last):
>   File "gael_demo.py", line 79, in <module>
>     lines = mlab.pipeline.stripper(src)
> AttributeError: 'module' object has no attribute 'stripper'

OK, stripper is new in 3.3.0. But you can simply remove it: it won't
change much, and it is useful only if you add tubes afterwards. If you
plot with lines, you won't see the difference.

HTH,

Gaël
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Mayavi documentation (Was: data structure for multiple lines?)

Gael Varoquaux
In reply to this post by Gael Varoquaux
On Thu, May 06, 2010 at 10:52:26AM +0200, Gael Varoquaux wrote:
> My week end in the mountains just got canceled because of bad weather. My
> partner will be off sailing. I think you just got me motivated to try and
> address some issues in the docs this week end. I know I won't be able to
> tackle much of the problems, but if I can spend a full day writing
> documentation, I might get something done.

So, my laptop died on Friday, but I had time to grab a new old one from
work and I spent the Saturday installing: Mayavi documentation lost one
day.

Sunday was fairly productive on the other side. I have implemented a few
of the suggested changes. Updated docs are up on the website, thanks to
Peter:
http://code.enthought.com/projects/mayavi/docs/development/html/mayavi/

Of course what lacks most, right now, is polish in the docs (and no, I am
not looking for a Polish translation :P). Things like inserting links
such as 'see also' are very important to help people find what they are
looking for. I guess that this will come with time.

Feedback welcome,

Gaël
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev