HDF5 continuous read/display

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

HDF5 continuous read/display

Eraldo Pomponi
Hi All, 

I wrote a simple HDF5 file inspector and display app that is, for the display part, inspired to the Corran example 
( example picture in attach) 

I'm loading my data through: 

np.array(self.table, self.table.dtype)

where self.table is a PyTables table Object instance and self.table.dtype the corresponding structure definition.

Everything works pretty well till the size of the table is reasonably small. Of course, when this is not true then 
comes the pain (with 1 GB table I got the system to hangs up). 

Reading just part of the table is quite straightforward but how to synchronize it with the GUI 
(in my case a tabular editor) not ... I wasn't able to figure it out. 

Would someone please point me in the right direction?  
Thanks ... 

Cheers, 
Eraldo 



 

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev

IAES_HDF5_File_Browser.png (189K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 continuous read/display

Robert Kern
On Mon, Jun 18, 2012 at 9:58 AM, Eraldo Pomponi
<[hidden email]> wrote:

> Hi All,
>
> I wrote a simple HDF5 file inspector and display app that is, for the
> display part, inspired to the Corran example
> https://mail.enthought.com/pipermail/enthought-dev/2011-December/030318.html.
> ( example picture in attach)
>
> I'm loading my data through:
>
> np.array(self.table, self.table.dtype)
>
> where self.table is a PyTables table Object instance and self.table.dtype
> the corresponding structure definition.
>
> Everything works pretty well till the size of the table
> is reasonably small. Of course, when this is not true then
> comes the pain (with 1 GB table I got the system to hangs up).
>
> Reading just part of the table is quite straightforward but how
> to synchronize it with the GUI
> (in my case a tabular editor) not ... I wasn't able to figure it out.
>
> Would someone please point me in the right direction?

You can access PyTables Table objects pretty much the same way you
would access the structured array. The TabularAdapter you wrote for
the ndarray probably works correctly for the Table object. PyTables
will load the data on-demand.

--
Robert Kern
Enthought
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 continuous read/display

Eraldo Pomponi
Dear Robert, 

Thanks for your fast reply but I'm not so sure that I understood your point


You can access PyTables Table objects pretty much the same way you
would access the structured array. The TabularAdapter you wrote for
the ndarray probably works correctly for the Table object. PyTables
will load the data on-demand.

####### This is the way I access the data #########################

table_ptr = Any  # PyTables table instance "pointer"
table = Property(Array(dtype=param_dtype), depends_on='table_ptr')

    def _get_table(self):
        if self._table is None or len(self._table) == 0:
            if self.table_ptr is not None:
                self._table = np.array(self.table_ptr, self.table_ptr.dtype)
            else:
                self._table = np.array([])
        return self._table

###########################################################

In which way can I sync it with the view? I need an event to look at an refresh the 
data properly but I don't know which is the proper one. 

Cheers,
Eraldo 


_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 continuous read/display

Robert Kern
On Mon, Jun 18, 2012 at 10:31 AM, Eraldo Pomponi
<[hidden email]> wrote:

> Dear Robert,
>
> Thanks for your fast reply but I'm not so sure that I understood your point
>
>>
>> You can access PyTables Table objects pretty much the same way you
>> would access the structured array. The TabularAdapter you wrote for
>> the ndarray probably works correctly for the Table object. PyTables
>> will load the data on-demand.
>>
> ####### This is the way I access the data #########################
>
> table_ptr = Any  # PyTables table instance "pointer"
> table = Property(Array(dtype=param_dtype), depends_on='table_ptr')
>
>     def _get_table(self):
>         if self._table is None or len(self._table) == 0:
>             if self.table_ptr is not None:
>                 self._table = np.array(self.table_ptr, self.table_ptr.dtype)
>             else:
>                 self._table = np.array([])
>         return self._table
>
> ###########################################################
>
> In which way can I sync it with the view? I need an event to look at an
> refresh the
> data properly but I don't know which is the proper one.

Don't convert the Table to an array. Just use the TabularEditor on
table_ptr directly.

--
Robert Kern
Enthought
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: HDF5 continuous read/display

Eraldo Pomponi
Dear Robert,

On Mon, Jun 18, 2012 at 11:43 AM, Robert Kern <[hidden email]> wrote:
On Mon, Jun 18, 2012 at 10:31 AM, Eraldo Pomponi
<[hidden email]> wrote:
> Dear Robert,
>
> Thanks for your fast reply but I'm not so sure that I understood your point
>
>>
>> You can access PyTables Table objects pretty much the same way you
>> would access the structured array. The TabularAdapter you wrote for
>> the ndarray probably works correctly for the Table object. PyTables
>> will load the data on-demand.
>>
> ####### This is the way I access the data #########################
>
> table_ptr = Any  # PyTables table instance "pointer"
> table = Property(Array(dtype=param_dtype), depends_on='table_ptr')
>
>     def _get_table(self):
>         if self._table is None or len(self._table) == 0:
>             if self.table_ptr is not None:
>                 self._table = np.array(self.table_ptr, self.table_ptr.dtype)
>             else:
>                 self._table = np.array([])
>         return self._table
>
> ###########################################################
>
> In which way can I sync it with the view? I need an event to look at an
> refresh the
> data properly but I don't know which is the proper one.

Don't convert the Table to an array. Just use the TabularEditor on
table_ptr directly.

Thanks a ton ... It works like a charm and the memory usage is negligible !!!! 

Cheers, 
Eraldo 
 

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev