Using traits with an ORM and web publishing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using traits with an ORM and web publishing

Stephen Waterbury
Note:  this message does *not* contain specific questions but
rather describes a couple of areas in which I'm working and
requests comments/feedback from the developers of traits.

First let me say as a traits newbie (larva, actually), from what
I've seen so far the traits developers have done a really
incredible job -- thanks for all your work!

I'm interested in perspectives on the following areas into which
I'm taking traits:

*ORM integration*

I haven't yet found direct support or recommendations for the use
of traits with an ORM but some the features described in
the "Persistence" section of "Advanced Topics" seem to be
designed to be applicable to various forms of persistence and
serialization.  I am experimenting with integrating traits with
storm (https://storm.canonical.com/), my favorite python orm.

Storm doesn't require an app's classes to subclass anything --
just give them a '__storm_table__' class attribute and define the
other attributes using storm attribute classes (in a similar way
to the traits in a Has_Traits object) and you're good to go.

My first naive cut is simply to extend trait types by subclassing
from both storm and trait datatypes.  This seems to be working so
far -- I'm still refactoring my code and co-evolving unit tests.
Interesting that traits uses a 'transient' attribute -- I had
originally taken the complementary approach of specifying what
would persist, the default being not to.  I've been using zope
schemas to specify the set of persistent attributes; not sure if
I'll try to use that to implement special __[get/set]_state__ or
not.

*web publishing*

My first target is some form of jsonrpc interface generation.
The app I'm working on as my test case has a relatively simple
set of object-creation, document-generation, and reporting
requirements.  I'm using pyjamas to create a
javascript/jsonrpc-ajax (or "ajaj", I guess ;) app and planning
to use traits, storm, and txjsonrpc on the back-end.  Aiming
initially for a relatively simple traits/schema-driven CRUD
interface for the app objects.

If anything related is being worked on in some form in the traits
development world, I'd love to hear about it!  Other than that,
*any* comments are welcome, really ...

Cheers,
Steve

P.S.  I'm aware that someone has made an initial attempt at a
traits-driven pyjamas desktop gui, but if I want a desktop gui
app I'll be happy to use the existing traits gui backends
(wx or qt), which look fine to me.

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using traits with an ORM and web publishing

Robert Kern
On Thu, May 5, 2011 at 4:10 PM, Stephen Waterbury
<[hidden email]> wrote:

> Note:  this message does *not* contain specific questions but
> rather describes a couple of areas in which I'm working and
> requests comments/feedback from the developers of traits.
>
> First let me say as a traits newbie (larva, actually), from what
> I've seen so far the traits developers have done a really
> incredible job -- thanks for all your work!
>
> I'm interested in perspectives on the following areas into which
> I'm taking traits:
>
> *ORM integration*
>
> I haven't yet found direct support or recommendations for the use
> of traits with an ORM but some the features described in
> the "Persistence" section of "Advanced Topics" seem to be
> designed to be applicable to various forms of persistence and
> serialization.  I am experimenting with integrating traits with
> storm (https://storm.canonical.com/), my favorite python orm.
>
> Storm doesn't require an app's classes to subclass anything --
> just give them a '__storm_table__' class attribute and define the
> other attributes using storm attribute classes (in a similar way
> to the traits in a Has_Traits object) and you're good to go.

You can take a look at how we made Traits work with SQLAlchemy.
SQLAlchemy works roughly similarly, except that it attaches the right
metadata and descriptors (the cognate of the storm's attribute
classes, if I am guessing correctly) instead of making you do it
explicitly on the class.

  https://svn.enthought.com/svn/sandbox/traited_orm/

We defined ORMapped, a subclass of HasTraits and base class for our
mapped model classes. It defines a Trait notification method for all
traits with the db_storage metadata. That method pumps the new value
through the SQLAlchemy descriptor machinery. Lists need special
handling because of how they get mapped to relations in the ORM. We
provide simple subclasses of the most interesting TraitTypes just to
add the db_storage metadata for easy reading later. We hook into
SQLAlchemy's Mappers via their extension mechanism to make sure all of
our objects get set up correctly.

A lot of that detailed implementation is SQLAlchemy-specific, and you
will have to figure out what concepts map to storm.

> My first naive cut is simply to extend trait types by subclassing
> from both storm and trait datatypes.  This seems to be working so
> far

I'm pleasantly surprised that this would work, but I'm not really
familiar with storm's implementation. Since the TraitTypes get
consumed by the metaclass, how does storm find them again?

> -- I'm still refactoring my code and co-evolving unit tests.
> Interesting that traits uses a 'transient' attribute -- I had
> originally taken the complementary approach of specifying what
> would persist, the default being not to.  I've been using zope
> schemas to specify the set of persistent attributes; not sure if
> I'll try to use that to implement special __[get/set]_state__ or
> not.

I'm not sure I would rely on transient to do what you want. It has a
specific meaning already: transient traits are not copied in
.clone_traits(), .copy_traits(), copy/deepcopy(), or pickling. That's
not quite the same thing as "store this in my database". Since you're
subclassing the TraitTypes anyways, you should rely on that, not other
trait metadata.

> *web publishing*
>
> My first target is some form of jsonrpc interface generation.
> The app I'm working on as my test case has a relatively simple
> set of object-creation, document-generation, and reporting
> requirements.  I'm using pyjamas to create a
> javascript/jsonrpc-ajax (or "ajaj", I guess ;) app and planning
> to use traits, storm, and txjsonrpc on the back-end.  Aiming
> initially for a relatively simple traits/schema-driven CRUD
> interface for the app objects.

I've never quite been sure about how I feel about sticking Traits UIs
inside of a web app. Database-backed web apps (event the AJAXy ones)
usually work with the following model:

1. Browser makes a request.
2. Server pulls the relevant objects from the database.
3. Server makes the requested changes to the objects.
4. Server stuffs the objects back into the database.
5. Server returns the appropriate response to the browser.
6. Repeat.

To some extent, anything persisting in memory between steps 5 and 6
must simply be an optimization and not a semantic difference. I've had
to 'apachectl restart graceful' enough times that I'm really glad that
that model is how web apps are written. The Traits UI model looks like
this:

1. Lots of events fly around between components and the widgets as the
user clicks on stuff.

I'm kidding a little. But only a little.

It seems a little overkill for CRUD interfaces. Honestly, Django's
admin interface seems to work really, really well for slapping a CRUD
interface onto a simple database. There are a few other existing
solutions for other ORM/template/framework combinations.

But that's just my opinion for a strawman app that I've built in my
head. I might be able to give better advice with some more
information. What are you trying to get out of Traits here?

--
Robert Kern
Enthought
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using traits with an ORM and web publishing

Stephen Waterbury-2
On 05/05/2011 06:57 PM, Robert Kern wrote:

> On Thu, May 5, 2011 at 4:10 PM, Stephen Waterbury
> <[hidden email]>  wrote:
>>
>> *ORM integration*
>>
>> I haven't yet found direct support or recommendations for the use
>> of traits with an ORM but some the features described in
>> the "Persistence" section of "Advanced Topics" seem to be
>> designed to be applicable to various forms of persistence and
>> serialization.  I am experimenting with integrating traits with
>> storm (https://storm.canonical.com/), my favorite python orm.
>>
>> Storm doesn't require an app's classes to subclass anything --
>> just give them a '__storm_table__' class attribute and define the
>> other attributes using storm attribute classes (in a similar way
>> to the traits in a Has_Traits object) and you're good to go.
>
> You can take a look at how we made Traits work with SQLAlchemy.
> SQLAlchemy works roughly similarly, except that it attaches the right
> metadata and descriptors (the cognate of the storm's attribute
> classes, if I am guessing correctly) instead of making you do it
> explicitly on the class.
>
>    https://svn.enthought.com/svn/sandbox/traited_orm/
>
> We defined ORMapped, a subclass of HasTraits and base class for our
> mapped model classes. It defines a Trait notification method for all
> traits with the db_storage metadata. That method pumps the new value
> through the SQLAlchemy descriptor machinery. Lists need special
> handling because of how they get mapped to relations in the ORM. We
> provide simple subclasses of the most interesting TraitTypes just to
> add the db_storage metadata for easy reading later. We hook into
> SQLAlchemy's Mappers via their extension mechanism to make sure all of
> our objects get set up correctly.
>
> A lot of that detailed implementation is SQLAlchemy-specific, and you
> will have to figure out what concepts map to storm.

Thanks for the quick reply and useful comments -- I will definitely
look at this and see how much is applicable to storm integration.

>
>> *web publishing*
>>
>> My first target is some form of jsonrpc interface generation.
>> The app I'm working on as my test case has a relatively simple
>> set of object-creation, document-generation, and reporting
>> requirements.  I'm using pyjamas to create a
>> javascript/jsonrpc-ajax (or "ajaj", I guess ;) app and planning
>> to use traits, storm, and txjsonrpc on the back-end.  Aiming
>> initially for a relatively simple traits/schema-driven CRUD
>> interface for the app objects.
>
> I've never quite been sure about how I feel about sticking Traits UIs
> inside of a web app. Database-backed web apps (event the AJAXy ones)
> usually work with the following model:
>
> 1. Browser makes a request.
> 2. Server pulls the relevant objects from the database.
> 3. Server makes the requested changes to the objects.
> 4. Server stuffs the objects back into the database.
> 5. Server returns the appropriate response to the browser.
> 6. Repeat.
>
> To some extent, anything persisting in memory between steps 5 and 6
> must simply be an optimization and not a semantic difference.

Depends on how the service is implemented.  With a django
app, it *would* be an optimization (caching I guess), since
django wasn't designed to be event-driven -- it's really
the old-fashioned get what you need to create this page,
as opposed to the objects with events paradigm.

In a pyjamas (or any javascript/jsonrpc) app, the browser acts
more like a gui client and less like a page display -- you can
use jsonrpc to get just the data you need from the server and use
it to directly update just one or more widgets in the interface
without refreshing the page, making it much more like a
traditional gui app.

> I've had
> to 'apachectl restart graceful' enough times that I'm really glad that
> that model is how web apps are written.

That would definitely refer to django apps -- I know, I do those too.

> The Traits UI model looks like
> this:
>
> 1. Lots of events fly around between components and the widgets as the
> user clicks on stuff.
>
> I'm kidding a little. But only a little.

Understood.  Actually, network apps can work that way too.
Not traditional, page-oriented web apps like django, but
ajax apps *can* be event-driven (yes, django can do some ajax
stuff too, but there are various ways to do ajax, some of which
are more page-oriented and some are more like the events flying
around mode).

> It seems a little overkill for CRUD interfaces.

Agreed -- but only a little.  A CRUD interface can feel nicer to
the end user if the page is not refreshing all the time. ;)
Besides, the CRUD stuff is only what the current app I'm working
on needs; my real target is events flying around.  I'll need that
for the engineering environment I'm developing, which is intended
to provide concurrent communications among computer-aided design
and analysis and other engineering tools using a message bus
architecture.

> Honestly, Django's
> admin interface seems to work really, really well for slapping a CRUD
> interface onto a simple database. There are a few other existing
> solutions for other ORM/template/framework combinations.

Yes, there are, and I agree django works quite well -- I've used
the admin interface to good advantage.  Unfortunately, django and
the other existing ORM/template/framework solutions (with the
possible exception of zope 3, but zope is a bit heavy) make it
difficult to drive multiple app aspects from one meta-hub.  For
example, in django, as in traits, and storm for that matter, you
define your classes using their built-in type systems, each of
which is slightly different.

If you want to mix and match, as I do, or combine them in ways
unforeseen by the authors, you're stuck with creating classes in
each, thus violating DRY.  What I'm trying to do is drive them
all from my own meta-hub, as it were ... since none were
engineered in such a way as to accept metadata from an external
source, I'll just create their classes on the fly.  That's okay
but I'd like to avoid having *too* many databases that I need to
keep in sync with events flying around ... that gets messy.  So
I'm making some effort to keep a coherent persistence api,
although it will require some work as I can see I'll probably
need several "physical" backends -- e.g., relational db,
rdf-style store with sparql, maybe hdf5 ...

> But that's just my opinion for a strawman app that I've built in my
> head. I might be able to give better advice with some more
> information. What are you trying to get out of Traits here?

Actually I want to use traits for what it has been designed for:
gui apps with events flying around.  The other stuff (web
publishing, network events, etc.) I'll use other frameworks for,
although I might try later on to make traits event notifications
interact asynchronously with a message bus -- there's that
example that used twisted, orbited, and stomp, although I don't
think I'd do it exactly that way (twisted yes, other ingredients
TBD).

Seems like there's not much interest in the core traits world in
network stuff, so that answers that question -- I'll just do what
I have to do and not worry that I'm duplicating anything that the
traits developers might be already working on.

Thanks for all the input!
Steve
_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using traits with an ORM and web publishing

Stephen Waterbury
On 05/06/2011 02:11 AM, Stephen Waterbury wrote:
> On 05/05/2011 06:57 PM, Robert Kern wrote:

>> I've never quite been sure about how I feel about sticking Traits UIs
>> inside of a web app. ...

Apologies for replying to my own reply, but having slept on it, I
woke up with sort of an epiphany:

It now seems clear to me that using full-up traited objects on
the server side doesn't make sense, since they are designed for
ui generation and event propagation.  That said, I'm still
thinking in terms of creating them from metadata on client nodes
and setting up communication with them to/from a network message
bus (on which a "meta-hub" will be a service).  Obviously there
will have to be some bootstrapping stuff in the traits-enabled
client nodes, but I've been thinking in those terms for a while
now anyway.

Since the traits property/datatype-system is really nice and
complete, the meta-hub will probably use it or something very
similar and map other datatype systems into it (e.g.  django,
storm, zope.schema, etc.).

Again, thanks for your reply and very useful perspectives,
Robert.

Cheers,
Steve

_______________________________________________
Enthought-Dev mailing list
[hidden email]
https://mail.enthought.com/mailman/listinfo/enthought-dev