| 00:00.32 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 00:02.38 | *** join/#storm aa_ (n=ali@pida/aa) |
| 00:02.42 | aa_ | hi everyone |
| 00:03.13 | aa_ | so, I want to plug storm into a web framework, and am wondering about strategy |
| 00:03.37 | aa_ | the framework makes pretty heavy use of a threadlocal-ish thing |
| 00:03.53 | aa_ | now I am thinking that perhaps I should have one Store per request |
| 00:04.36 | aa_ | this would be the easiest to implement I guess |
| 00:05.07 | aa_ | just open the store at the start of the request, and then clean it up later |
| 00:05.20 | aa_ | can anyone see a problem with this approach, or are there perhaps better approaches? |
| 00:09.55 | *** join/#storm ubot3 (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 00:28.55 | niemeyer | aa_: Yeah, that's usually how it goes |
| 00:29.19 | aa_ | niemeyer: hi, and that won't be unduly expensive? |
| 00:29.20 | niemeyer | aa_: Have you had a look at storm/zope/? |
| 00:29.30 | aa_ | niemeyer: yeah, I did try to look at it |
| 00:29.55 | niemeyer | aa_: The strategy in zstorm (the zope hook up) is to have per-thread named stores |
| 00:30.01 | aa_ | am looking now, not sure what a transaction manager is though |
| 00:30.12 | niemeyer | aa_: That's not expensive at all, because stores are reused over requests |
| 00:30.31 | aa_ | so it has like a Store pool? |
| 00:30.43 | aa_ | or there is some magic when constructing stores? |
| 00:31.07 | niemeyer | aa_: The zope transaction manager allows multiple transactioned storage mechanisms to hook into it, so that they can all be committed at the same time |
| 00:31.19 | niemeyer | aa_: Including two-phase logic, when that's supported, and so on |
| 00:31.32 | niemeyer | aa_: Not really a pool, but yeah, something similar |
| 00:31.52 | niemeyer | aa_: A pool usually is associated with something you dispose the things you're not using, and reuse them |
| 00:32.12 | niemeyer | aa_: In this case it's basically just keeping the stores around so that when you ask for the given named store it'll be ready |
| 00:33.03 | aa_ | and how would the threads know which named store to ask for? |
| 00:33.09 | aa_ | s/threads/requests/ |
| 00:33.20 | aa_ | heh, nice |
| 00:34.20 | thumper | niemeyer: hi |
| 00:34.23 | niemeyer | aa_: Not sure I understand the question exactly |
| 00:34.26 | thumper | niemeyer: just the person I'm looking for |
| 00:34.33 | niemeyer | therve: Hey! |
| 00:34.39 | thumper | niemeyer: :-) close |
| 00:34.48 | niemeyer | thumper: Hey!! :-) |
| 00:35.03 | thumper | I have a query that I'm having trough formulating |
| 00:35.21 | thumper | We have some tables: Branch, BranchRevision and Revision |
| 00:35.34 | niemeyer | aa_: The store name depends on what you're storing on it |
| 00:35.37 | thumper | BranchRevision is the linking table (kinda obvious) |
| 00:36.02 | niemeyer | aa_: If you have a database storing session information, you might name your store "session" |
| 00:36.10 | niemeyer | therve: Ok |
| 00:36.11 | thumper | I want to select the Branch that has the most Revisions (for some specific RevisionAuthor) |
| 00:36.12 | niemeyer | therve: Sorry |
| 00:36.15 | niemeyer | thumper: Ok :) |
| 00:36.59 | thumper | niemeyer: can I use the builder syntax or do I need SQL |
| 00:36.59 | niemeyer | thumper: The most revisions, hmm.. interesting |
| 00:36.59 | aa_ | sorry my connection keeps timing out, I am keeping up |
| 00:37.15 | aa_ | niemeyer: I get it after looking at the code, _stores is thread local |
| 00:37.34 | niemeyer | thumper: You can most probably use the builder syntax |
| 00:37.43 | niemeyer | thumper: How does the query look like in SQL/ |
| 00:37.44 | niemeyer | ? |
| 00:37.46 | aa_ | so every request can ask for 'session' or whatever |
| 00:37.47 | thumper | niemeyer: I was hoping you'd say that |
| 00:37.55 | niemeyer | aa_: That's it |
| 00:37.59 | thumper | goes to write it in a scratch buffre |
| 00:38.02 | niemeyer | thumper: Aha, ok :-) |
| 00:38.18 | niemeyer | thumper: Yeah, there are some groupings and orderings there :) |
| 00:40.05 | niemeyer | thumper: It should be simple actually. It'd involve only the middle table.. with "GROUP BY branch_revision.branch_id" and COUNT |
| 00:40.20 | niemeyer | (and order) |
| 00:40.44 | thumper | right |
| 00:41.34 | thumper | http://paste.ubuntu.com/34234/ |
| 00:41.44 | thumper | niemeyer: something like that |
| 00:41.55 | thumper | niemeyer: is the group stuff in trunk? |
| 00:42.07 | thumper | niemeyer: or at least in the LP storm branch? |
| 00:42.25 | niemeyer | thumper: It is in both |
| 00:42.47 | thumper | cool |
| 00:43.07 | thumper | is the group syntax in the tutorial on the web? |
| 00:43.25 | thumper | couldn't find it |
| 00:43.36 | niemeyer | I'm not sure |
| 00:43.48 | niemeyer | thumper: But it's basically store.find(...).group_by(...) |
| 00:43.57 | thumper | niemeyer: also, once I'm finished with this, I'm back onto the merge queues |
| 00:44.31 | thumper | niemeyer: so something like .group_by(Count(Revision.id)) ?? |
| 00:44.53 | niemeyer | therve: Yeah, except not with count |
| 00:44.54 | thumper | also .order_by(Desc(Count(Revision.id))) ? |
| 00:45.00 | thumper | ah |
| 00:45.04 | thumper | right, no count in group by |
| 00:45.06 | niemeyer | thumper: Let me see if I get you the query, hold on |
| 00:45.08 | thumper | but needed in order by? |
| 00:46.27 | niemeyer | thumper: SELECT branch_id, COUNT(revision_id) AS n FROM branch_revision GROUP BY branch_id ORDER BY n DESC; |
| 00:46.54 | thumper | niemeyer: but how to write that in builder? |
| 00:48.18 | thumper | niemeyer: http://paste.ubuntu.com/34238/ perhaps? |
| 00:48.31 | thumper | I'm not sure about the Desc(Count(...)) thing |
| 00:48.33 | niemeyer | store.find((Branch, Alias(Count(BranchRevision.revision_id), "n")), Branch.branch_id == BranchRevision.branch_id).group_by(BranchRevision.branch_id).order_by(Desc("n")) |
| 00:48.44 | thumper | ah |
| 00:48.47 | niemeyer | thumper: I don't know if that's right |
| 00:48.52 | niemeyer | thumper: But it's close :) |
| 00:48.58 | niemeyer | The SQL query definitely is |
| 00:49.20 | niemeyer | thumper: So in the worse case, just use a subquery if you can't find the proper builder syntax |
| 00:49.23 | thumper | I did it without the alias or the count in the select |
| 00:49.24 | niemeyer | (with raw sql) |
| 00:49.45 | thumper | I don't actually care what the count is |
| 00:50.06 | niemeyer | thumper: Hmm |
| 00:50.31 | thumper | I suppose I could use the storm tracer to check the query :) |
| 00:51.04 | niemeyer | thumper: Yeah, I guess that's simpler! |
| 00:51.16 | niemeyer | thumper: So just COUNT(revision_id) in the order_by |
| 00:51.28 | thumper | niemeyer: Desc too |
| 00:51.33 | thumper | :) |
| 00:51.36 | niemeyer | Oh, right |
| 00:51.54 | niemeyer | Anyway |
| 00:51.58 | niemeyer | My eyes are bleeding :-) |
| 00:52.00 | niemeyer | I have to step out |
| 00:52.14 | niemeyer | Have a good time folks |
| 00:52.22 | thumper | niemeyer: night |
| 00:52.30 | niemeyer | Night! |
| 01:17.24 | thumper | :( |
| 01:17.32 | thumper | damn those group by statements |
| 03:28.12 | *** join/#storm jamesh (n=james@canonical/launchpad/jamesh) |
| 04:59.37 | *** join/#storm kov (i=kov@debian/developer/kov) |
| 06:17.47 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 07:23.01 | *** join/#storm goschtl (n=goschtl@p5B0BD184.dip.t-dialin.net) |
| 07:58.53 | *** join/#storm ubot3` (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 08:07.58 | *** join/#storm mcella (n=mcella@ip-178-152.sn2.eutelia.it) |
| 08:23.10 | *** join/#storm ubot3` (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 08:27.17 | *** join/#storm ubot3_ (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 08:34.19 | *** join/#storm lightyear (n=ben@nat/fluendo/x-d1af7fc5bad4c5cd) |
| 09:42.19 | goschtl | how can i create a expression for the *args in find()? |
| 09:47.10 | therve | goschtl: please develop? |
| 09:48.10 | goschtl | it´s not yet implemented? |
| 09:48.56 | therve | goschtl: no, please develop your problem :) |
| 09:49.03 | therve | I don't understand what you mean |
| 09:50.46 | goschtl | therve: ok i try the find method of a store has an args parameter. How looks a expression for args? |
| 09:51.14 | goschtl | is it store.expr.Select(obj, obj.name="klaus") |
| 09:51.38 | therve | ah ok |
| 09:51.54 | therve | goschtl: https://storm.canonical.com/Tutorial |
| 09:52.19 | goschtl | therve: ok thanks i will look again :) |
| 09:52.28 | therve | in particular, https://storm.canonical.com/Tutorial#Finding an object |
| 09:53.23 | therve | looking for "store.find" in this page should give you most syntax |
| 10:50.54 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 11:37.13 | aa_ | yay, so I added storm support to a web framework last night, and here are the documentations: http://glashammer.welterde.de/bundle_stormdb.html perhaps I could link that to the wiki? |
| 11:37.37 | aa_ | weyou need storm on the server |
| 11:37.42 | aa_ | oops wrong window |
| 12:14.22 | *** join/#storm niemeyer (n=niemeyer@200-103-247-127.ctame705.dsl.brasiltelecom.net.br) |
| 12:16.11 | *** join/#storm mup (n=mup@200-103-247-127.ctame705.dsl.brasiltelecom.net.br) |
| 13:14.26 | *** join/#storm ubot3 (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 13:16.28 | *** join/#storm jamesh (n=james@canonical/launchpad/jamesh) |
| 13:41.09 | tlynn | aa_: glashammer looks interesting. I think you have a few documentation typos: |
| 13:41.47 | tlynn | http://glashammer.welterde.de/events.html - app.connect_event('content-added', *build_index*) |
| 13:42.22 | tlynn | http://glashammer.welterde.de/endpoints.html - "Add a an" |
| 13:43.02 | tlynn | I think you should use dot separators for endpoints rather than / too. |
| 13:45.56 | tlynn | http://glashammer.welterde.de/events.html - "predetefined" -> "predetermined" |
| 13:46.29 | tlynn | http://glashammer.welterde.de/events.html - "resonse-end" -> "response-end" |
| 13:51.53 | aa_ | tlynn: thanks very much |
| 13:53.11 | aa_ | tlynn: also nothing to stop you using .'s as endpoint separators |
| 13:56.35 | tlynn | are you working on glashammer with support from the werkzeug people? |
| 13:57.22 | tlynn | aa_: I think '.' is a better separator because it distinguishes when you're talking about objects and when URLs. |
| 13:57.55 | tlynn | it also makes clear that the lookup will be done via getattr |
| 13:58.02 | tlynn | (presumably) |
| 14:08.43 | *** join/#storm ubot3` (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 14:12.19 | aa_ | tlynn: which it isn't :) |
| 14:12.30 | tlynn | how is it done then? |
| 14:12.31 | aa_ | it's an explicit dict lookup |
| 14:12.47 | aa_ | actually, for that part of it it is a getattr |
| 14:12.59 | aa_ | you can either define explicit views or sets of views |
| 14:14.11 | tlynn | where a set of views may be a class, instance or dict? |
| 14:14.34 | aa_ | instance |
| 14:14.39 | aa_ | using getattr as you say |
| 14:14.56 | aa_ | but I only added that feature to placate people, it reminds me too much of pylons |
| 14:15.18 | aa_ | tlynn: well, does it make much difference to you if it's / or . ? |
| 14:16.09 | tlynn | aa_: just that it's not immediately clear that they aren't URL fragments, it's a minor niggle :-) |
| 14:17.21 | aa_ | tlynn: yes, I see the point, it is a good one |
| 14:18.12 | tlynn | but in general it looks like the kind of framework I'd want to use, so I may well try it out at some point |
| 14:18.22 | tlynn | what stage is the update_database functionality at? |
| 14:19.03 | aa_ | tlynn: it's not |
| 14:19.23 | aa_ | tlynn: it doesn't really do much database stuff for you at the moment |
| 14:20.43 | aa_ | tlynn: do you mean schema migrations and things? |
| 14:21.09 | tlynn | yep. that and the integration of WTForms and Storm/SQLAlchemy are where you should spend time then (even if just documenting how you'd do it). |
| 14:21.26 | tlynn | anyway, perhaps we should take further discussion to #glashammer :-) |
| 14:30.19 | *** join/#storm kov (i=kov@debian/developer/kov) |
| 15:42.21 | *** join/#storm salgado (n=salgado@canonical/launchpad/salgado) |
| 15:42.54 | salgado | hi there |
| 15:44.51 | salgado | I'm working on improving the performance of the Launchpad webservice and I need to solve a tricky issue. let's see if you guys can help me out |
| 15:46.21 | therve | sure thing |
| 15:46.30 | salgado | I have a method which gives me a bunch of people (in a ResultSet) and then a template which iterates over these people and, for each of them, looks up stuff from other tables |
| 15:46.47 | salgado | (e.g email addresses) |
| 15:47.04 | therve | ah I think I already see your problem :) |
| 15:47.14 | salgado | that's a problem because we have to hit the DB once for every person in the set |
| 15:49.06 | therve | I think the only solution I know is too change your method to return (People, Email) |
| 15:49.17 | therve | maybe niemeyer has another idea |
| 15:50.59 | salgado | therve, right, but then if I do person.email it won't use the already fetched email because .email is a property which looks up an email address which points to the person |
| 16:06.42 | *** join/#storm lightyear (n=ben@nat/fluendo/x-d1af7fc5bad4c5cd) [NETSPLIT VICTIM] |
| 16:07.34 | *** join/#storm ubot3` (n=ubot3@unaffiliated/nalioth/bot/ubot3) [NETSPLIT VICTIM] |
| 16:07.45 | *** join/#storm mup (n=mup@200-103-247-127.ctame705.dsl.brasiltelecom.net.br) [NETSPLIT VICTIM] |
| 16:07.45 | *** join/#storm niemeyer (n=niemeyer@200-103-247-127.ctame705.dsl.brasiltelecom.net.br) [NETSPLIT VICTIM] |
| 16:07.45 | *** join/#storm radix (n=r@wordeology.com) [NETSPLIT VICTIM] |
| 16:07.45 | *** join/#storm jml (n=jml@mumak.net) [NETSPLIT VICTIM] |
| 16:07.45 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) [NETSPLIT VICTIM] |
| 16:07.46 | *** join/#storm jamesh (n=james@canonical/launchpad/jamesh) [NETSPLIT VICTIM] |
| 16:08.43 | *** join/#storm salgado (n=salgado@canonical/launchpad/salgado) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm mcella (n=mcella@ip-178-152.sn2.eutelia.it) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm jrydberg_ (n=johan@212.112.160.94) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm bac (n=bac@canonical/launchpad/bac) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm djinni (n=djinni@ludios.net) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm thumper (n=tim@canonical/launchpad/thumper) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm oohlaf (n=olaf@unaffiliated/oohlaf) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm therve (n=th@88.191.37.10) [NETSPLIT VICTIM] |
| 16:08.44 | *** join/#storm rockstar (n=rockstar@71-218-148-91.hlrn.qwest.net) [NETSPLIT VICTIM] |
| 16:09.51 | *** join/#storm tlynn (n=tlynn@host81-149-244-57.in-addr.btopenworld.com) [NETSPLIT VICTIM] |
| 16:09.51 | *** join/#storm kov (i=kov@debian/developer/kov) [NETSPLIT VICTIM] |
| 16:09.51 | *** join/#storm aa_ (n=ali@pida/aa) [NETSPLIT VICTIM] |
| 16:09.51 | *** join/#storm ivan (n=ivan@unaffiliated/ivan/x-000001) [NETSPLIT VICTIM] |
| 16:11.17 | *** join/#storm Treenaks (n=martijn@facecrime.net) |
| 16:11.21 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 16:59.32 | niemeyer | salgado-lunch: If the email is a reference, the query won't be done if you do what therve suggested |
| 16:59.52 | niemeyer | salgado-lunch: Because store.get(SomeClass, the_key) doesn't execute a query when the object is already in memory |
| 16:59.58 | niemeyer | (and that's what's used internally) |
| 17:02.21 | therve | ah right |
| 17:02.36 | therve | I forgot about that |
| 17:16.39 | salgado | niemeyer, right, but in this case the email holds the reference to the person |
| 17:24.06 | therve | I'm not sure I understand |
| 17:24.15 | therve | you're fetching person objects right? |
| 17:30.47 | salgado | right |
| 17:31.17 | therve | so, in your person class, you have something like 'email = Reference(blah..)', no? |
| 17:31.24 | salgado | and email addresses have a reference to person |
| 17:31.34 | salgado | instead of the person having a reference to the email address |
| 17:31.38 | salgado | that's the problem |
| 17:31.51 | therve | okayyy |
| 17:32.00 | therve | I don't think we're talking about the same thing |
| 17:32.31 | therve | when we say 'reference', we're talking about storm.references.Reference object |
| 17:33.14 | salgado | in our case there's a "person = Reference(...)" in the EmailAddress |
| 17:33.30 | therve | ah |
| 17:33.35 | therve | I don't understand then :) |
| 17:33.48 | therve | how do you get your email when you have your person? |
| 17:34.39 | salgado | we query the EmailAddress table where person = self.id |
| 17:35.28 | therve | ah right you have several email per person |
| 17:35.32 | salgado | maybe what's confusing is that I forgot to mention that .email is a non-storm property which queries the EmailAddress table like I described above |
| 17:35.57 | salgado | that's correct -- a person may have as many email addresses as they want |
| 17:37.11 | therve | so yeah that's tricky |
| 17:37.28 | therve | because the search on the EmailAddress table doesn't use the cache |
| 17:38.23 | therve | but you don't have person.email then, as you previously said? |
| 17:39.28 | salgado | we do have person.preferredemail, but that's not because of a Reference() -- it was written by ourselves |
| 17:40.39 | therve | if you're doing several store.find(), there's no way storm can do only one query |
| 17:40.49 | therve | at least without heavy black magic :) |
| 17:43.47 | niemeyer | salgado: Can preferredemail do store.get(EmailClass, self.the_preferred_email_id)? |
| 17:44.27 | salgado | and how about a special ResultSet which instead of returning Person objects, gives me tuples of persons and their preferred emails? I mean, how complex would it be for me to write a special resultset which did that? |
| 17:45.02 | salgado | niemeyer, no, we don't have the ID of the preferred email |
| 17:47.05 | salgado | not before we query the EmailAddress table, I mean |
| 17:47.10 | therve | salgado: you can already have a ResultsSet which return (Person, Email), as I said |
| 17:51.47 | salgado | indeed |
| 17:52.28 | salgado | but I'm on crack; that doesn't really help me |
| 17:53.01 | therve | heh |
| 18:16.32 | salgado | therve, ok, how about a special ResultSet which, instead of returning (Person, Email) would store that Email in Person._cached_email for later use in one of the methods/properties of Person? |
| 18:17.42 | therve | salgado: that would work |
| 18:18.04 | therve | salgado: I don't really see an easy API though |
| 18:18.56 | salgado | any idea how much work would it be? would ResultSet._load_objects() be a suitable place to do this? |
| 18:21.02 | therve | salgado: imho the place is not the most importang thing |
| 18:21.13 | therve | salgado: how do you tell storm to do that is the real question |
| 18:23.32 | salgado | well, for a start I guess I could achieve that by monkey patching storm to use my custom ResultSet |
| 18:24.35 | therve | ah right I see what you mean |
| 18:26.04 | therve | _load_objects is a nice try indeed |
| 19:33.25 | *** join/#storm pyCube (n=jmayfiel@12.87.213.22) |
| 19:44.17 | pyCube | hmm.. so i brought this up a few weeks back but havent had time to revisit it. I am working with a db that has a global toc table. when a row is created in some table, a trigger func inserts into the toc and hands the id back to be used as the primary key. When i create someTable obj and try store.add(someObj), it bails: raise LostObjectError("Can't obtain values from the database " |
| 19:45.12 | pyCube | my hack around it is to very unsafely query out the next id from the toc and manually add it to the created sotrm obj before i do commit() |
| 19:45.25 | pyCube | mysql, btw |
| 19:46.46 | pyCube | theres got to be a better way |
| 21:44.24 | *** join/#storm bla44 (n=bla@63.225.98.148) |
| 23:55.31 | *** join/#storm jukart (n=jukart@85-124-221-45.static.xdsl-line.inode.at) |