| 00:50.37 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 02:05.37 | *** join/#storm mwhudson (n=mwh@canonical/launchpad/mwhudson) |
| 02:06.11 | mwhudson | can you do "insert into table (...) select ... from ..." with storm? |
| 02:12.43 | mwhudson | finds a but report with a comment from thumper, inevitably enoguh |
| 02:12.53 | mwhudson | wants this: https://bugs.edge.launchpad.net/storm/+bug/337494/comments/7 |
| 02:12.56 | mup | Bug #337494: Use ResultSets in subselects <Storm:In Progress by jkakar> <https://launchpad.net/bugs/337494> |
| 02:27.37 | jkakar | mwhudson: There's some debate about how to best impement it. |
| 02:27.55 | jkakar | mwhudson: The main issue is that it breaks the clean separation of responsibilities between Store/ResultSet and expressions. |
| 02:28.20 | jkakar | mwhudson: One thought is to make ResultSet itself an expr, which would probably be a better factoring than what's been proposed so far. |
| 02:28.41 | jkakar | mwhudson: I've been meaning to hack on it, but as usual, too many things, too little time. :) |
| 02:28.44 | mwhudson | i'm not totally sure how that bug applies to inserting from subselects tbh |
| 02:29.06 | mwhudson | in fact, i'm not sure how you'd do inserting from subselects in an orm at all... |
| 02:29.26 | jkakar | mwhudson: Oh, I guess I was mostly thinking about the branch. Yeah, it'd be nice if we could support inserts from subselects. Also, mass insertions. |
| 02:29.49 | mwhudson | in this specific case i'm just over optimizing like a madman |
| 02:30.20 | mwhudson | we have some other crazy code that does mass inserts by building sql with ''.join :/ |
| 02:33.22 | mwhudson | jkakar: making a resultset an expr would make me happy i think |
| 02:33.34 | jkakar | mwhudson: Cool. |
| 02:39.33 | *** join/#storm vvinet (n=vince@modemcable041.205-203-24.mc.videotron.ca) |
| 03:36.48 | *** join/#storm stub (n=stub@ppp-58-8-14-80.revip2.asianet.co.th) |
| 05:27.45 | jamesh | mwhudson: https://code.edge.launchpad.net/~jkakar/storm/resultselect/+merge/9870 <- here's jkakar's branch to make an expression out of a ResultSet if you're interested |
| 06:12.54 | *** join/#storm stub (n=stub@canonical/launchpad/stub) |
| 08:46.37 | *** join/#storm mikex2 (n=mikael@193.11.22.34) |
| 09:39.25 | jrydberg | therve: have you done any performance measurements on the twisted integration branch? |
| 09:40.12 | therve | jrydberg: no |
| 09:41.41 | *** join/#storm zmijunkie (n=chatzill@i59F5F088.versanet.de) |
| 09:49.46 | jrydberg | i'm trying to decide wish road to head down; the twisted integration branch, or just use adbapi |
| 09:51.30 | therve | it's fairly easy to choose from a feature perspective |
| 09:51.49 | therve | don't let performance interfere at this stage, premature optimization etc |
| 09:52.36 | jrydberg | mmm |
| 09:53.47 | jrydberg | i'm doing a start schema, and all these deferred references makes my head go *pop* |
| 10:08.11 | *** join/#storm elmom_ (n=elmom@vallila-gw.hupnet.helsinki.fi) |
| 10:08.42 | jamesh | jrydberg: isn't adbapi pretty much equivalent to using plain storm in a thread pool? |
| 10:08.58 | jamesh | (at least, that's how I remember it) |
| 10:10.12 | jrydberg | jamesh: adbapi is pretty much using standard dbapi in a thread pool, IIRC. |
| 10:10.52 | jamesh | jrydberg: I guess I remembered right then. That's pretty much how we use Storm in a number of Twisted apps at Canonical |
| 10:11.04 | jamesh | using the storm.zope code to manage the per-thread stores |
| 10:11.20 | jamesh | (this is without using the twisted-integration branch) |
| 10:14.53 | jrydberg | so you're running everything that's using Strom in a separate thread? |
| 10:15.48 | jamesh | we run an entire transaction with deferToThread() |
| 10:16.00 | jamesh | have some decorators to do the transaction begin/commit or abort |
| 10:16.34 | jamesh | as the thread pool reuses threads for the calls, storm.zope gives us connection reuse |
| 10:16.57 | jamesh | the one limitation is that you can't pass Storm objects or results over the call boundary |
| 10:18.08 | therve | yeah we've been doing that too |
| 10:18.26 | therve | also running with one Store in the main thread |
| 10:18.43 | jamesh | jrydberg: here's some public code using the technique: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/files/head%3A/lib/canonical/librarian |
| 10:19.19 | jamesh | at the top of db.py, you can see the decorators |
| 10:20.41 | jamesh | storage.py and web.py use those decorators |
| 10:21.35 | jamesh | note that if you're using the retry portions of those decorators, make sure your code acts transactionally: i.e. it is safe to rerun the function if the first run aborts |
| 10:23.20 | jrydberg | i see |
| 10:23.29 | jrydberg | thank you. maybe i should try that approach instead |
| 10:28.55 | jrydberg | jamesh: http://bazaar.launchpad.net/%7Elaunchpad-pqm/launchpad/db-devel/annotate/head%3A/lib/canonical/librarian/db.py#L16 |
| 10:29.01 | jrydberg | what module is that? |
| 10:29.20 | jamesh | jrydberg: Zope's transaction module. |
| 10:29.43 | jamesh | http://pypi.python.org/pypi/transaction |
| 10:30.04 | jrydberg | oh |
| 10:30.08 | jamesh | it depends on zope.interface, which you're already using |
| 10:31.41 | jrydberg | it's part of the zope3 package? |
| 10:31.50 | jrydberg | on ubuntu |
| 10:32.20 | jamesh | python-transaction |
| 10:33.14 | jrydberg | i guess my installation is too old :) E: Couldn't find package python-transaction |
| 10:33.14 | jamesh | when Zope3 was monolithic, I believe it was part of the big package |
| 10:33.42 | jamesh | you aren't running Karmic yet? :) |
| 10:34.29 | jamesh | looking at the Replaces: header, perhaps python-zodb is what you want |
| 10:35.36 | jrydberg | nah! :( well. i'll figure it out. thanks. |
| 10:37.33 | jamesh | Zope 3 was split into a number of smaller packages for the last release, but it has taken a while for distro packagers to catch up |
| 10:37.57 | jamesh | but it does mean that it is easier to make use of just the Zope bits you need |
| 11:06.40 | *** join/#storm elmom_ (n=elmom@vallila-gw.hupnet.helsinki.fi) |
| 11:43.09 | \sh | jamesh, thx for the storm <-> django bits :) |
| 11:43.47 | jamesh | \sh: no problem |
| 11:47.45 | \sh | jamesh, I hope django upstream will use storm by default, so they can make use of "multiple datastores in django", it's really annoying to workaround this problem |
| 11:48.16 | jamesh | \sh: I doubt that will happen. They are working on the multiple connection problem independently though. |
| 11:48.52 | \sh | jamesh, yes...but it looks like that they don't come to an end...it's one of the missing django features which could help makeing django more "enterprise ready" |
| 11:49.30 | jamesh | \sh: I wouldn't consider Django's default mode of operation with one database enterprise ready ... |
| 11:49.54 | jamesh | you can write proper transactional code, but it is not the default. |
| 11:53.59 | \sh | jamesh, that's what I said..without using more then one database django failes somewhat..good for "single private websites", but not enterprise related |
| 11:55.52 | jamesh | \sh: I was referring to the fact that in its default mode of operation you're likely to execute a number of transactions to satisfy a single HTTP request |
| 11:56.28 | jamesh | \sh: if the first few succeed but then a later fails, you can end up with inconsistent state in the db |
| 12:59.18 | jrydberg | I got a DATETIME field, and I want to aggregate by hour, week, month, ... Can I extract parts of the date in a portable way using Storm and group by that? |
| 12:59.40 | jrydberg | I know sqlite has a strftime function |
| 13:00.11 | therve | I don't think there is portable way |
| 13:01.16 | *** join/#storm fcorrea (n=fcorrea@189-54-24-33-nd.cpe.vivax.com.br) |
| 13:04.14 | jrydberg | prob not |
| 13:07.26 | jrydberg | can I define functions myself+ |
| 13:07.27 | jrydberg | ? |
| 13:08.04 | therve | sure |
| 13:17.25 | *** join/#storm mikex2 (n=mikael@193.11.22.34) |
| 13:17.29 | jrydberg | How do I do one of these in Storm: 'SELECT host.data, SUM(fact.bytes) FROM fact INNER JOIN host ON host.id=fact.host_id GROUP BY host.data' ? what I really can't figure out is the SUM() |
| 13:26.05 | *** join/#storm keppla (n=keppla@i577B25DA.versanet.de) |
| 13:40.43 | *** join/#storm srichter_ (n=quassel@c-76-19-135-180.hsd1.ma.comcast.net) |
| 13:58.58 | *** join/#storm vvinet (n=vince@132.210.76.201) |
| 14:09.57 | *** join/#storm stub (n=stub@ppp-58-8-16-190.revip2.asianet.co.th) |
| 14:21.57 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 15:03.28 | *** join/#storm oubiwann (n=oubiwann@c-24-8-208-199.hsd1.co.comcast.net) |
| 15:35.22 | *** join/#storm artista_frustrad (n=artista_@201-25-164-10.ctame704.dsl.brasiltelecom.net.br) |
| 18:40.53 | *** join/#storm mikex2 (n=mikael@193.11.22.34) |
| 19:00.59 | *** join/#storm artista_frustrad (n=artista_@201-15-204-64.ctame704.dsl.brasiltelecom.net.br) |
| 21:09.16 | *** join/#storm keppla1 (n=keppla@i577B081C.versanet.de) |
| 22:00.04 | *** join/#storm vvinet (n=vince@modemcable041.205-203-24.mc.videotron.ca) |
| 23:18.30 | *** join/#storm fcorrea (n=fcorrea@189-54-24-33-nd.cpe.vivax.com.br) |