| 00:24.21 | *** join/#storm therve` (n=therve@xvm-14-10.ghst.net) |
| 00:25.02 | *** join/#storm zmijunkie (n=chatzill@i59F5CE3F.versanet.de) [NETSPLIT VICTIM] |
| 00:31.12 | *** join/#storm vvinet (n=vince@modemcable244.133-202-24.mc.videotron.ca) |
| 00:39.05 | *** join/#storm jamesh_ (n=james@canonical/launchpad/jamesh) |
| 00:42.07 | *** join/#storm zmijunkie_ (n=chatzill@i59F5C612.versanet.de) |
| 01:17.05 | *** join/#storm zmijunkie (n=chatzill@i59F5FA43.versanet.de) |
| 02:55.29 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 05:55.24 | *** join/#storm stub (n=stub@ppp-58-8-3-240.revip2.asianet.co.th) |
| 08:12.45 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 08:53.12 | *** join/#storm danilos (n=danilo@canonical/launchpad/danilos) |
| 10:35.24 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 10:41.59 | *** join/#storm salgado (n=salgado@canonical/launchpad/salgado) |
| 10:47.10 | *** join/#storm niemeyer (n=niemeyer@189-30-50-50.pltce701.dsl.brasiltelecom.net.br) |
| 11:06.47 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 11:15.05 | *** join/#storm jukart_ (n=jukart@91.112.115.74) |
| 11:29.29 | *** join/#storm jukart__ (n=jukart@91.112.115.74) |
| 11:31.50 | *** join/#storm jukart___ (n=jukart@91.112.115.74) |
| 12:12.42 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 12:16.51 | *** join/#storm jukart_ (n=jukart@91.112.115.74) |
| 12:46.07 | *** join/#storm fcorrea (n=fcorrea@187.3.137.148) |
| 13:28.07 | *** join/#storm vvinet (n=vince@smtpin.revolutionlinux.com) |
| 14:38.19 | *** join/#storm danilos (n=danilo@canonical/launchpad/danilos) |
| 15:12.38 | *** join/#storm danilos (n=danilo@canonical/launchpad/danilos) |
| 15:19.51 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 15:54.46 | *** join/#storm allenap` (n=gavin@tobermory.gromper.net) |
| 15:59.34 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 15:59.41 | allenap` | We've been having some performance issues with a query in Launchpad Bugs. |
| 15:59.59 | allenap` | Suppose we do store.find(Bug, ...) |
| 16:00.18 | allenap` | And it's expected to return, say, 100000 rows, |
| 16:00.44 | allenap` | If we iterate over the result set: for bug in store.find(Bug, ...): print bug.id |
| 16:00.55 | allenap` | Will Storm first try to pull back the whole result set, |
| 16:01.09 | allenap` | or does it pull in rows lazily, either individually or in batches? |
| 16:12.23 | salgado | allenap`, it used to pull them lazily, in batches |
| 16:13.25 | salgado | and I think it still does, or else we'd be having a lot more timeouts in Launchpad |
| 16:14.55 | allenap` | salgado: That was my hunch. Thanks. |
| 16:19.08 | salgado | allenap`, I'm wrong, I think |
| 16:19.52 | salgado | allenap`, it does fetch all items at once |
| 16:21.31 | salgado | and I don't know where I got the idea that it'd fetch them lazily -- I'm now pretty sure it's always been like this |
| 16:21.38 | therve | hum what? |
| 16:21.54 | therve | salgado: not it doesn't fetch all |
| 16:22.04 | therve | it uses fetchmany |
| 16:22.31 | therve | and yield objects one by one |
| 16:24.43 | salgado | therve, how does fetchmany work? |
| 16:25.09 | therve | it's database dependent I guess |
| 16:25.29 | salgado | I did iterate over a resultset here and watched the postgres logs but there was only one statement fetching all items at once |
| 16:25.44 | salgado | maybe there weren't enough items in that resultset, though |
| 16:26.05 | therve | well you will only get one postgres query for sure |
| 16:26.16 | therve | but that doesn't mean the cursor will get all the data at one |
| 16:27.09 | salgado | but wouldn't there be more queries for every item fetched? |
| 16:28.44 | therve | no, there is a single query sent to the database |
| 16:29.22 | therve | then you iterate over the rows of the result set |
| 16:29.39 | therve | but that doesn't create new queries |
| 16:38.08 | salgado | allenap`, so, next time you better not trust me. ;) |
| 16:38.43 | salgado | thanks for jumping in and clarifying things, therve. /me goes read psycopg2's source to understand how that's possible |
| 16:41.10 | therve | salgado: no problem! |
| 16:48.00 | allenap` | therve, salgado: Thanks :) |
| 16:49.25 | therve | allenap`: if you have a lot of objects, you may want to tweak arraysize on your cursor, althought most of the time is probably spent building python objects anyway |
| 16:50.55 | allenap` | therve: Is arraysize exposed in Storm, or do I have to dig through to the native cursor? |
| 16:51.43 | therve | allenap`: I don't think it's exposed, no :/ |
| 16:54.40 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 17:24.35 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 17:27.03 | *** join/#storm elmom_ (n=elmom@hoasnet-fe29dd00-137.dhcp.inet.fi) |
| 20:45.57 | *** join/#storm jukart (n=jukart@91.112.115.74) |
| 20:55.54 | *** join/#storm jukart_ (n=jukart@91.112.115.74) |
| 21:24.10 | *** join/#storm wgrant (n=wgrant@ubuntu/member/wgrant) |
| 23:28.58 | *** join/#storm fcorrea (n=fcorrea@201-75-152-196-nd.cpe.vivax.com.br) |