| 02:44.27 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 02:47.35 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 02:49.56 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 02:52.37 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 03:07.44 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 03:15.16 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 03:18.19 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 05:17.28 | *** join/#storm wallyworld (~quassel@27-33-46-253.static.tpgi.com.au) |
| 06:00.02 | *** join/#storm _mup__ (~mup@li37-90.members.linode.com) |
| 07:34.07 | *** join/#storm thumper (~thumper@125-236-193-95.adsl.xtra.co.nz) |
| 07:34.07 | *** join/#storm thumper (~thumper@canonical/launchpad/thumper) |
| 07:51.01 | *** join/#storm zmijunkie (~sascha@pd95b6fd9.dip0.t-ipconnect.de) |
| 08:26.25 | *** join/#storm stub (~stub@125.24.106.249.adsl.dynamic.totbb.net) |
| 08:26.25 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 08:53.44 | *** join/#storm esteve (~quassel@cli-5b7ee22f.bcn.adamo.es) |
| 08:55.44 | *** join/#storm bigjools (~quassel@canonical/launchpad/bigjools) |
| 10:59.43 | *** join/#storm fcorrea (~fcorrea@189.85.82.50) |
| 11:24.49 | *** join/#storm paul__ (~paul@pat-address.sciences-po.fr) |
| 11:25.55 | *** part/#storm paul__ (~paul@pat-address.sciences-po.fr) |
| 11:26.37 | *** join/#storm pom_ (~pom@pat-address.sciences-po.fr) |
| 11:54.31 | pom_ | Hi there, I am new to storm. I am facing a problem regarding manipulation of large number of queries. I have to create over 1 billion rows/objects. But I am not sure how to do this correctly so that the all query is sent at once to the database backend (Mysql in my case). |
| 11:55.16 | pom_ | the question can then be rephrased : Does storm do a commit when store.add is called ? |
| 12:00.57 | therve | no |
| 12:01.24 | therve | if you're doing a very large number of queries, you should do commits from time to time |
| 12:01.33 | therve | how often depends on your application |
| 12:01.43 | pom_ | not the billion at once then |
| 12:02.35 | therve | it's probably not a good idea :) |
| 12:02.41 | pom_ | ok thanks, good to know I am going to test this |
| 12:03.01 | therve | otoh, if you really have a billion rows to insert, using storm may not be the best idea either |
| 12:03.45 | pom_ | yeah you're probably right but you know once you start with it you don't want to get back to sql |
| 12:04.22 | pom_ | but using storm to inject SQL directly to the backend doesn't have any performance effect does it ? |
| 12:05.19 | therve | no not really |
| 12:05.41 | therve | but I was thinking about a low level tool importing a csv file or something |
| 12:06.49 | pom_ | well yeah but my billion rows are defined dynamicly from store.find() actually |
| 12:08.01 | therve | ah, ok |
| 12:25.33 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 12:27.56 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 12:37.29 | *** join/#storm Aiwass (~Aiwass4@188.27.186.124) |
| 12:47.48 | *** join/#storm niemeyer (~niemeyer@201-11-241-43.pltce701.dsl.brasiltelecom.net.br) |
| 12:59.30 | *** join/#storm fcorrea (~fcorrea@186.213.201.106) |
| 13:09.05 | *** join/#storm yosafbridge (~yosafbrid@li125-242.members.linode.com) |
| 13:26.58 | *** part/#storm bigdog1 (~scmikes@72-197-8-8-arpa.cust.cincinnaticomm.net) |
| 14:21.31 | *** join/#storm jdobrien (~john@rrcs-97-79-119-8.se.biz.rr.com) |
| 15:14.23 | *** part/#storm magmatt (~Matt@74-92-245-181-Utah.hfc.comcastbusiness.net) |
| 15:48.44 | *** join/#storm artista_frustrad (~artista_f@187.7.151.137) |
| 17:05.12 | *** join/#storm hagna (~hagna@74-92-245-181-Utah.hfc.comcastbusiness.net) |
| 17:06.05 | hagna | how can I make storm use an existing postgres table that doesn't have a primary key? |
| 17:07.45 | jkakar | hagna: Add a sequence column to it, that can act as a primary key... Storm can't work with a table without a primary key, as far as I know. |
| 17:08.05 | jkakar | hagna: The reason being that it has no way to cache live objects without a primary key. |
| 17:11.57 | hagna | postgres has a builtin oid column on tables maybe I could use that |
| 17:12.21 | jkakar | hagna: Perhaps, yes. |
| 17:15.13 | *** part/#storm medberry (~medberry@c-75-71-81-190.hsd1.co.comcast.net) |
| 17:19.25 | hagna | jkakar: ok I'll try that |
| 17:26.39 | hagna | I have heard there is a way to make an object out of a regular "select * from tablename" How do I do that/ |
| 17:26.42 | hagna | ? |
| 17:47.25 | jkakar | hagna: I'm not sure, I always used Store.find(MyObject) whenever I want that behaviour. |
| 17:47.57 | jkakar | hagna: You could use private methods on ResultSet to do it from a bare cursor result, but that's obviously not recommended. |
| 17:48.59 | hagna | <PROTECTED> |
| 17:49.52 | jkakar | hagna: Why? |
| 17:49.52 | jkakar | hagna: Storm is already well tested...? |
| 17:50.27 | hagna | my code isn't :) |
| 17:51.47 | hagna | the test tests that I got the table names right the oids work and probably other things I'm not aware of. |
| 17:52.45 | hagna | jkakar: so I'm not testing store.execute because Storm is already well tested. Make sense? |
| 17:55.41 | jkakar | hagna: I typically provide functions for creating objects, and also got getting them. |
| 17:56.13 | jkakar | hagna: Then I write a test like, thing = create_thing(); result = store.find(Thing); assertIdentical(thing, result.one()). |
| 17:56.37 | jkakar | hagna: Since store.find always runs a query, this ensures that create_thing creates an object and adds it to the database. |
| 18:02.08 | *** join/#storm fcorrea_ (~fcorrea@186.213.215.171) |
| 18:04.01 | hagna | jkakar: in my scenario the data already exists and I just want to make sure storm is doing the right queries based on my possibly crappy definitions |
| 18:04.18 | jkakar | hagna: I see. That's kind of tricky. :) |
| 18:04.20 | hagna | s/definitions/objects |
| 18:04.22 | *** join/#storm jdobrien (~john@50-88-19-25.res.bhn.net) |
| 18:06.20 | hagna | thanks :) |
| 18:08.33 | hagna | anyway I'm pretty close. storm.execute('select ...') returns a tuple with fields in a certain order and I'd just compare that somehow to store.find(Myobj, conditions).one() for a few data points |
| 18:09.12 | hagna | so I'm stuck on finding a clever way to test for equality between a tuple and a storm object |
| 18:09.47 | hagna | the simple way is to do tup[0] == stormObj.attr0 something like that? |
| 18:11.21 | jkakar | hagna: You could do something like assertEqual(list(store.execute("select ...")), list(store.find(Thing).values(Thing.column0, Thing.column1, ...))). |
| 18:11.39 | jkakar | hagna: ie, build a tuple representing the columns both from the raw SELECT and from the ResultSet. |
| 18:15.37 | hagna | ok I'll try that |
| 18:15.44 | hagna | thanks |
| 18:15.57 | hagna | didn't know about .values |
| 18:16.57 | jkakar | hagna: It's really handy because it pulls columns from the database without building live objects, which can be important (performance-wise) if you want to pull a lot of data. |
| 18:17.29 | jkakar | hagna: store.find((Table.column0, Table.column1, ...)) is equivalent to store.find(Table).values(Table.column0, Table.column1), FYI. |
| 18:29.50 | hagna | ok |
| 18:32.36 | *** join/#storm pom__ (~pom@nor75-17-82-67-199-199.fbx.proxad.net) |
| 18:55.40 | m4v | How do a query with a bitwise operator? |
| 18:55.51 | m4v | I have a query like "select * from table where number & 1", how I can do this in storm? store.find(Obj, Obj.number & 1) translates it to "number AND 1" which isn't the same. |
| 18:59.36 | jkakar | m4v: You need to implement your own expression, since there isn't a builtin Bitwise expression. |
| 19:01.27 | jkakar | m4v: Like this: http://paste.ubuntu.com/581257/ |
| 19:02.42 | m4v | jkakar: oh, thanks. |
| 19:03.07 | jkakar | m4v: np. |
| 19:40.43 | *** join/#storm zmijunkie (~sascha@i53873D4E.versanet.de) |
| 20:03.08 | *** join/#storm keppla (~keppla@i577B1CEB.versanet.de) |
| 21:32.28 | *** join/#storm fcorrea (~fcorrea@189.85.82.50) |
| 22:07.18 | *** join/#storm pom_ (~pom@pat-address.sciences-po.fr) |