| 02:32.52 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 06:52.36 | *** join/#storm seiflotfy1 (n=seif@41.234.10.102) |
| 08:33.34 | *** join/#storm seiflotfy2 (n=seif@41.234.15.44) |
| 11:54.24 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 12:19.01 | mup | storm/trunk r306 committed by thomas@canonical.com |
| 12:19.01 | mup | Merge django-compatibility-fix [r=niemeyer,therve] [a=statik] |
| 12:19.01 | mup | Add support for newer versions of Django by not checking arguments passed to |
| 12:19.01 | mup | _cursor. |
| 14:58.52 | *** join/#storm fcorrea (n=fcorrea@c9342788.virtua.com.br) |
| 15:26.03 | *** join/#storm Busata (n=Dries_De@dD5776017.access.telenet.be) |
| 15:45.33 | *** join/#storm vvinet (n=vince@modemcable101.133-202-24.mc.videotron.ca) |
| 15:56.40 | *** join/#storm RainCT (n=RainCT@ubuntu/member/rainct) |
| 15:56.52 | RainCT | Hi |
| 15:57.46 | RainCT | How can I do a group_by so that for each group it takes the item which was inserted the last? |
| 15:58.08 | Busata | the id but descending? |
| 15:58.15 | Busata | if the table has an unique id, that is |
| 15:59.27 | RainCT | The table has fields like: one_id, another_id, timestamp. I want to get the 5 items with the highest timestamp but without more than one item with the same other_id |
| 16:02.43 | RainCT | so I'm doing a order_by(Desc(timestamp)) group_by(another_id) [:5], but it seems like it's taking in each group the item with the lowest timestamp or something and so I don't get the items |
| 16:04.48 | Busata | the first question you asked was easier for me to solve :( |
| 16:05.05 | RainCT | hehe |
| 16:05.42 | RainCT | You understand what I mean though? |
| 16:05.51 | Busata | yes, but not quite sure how to do it |
| 16:07.14 | Busata | not quite familiar with more advanced queries yet :) |
| 16:18.30 | RainCT | ok, I need to do SELECT *, MAX(timestamp) |
| 16:18.36 | RainCT | Do you know how to do this with storm? |
| 16:20.12 | *** join/#storm thekorn (n=markus@a89-182-9-128.net-htp.de) |
| 16:25.07 | Busata | mm, not really, maybe you can do it with store.execute, https://storm.canonical.com/Tutorial , there's an example @ "Executing expression" |
| 16:25.37 | Busata | but there will be more helpfull people here I'm sure, I just lurk! |
| 16:25.50 | RainCT | yeah, but if I'm going to use .execute I can quick out Storm directly :P |
| 16:26.19 | RainCT | Busata: thanks for your help though |
| 16:26.36 | Busata | oh & a but lower there's the "expresion values", that might be even more usefull |
| 16:26.44 | Busata | bit* & no problem |
| 16:45.37 | RainCT | Busata: turns out it's just .max(Table.timestamp) on the self.store.find object |
| 16:45.40 | therve | Busata: you can do that iirc: store.find((MyObj, Max(MyObj.field1)).group_by(MyObj.field2) |
| 16:45.54 | RainCT | too easy :) |
| 16:45.56 | therve | or RainCT ^^ |
| 16:46.16 | therve | RainCT: it doest do what you need though? |
| 16:47.50 | RainCT | Yes, but good to know that there's also another way to write it |
| 16:48.43 | RainCT | (but the .max approach is better here because I only do the GROUP BY sometimes, so I can just do if do_grouping: x.max(..) x.group_by(..)) |
| 16:50.40 | RainCT | thanks therve |
| 16:50.50 | therve | RainCT: no problem! |
| 16:57.23 | Busata | see, helpfull people! |
| 16:58.08 | RainCT | btw, how fast is storm at inserting items into multiple tables? |
| 16:58.23 | RainCT | (talking about over 50.000 items at once) |
| 16:59.36 | Busata | the only thing I timed once was inserting 1000 transactions in one table , including downloading em from a public api, and it was done in 4 seconds :p |
| 17:03.20 | RainCT | OK, that's not that bad. Inserting 80.000 items here took like 15 minutes when I last tried it yesterday, but I guess our code needs improvement |
| 17:05.05 | Busata | yes, it was first 40 seconds, then I noticed I committed the changes after _every_ item |
| 17:38.17 | *** join/#storm fcorrea (n=fcorrea@c9342788.virtua.com.br) |
| 18:15.47 | jkakar | RainCT: I've found that running store.commit every N (where N is in the range of 100-250) inserts helps performance. At some point, trying to insert too many items in a single transaction results in poor performance on the database side of things. |
| 19:14.20 | RainCT | jkakar: will try that out, thanks |
| 19:59.58 | *** part/#storm Busata (n=Dries_De@dD5776017.access.telenet.be) |
| 20:16.41 | *** join/#storm RainCT_ (n=RainCT@62.175.149.58.dyn.user.ono.com) |
| 22:31.59 | *** join/#storm seiflotfy1 (n=seif@41.234.15.44) |