| 00:07.19 | *** join/#storm keppla (n=keppla@i577B3D05.versanet.de) |
| 00:10.21 | jkakar | thumper: back. |
| 00:10.41 | thumper | hi |
| 00:10.52 | thumper | I sort of got around it by using reset and reloading the object |
| 00:11.03 | thumper | but now I have another very weird situation |
| 00:11.14 | thumper | I'm doing a single query to realise 4 objects |
| 00:11.21 | thumper | but I'm getting two queries |
| 00:11.41 | thumper | is there a way that one of the objects when being created is causing another database query? |
| 00:12.41 | jkakar | thumper: Yeah, it could be causing flushing. |
| 00:13.11 | thumper | why? |
| 00:13.41 | thumper | I'm doing something like: |
| 00:13.58 | thumper | return [a for a,b,c,d in store.find((A,B,C,D), ...)) |
| 00:14.23 | jkakar | thumper: Oh, in that sense. |
| 00:14.44 | thumper | I'm priming the object cache |
| 00:14.51 | thumper | I know I'm going to be traversing through b,c and d |
| 00:14.53 | jkakar | thumper: I'm not sure if another query could be caused by doing that kind of thing. |
| 00:15.04 | thumper | it is |
| 00:15.04 | jkakar | thumper: Yeah, I've done that in places, too. |
| 00:15.08 | thumper | I just don't know why |
| 00:23.12 | *** join/#storm fcorrea (n=fcorrea@189-54-35-134-nd.cpe.vivax.com.br) |
| 01:32.09 | *** join/#storm stub (n=stub@ppp-58-8-240-134.revip2.asianet.co.th) |
| 01:48.51 | *** part/#storm fcorrea (n=fcorrea@189-54-35-134-nd.cpe.vivax.com.br) |
| 03:01.03 | *** join/#storm lifeless (n=robertc@ppp245-86.static.internode.on.net) |
| 03:01.08 | lifeless | ah hi |
| 03:01.29 | lifeless | I am trying storm again, for a uhm, ambitious, project |
| 03:01.57 | lifeless | I've got a project where the storm cache is going to be totally useless - can it be disabled? |
| 03:03.09 | *** join/#storm maxb (n=maxb@jabberwock.vm.bytemark.co.uk) [NETSPLIT VICTIM] |
| 03:28.38 | stub | lifeless: set the cache size to 0 |
| 03:31.32 | lifeless | stub: thanks. I've tried that but storm still seems to be caching up to the next transaction |
| 03:31.58 | lifeless | stub: I'm dealing with millions of rows, and don't have any 'userspace' references to new/modified objects. |
| 03:32.21 | stub | lifeless: Check out the add() method in cache.py - if the cache size is 0 it is a noop. |
| 03:32.31 | lifeless | So is there some way to get them flushed down to SQL and not take up space (and time!) anymore. |
| 03:32.45 | lifeless | stub: 26028 robertc 20 0 3182m 3.1g 3328 R 100 53.3 17:33.45 python |
| 03:34.11 | stub | lifeless: store.flush() |
| 03:35.59 | lifeless | stub: /something/ is still holding objects around :( |
| 03:36.04 | *** join/#storm flacom (n=sdf@200.73.47.130) |
| 03:36.30 | lifeless | I guess I'll have to analyse this more carefully. The irony is that I'm working on a db store for meliae, john's memory usage analyser! |
| 03:37.09 | stub | You have both the cache size set to 0 and am calling store.flush() regularly? |
| 03:37.25 | lifeless | yes |
| 03:37.59 | lifeless | <PROTECTED> |
| 03:38.06 | lifeless | <PROTECTED> |
| 03:38.10 | lifeless | <PROTECTED> |
| 03:38.13 | lifeless | <PROTECTED> |
| 03:38.38 | stub | Oh.... store.flush(); store.invalidate() |
| 03:38.45 | stub | (which is what commit does) |
| 03:41.00 | stub | Hmm... although that shouldn't help because the cache size is already 0 |
| 03:41.00 | stub | Dunno sorry |
| 03:41.00 | lifeless | no worries; thanks for poking around |
| 03:43.39 | thumper | lifeless: invalidate doesn't remove the objects |
| 03:43.43 | thumper | try store.reset() |
| 03:43.56 | thumper | flush, then reset |
| 03:44.04 | thumper | if you don't want to commit |
| 03:45.25 | lifeless | thumper: I do want to commit |
| 03:45.38 | lifeless | but not per insert, I have 40Million rows to insert |
| 03:45.52 | thumper | lifeless: commit then reset the store |
| 03:46.00 | thumper | the store has an _active weak dictionary |
| 03:47.25 | lifeless | that still doesn't help when adding [say] a single row with 900K reference nodes |
| 03:47.38 | lifeless | expresses frustration at caches |
| 03:49.16 | lifeless | (I don't mean to be ungrateful, I appreciate the extra data the storm docs skipped right over) |
| 03:50.30 | lifeless | I'm just finding it hard to get a solid performance model on storm; I found this with the conflictchecker way back. |
| 04:04.06 | *** join/#storm flacom (n=sdf@200.73.47.130) |
| 04:27.25 | lifeless | ok, it was _dirty |
| 04:27.35 | lifeless | just adding 900000 link nodes |
| 04:28.07 | lifeless | It would be great if _dirty could be disabled : its not at all clear from the docs etc that _dirty a) exists, and b) is a separate cache to the cache. |
| 04:37.46 | jamesh | _dirty contains the objects that have unflushed changes |
| 04:38.30 | jamesh | Storm uses it to keep those objects alive, and as a way to scan them during the flush |
| 04:39.41 | jamesh | it isn't a cache |
| 04:52.09 | *** join/#storm flacom (n=sdf@200.73.47.130) |
| 04:53.31 | lifeless | jamesh: it behaves like a write behind cache, except it's unbounded |
| 04:54.00 | lifeless | jamesh: I realise it has unflushed changes; however flush() is documented as something one shouldn't have to call. |
| 04:54.40 | lifeless | I think its reasonable if you don't have to call it, to insist that storm manage these references - e.g. by using a weak ref dict and flushing when the last ref goes, or observing the size of _dirty |
| 05:09.26 | jamesh | lifeless: weak refs don't work that way |
| 05:10.03 | jamesh | the weak ref callback doesn't get passed the object being finalised |
| 05:12.12 | lifeless | oh hmm |
| 05:12.47 | lifeless | its odd that it works that way |
| 05:13.32 | lifeless | anyhow, having an explicit cache would provide a clear separation of concerns: store.flush would be store._dirty.flush(); a write policy could be set etc |
| 05:13.44 | lifeless | unless you think that that wouldn't fit in the intent of storm ? |
| 05:15.01 | jamesh | http://paste.ubuntu.com/271341/ |
| 05:15.25 | lifeless | jamesh: yeah, I read the docs when you said |
| 05:15.38 | lifeless | :) |
| 05:15.48 | lifeless | its still odd IMO |
| 05:16.35 | jamesh | well, the weak ref systems I played with in C often gave you the pointer but warned you that you shouldn't dereference it because the object was no longer valid |
| 05:17.23 | jamesh | you can't very well do that in Python, so providing the object that referenced the destroyed object allows you to do similar things |
| 05:17.42 | lifeless | ideally a weakref can replace __del__, to allow sideways extension |
| 05:18.02 | lifeless | however, it can't, so we can ignore this tangent :) |
| 05:18.03 | jamesh | I'm not sure separating the flush handling from the store makes sense. |
| 05:18.34 | jamesh | making the flush handling more intelligent might be good, but the current behaviour has the benefit of being deterministic |
| 05:19.06 | jamesh | if a flush occurs, I know why. |
| 05:19.10 | lifeless | so, separating wouldn't make it random :) |
| 05:19.26 | lifeless | nor less deterministic; it would allow user selection of policy |
| 05:19.49 | lifeless | there is already a policy knob, but it seems to only have 2 settings |
| 05:20.05 | jamesh | the current policy is not simple |
| 05:20.30 | jamesh | if a user has a better idea of when flushes should occur, they can call store.flush() |
| 05:21.07 | lifeless | s/not// ? |
| 05:21.09 | jamesh | but I don't see any value in letting them e.g. replace the flush ordering logic or similar |
| 05:22.07 | jamesh | I'm trying to understand what exactly you'd want to change. |
| 05:23.26 | lifeless | Going back to my original problem, I'd like storm to not have a private variable that grows without bound and which I need to manually control its behaviour to have it work half decently |
| 05:23.27 | jamesh | and having flushes occur during GC is an incredibly bad idea: the GC doesn't enforce any thread affinity, and many web apps use thread local stores. |
| 05:23.38 | lifeless | jamesh: but I'm not proposing flush on gc |
| 05:24.09 | lifeless | jamesh: That was a off the cuff way of maybe doing it; forget about the weakref comment entirely, please. |
| 05:25.27 | jamesh | lifeless: so when do you want the flushes to occur? |
| 05:25.48 | lifeless | jamesh: when it makes sense |
| 05:26.19 | lifeless | caching every altered object in memory only works when few objects are altered |
| 05:26.47 | jamesh | lifeless: in most workflows, flushes occur fairly regularly |
| 05:26.54 | lifeless | flushing on every change is expensive because of db syncrhonisation |
| 05:27.07 | lifeless | bah, spellingk. :P |
| 05:27.09 | jamesh | it is bulk data inserts that usually cause problems |
| 05:27.48 | jamesh | every time you do store.find(), store.get() or store.execute(), a flush occurs |
| 05:28.02 | lifeless | jamesh: so, if storm were to do a flush every 5000 objects-since-a-flush, most folk wouldn't be affected. |
| 05:28.28 | lifeless | jamesh: and those that are doing bulk inserts would benefit [less multiple iterations and [relatively] slow dict manipulations |
| 05:28.59 | jamesh | lifeless: a policy like that is going to cause annoyingly hard to track down bugs. |
| 05:29.38 | lifeless | I just had an annoyingly hard to track down bug, because of the lack of the policy. |
| 05:29.47 | lifeless | which is a reason to make it clear, explicit and tunable. |
| 05:30.00 | jamesh | e.g. if it causes a flush at the wrong point and causes an insert or update that violates a database constraint |
| 05:30.11 | jamesh | that wouldn't happen if the flush happens one change later. |
| 05:30.33 | lifeless | uhm |
| 05:31.00 | lifeless | to me that doesn't make sense; as I model ORM's they are not meant to represent invalid database states ever anyway. |
| 05:31.33 | lifeless | e.g. flushing on every operation should behave the same, but slower, than flushing on commit - as you say doing 'store.find()' will flush anyway. |
| 05:31.48 | jamesh | lifeless: the one that people usually trip over is writing a constructor for their object, and triggering a flush midway through it. |
| 05:32.14 | lifeless | there is a block method to prevent automatic flushes anyway.. |
| 05:32.26 | lifeless | that would need to be honoured |
| 05:33.10 | jamesh | that was added to help port Launchpad over to Storm -- I'm not sure if anyone else uses it |
| 05:33.21 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 05:34.06 | jamesh | the hacked version of sqlobject Launchpad was using had an even simpler flush policy that basically amounted to only flushing on commit and when the user explicitly asks for it |
| 06:10.32 | *** join/#storm stub (n=stub@ppp-58-8-240-134.revip2.asianet.co.th) |
| 06:28.12 | *** join/#storm flacom (i=flacom@200.73.47.130) |
| 06:56.59 | *** join/#storm jukart_ (i=lovely@81.189.156.94) |
| 07:37.45 | *** join/#storm flacom (n=sdf@200.73.47.130) |
| 09:36.40 | *** join/#storm keppla (n=keppla@i577B3D05.versanet.de) |
| 09:59.15 | *** join/#storm zmijunkie (n=chatzill@i538725DF.versanet.de) |
| 10:33.52 | *** join/#storm stub1 (n=stub@58.8.17.147) |
| 11:00.07 | *** join/#storm seiflotfy (n=seif@ce3.net.hrz.tu-darmstadt.de) |
| 11:06.42 | jamesh | lifeless: about your suggestions/complaints about the flush handling, could you either send an email to the list about the problem or file a bug? |
| 11:06.48 | jamesh | (the first is probably better) |
| 11:08.05 | jamesh | I don't think the developers in other time zones will see the IRC conversation in the scrollback |
| 11:43.22 | lifeless | jamesh: sure |
| 12:02.15 | *** join/#storm bsdemon (n=bsdemon@mail.moneyps.ru) |
| 12:43.18 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 12:43.27 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 15:05.49 | *** join/#storm jdobrien (n=john@12.181.47.51) |
| 15:06.17 | *** join/#storm artista_frustrad (n=artista_@201-15-218-106.ctame704.dsl.brasiltelecom.net.br) |
| 15:06.28 | jdobrien | I would like to know when a remove is called on a storm class, even when find(StormClass, ...).remove() is called. Is this possible? |
| 15:12.04 | therve | I don't think so |
| 15:14.46 | jdobrien | :( |
| 17:20.02 | *** join/#storm jukart (n=jukart@d86-32-163-51.cust.tele2.at) |
| 17:34.31 | *** join/#storm artista_frustrad (n=artista_@201-11-10-223.ctame704.dsl.brasiltelecom.net.br) |
| 18:23.14 | *** join/#storm fcorrea (n=fcorrea@187.3.196.43) |
| 19:03.56 | *** join/#storm vvinet (n=vince@ip203-209.natdynamique.USherbrooke.ca) |
| 20:04.32 | mzz | aww, a question I had an answer for |
| 20:04.55 | mzz | if jdobrien gets back someone tell him that's intentionally not supported because you can delete rows while the corresponding storm object isn't in memory |
| 21:34.25 | *** join/#storm artista_frustrad (n=artista_@201-40-201-57.ctame704.dsl.brasiltelecom.net.br) |
| 22:07.27 | *** join/#storm shaunm (n=shaunm@98.212.133.244) |
| 22:25.12 | *** join/#storm vvinet (n=vince@modemcable041.205-203-24.mc.videotron.ca) |
| 23:39.54 | lifeless | jamesh: https://bugs.edge.launchpad.net/storm/+bug/430359 |
| 23:39.55 | mup | Bug #430359: store._dirty can hide large temporary leaks <Storm:New> <https://launchpad.net/bugs/430359> |
| 23:46.54 | *** join/#storm vvinet (n=vince@modemcable041.205-203-24.mc.videotron.ca) |
| 23:48.42 | *** join/#storm keppla1 (n=keppla@i577B2807.versanet.de) |
| 23:55.27 | *** join/#storm fcorrea (n=fcorrea@189-54-35-134-nd.cpe.vivax.com.br) |