| 00:00.19 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 00:00.41 | *** join/#storm mup_ (n=mup@li37-90.members.linode.com) |
| 00:01.27 | *** join/#storm tchough (n=tc@cpe-66-68-103-215.austin.res.rr.com) |
| 00:04.16 | tchough | hi guys, does anyone know if storm calls the constructors of objects instantiated from a datastore? |
| 00:05.22 | mzz | iirc no |
| 00:05.24 | mzz | checks |
| 00:05.53 | mzz | https://storm.canonical.com/Tutorial and look for "loading hook" |
| 00:06.53 | tchough | awesome, thanks for the help! |
| 00:06.58 | mzz | np |
| 00:31.14 | *** join/#storm gord (n=gord@5ac32897.bb.sky.com) |
| 03:12.00 | *** join/#storm pyCube_ (n=jmayfiel@c-24-5-148-105.hsd1.ca.comcast.net) |
| 03:47.51 | *** join/#storm benanzo (n=ben@c-98-232-16-38.hsd1.wa.comcast.net) |
| 04:18.51 | *** part/#storm benanzo (n=ben@c-98-232-16-38.hsd1.wa.comcast.net) |
| 05:34.24 | shaunm | besides the cache, does storm keep references to objects? |
| 05:34.52 | shaunm | I'm thinking it must at least keep references to things until they're flushed |
| 05:35.28 | shaunm | does it keep references until those changes are committed? are there other reasons it might keep objects alive? |
| 05:35.56 | shaunm | I'm having some pretty serious memory usage problems since switching to storm that I'm trying to figure out |
| 05:47.31 | jamesh | shaunm: if you follow a Reference(), storm keeps a reference to tho the referenced object |
| 05:47.50 | jamesh | but if you don't hold a ref to the object on the other end, then this shouldn't cause a problem. |
| 05:48.18 | jamesh | if you have unflushed changes, Storm will keep a strong reference to those objects too |
| 06:03.57 | shaunm | jamesh: but it doesn't hold flushed objects until commit? |
| 06:09.15 | shaunm | will definitely have to check on references |
| 06:10.52 | shaunm | it only keeps a ref if I actually access it through the Reference, yes? it doesn't auto-set and ref References on existing objects if I happen to get() the right record? |
| 06:10.55 | jamesh | shaunm: no. once the object is flushed, there is no reason that it must stay alive. |
| 06:11.38 | jamesh | it'll still hold a weak reference, but that won't stop the object from being collected |
| 06:12.04 | jamesh | if it is in the cache, that strong ref will however\ |
| 06:12.59 | shaunm | right, I switched to the GenerationalCache, and I put in some code to remove things from the cache that I know don't need to be there |
| 06:13.51 | jamesh | you might be able to use the gc module to track down what is holding the reference though. |
| 06:14.15 | jamesh | gc.get_referers(obj) |
| 06:15.21 | shaunm | sure |
| 06:15.45 | shaunm | I really need to figure out what is eating up so much memory though |
| 06:15.59 | shaunm | before worrying about why it's still alive |
| 06:16.30 | shaunm | no doubt there are plenty of places I could improve besides the database access |
| 06:17.12 | shaunm | but the fact that memory usage shot through the roof after I converted to storm makes my database access patterns pretty suspect |
| 06:17.14 | jamesh | as instances hold a reference to their classes, len(gc.get_referers(some_class)) can be a quick way to count how many of a particular object exists |
| 06:18.20 | shaunm | oh, nice. thanks |
| 06:19.11 | jamesh | actually, sys.getrefcount(some_class) is probably better |
| 06:19.28 | jamesh | no need to walk the object graph |
| 06:49.16 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 08:21.35 | *** join/#storm goschtl (n=goschtl@p5B0BC322.dip.t-dialin.net) |
| 10:12.27 | *** join/#storm jamesh (n=james@canonical/launchpad/jamesh) |
| 10:25.09 | *** join/#storm goschtl_ (n=goschtl@p5B0BBBF8.dip.t-dialin.net) |
| 10:45.50 | *** join/#storm cody-somerville (n=cody-som@ubuntu/member/somerville32) |
| 10:49.53 | cody-somerville | How would I do this in storm?: "select project_name from buildresults where date(finished_at) != date(now()) GROUP BY project_name" |
| 10:50.13 | cody-somerville | but I'd like to get the actual project object rather then just the project's name (which is the pk btw) |
| 10:58.16 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 10:58.16 | *** join/#storm gord (n=gord@5ac32897.bb.sky.com) [NETSPLIT VICTIM] |
| 11:51.01 | munin | store.get(Project, Project.date != now) ? |
| 11:51.34 | munin | that sql query will just get the projects name |
| 12:05.20 | *** join/#storm niemeyer (n=niemeyer@200-138-42-229.ctame705.dsl.brasiltelecom.net.br) |
| 12:36.23 | cody-somerville | munin, that won't work, there is no date field on Project |
| 12:36.45 | cody-somerville | munin, the finished_at field is a column in another table |
| 12:45.26 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 12:51.27 | *** join/#storm goschtl (n=goschtl@p5B0BB812.dip.t-dialin.net) |
| 13:06.42 | *** join/#storm vvinet (n=vince@132.210.76.199) |
| 15:16.19 | *** join/#storm oubiwann (n=oubiwann@97-119-85-2.omah.qwest.net) |
| 15:42.55 | shaunm | cody-somerville: store.find (Project. AnotherTable.project_id == Project.id, AnotherTable.finished_at != now) |
| 15:43.49 | cody-somerville | shaunm, and that'll be all of them. I wanted to case finished_at to just have the date, not the date and time |
| 15:44.11 | cody-somerville | shaunm, So that way I could see if any projects did not have a build record where the finished_at was today |
| 15:46.23 | shaunm | does sql do that? |
| 15:47.05 | shaunm | I'd probably just do (AnotherTable.finished_at >= start_of_today, AnotherTable.finished_at < start_of_tomorrow) |
| 15:47.36 | shaunm | pretty straightforward. I do similar sorts of things quite a lot in my code |
| 16:09.35 | *** join/#storm goschtl_ (n=goschtl@p5B0BF007.dip.t-dialin.net) |
| 16:14.34 | *** join/#storm shaunm (n=shaunm@proxyserver.wolfram.com) |
| 16:18.23 | *** join/#storm goschtl__ (n=goschtl@p5B0BD24A.dip.t-dialin.net) |
| 16:24.05 | cody-somerville | That might work. |
| 16:27.11 | shaunm | jamesh: hey, thanks for the tips yesterday. I've figured what's eating up memory. now I just need to figure out why |
| 16:28.58 | *** join/#storm goschtl (n=goschtl@p5B0BBB69.dip.t-dialin.net) |
| 16:39.13 | shaunm | oh wow |
| 16:39.43 | shaunm | so if I do "bar = foo.bar", where foo.bar is a Reference() |
| 16:39.55 | shaunm | foo has a ref on bar, yes? but does bar have a ref on foo? |
| 16:42.12 | shaunm | I'm creating a new object that has a number of references to other objects |
| 16:42.44 | shaunm | if I set them with "rev.branch = branch", they never go away. they sit in memory forever |
| 16:43.09 | shaunm | but if I set them with "rev.branch_ident = branch.ident", things work fine |
| 16:43.34 | shaunm | it's worth mentioning that, for the course of this operation, branch never goes out of scope |
| 16:44.12 | shaunm | this seems strange to me |
| 16:50.13 | mzz | shaunm: variation of https://bugs.launchpad.net/storm/+bug/250819 perhaps? |
| 16:50.14 | mup | Bug #250819: reference set leaks memory <Storm:Confirmed> <https://launchpad.net/bugs/250819> |
| 16:59.58 | shaunm | hmm, could be |
| 17:04.10 | shaunm | by switching from "rev.branch=branch" to "rev.branch_ident=branch.ident", I'm able to keep things from blowing up |
| 17:05.02 | shaunm | I put a bit of code in my local copy of GenerationalCache so that, each time it bumps the generation, it looks at all my tables and prints out those that have more than 100 refs |
| 17:05.25 | jkakar | Oh nice, I could see that being quite handy. |
| 17:05.27 | shaunm | before I did this change, the number of Revision and RevisionFile objects was in the tens of thousands. |
| 17:05.42 | jkakar | Just because of the ref-holding issue you just described? |
| 17:05.47 | shaunm | yes |
| 17:06.01 | jkakar | Hmm. |
| 17:06.12 | shaunm | but I'm still getting a few hundred of them. I don't understand why it's not almost 0 |
| 17:06.46 | shaunm | I forcibly remove these objects from the cache as soon as I get them |
| 19:43.06 | *** join/#storm vvinet (n=vince@132.210.76.199) |
| 20:02.40 | *** join/#storm deryck (n=deryck@samba/team/deryck) |