| 00:07.32 | *** join/#storm zmijunkie_ (~sascha@i59F5E830.versanet.de) |
| 00:38.42 | *** part/#storm magmatt (~Matt@74-92-245-181-Utah.hfc.comcastbusiness.net) |
| 02:57.51 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 03:18.56 | *** join/#storm jdobrien (~john@184.88.18.24) |
| 04:07.30 | *** join/#storm wallyworld (~quassel@27-33-46-253.static.tpgi.com.au) |
| 04:21.21 | *** join/#storm EdwinGrubbs (~quassel@70-239-164-128.lightspeed.snantx.sbcglobal.net) |
| 04:21.21 | *** join/#storm EdwinGrubbs (~quassel@canonical/launchpad/edwingrubbs) |
| 05:46.22 | *** join/#storm stub (~stub@125.24.156.66.adsl.dynamic.totbb.net) |
| 05:46.23 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 08:12.17 | *** join/#storm wallyworld_ (~quassel@27-33-46-253.static.tpgi.com.au) |
| 08:20.07 | *** join/#storm keppla (~keppla@i577B3887.versanet.de) |
| 09:41.00 | *** join/#storm danilos (~danilo@cable-89-216-144-44.dynamic.sbb.rs) |
| 09:41.00 | *** join/#storm danilos (~danilo@canonical/launchpad/danilos) |
| 09:44.50 | *** join/#storm niemeyer (~niemeyer@201-2-134-72.pltce701.dsl.brasiltelecom.net.br) |
| 11:00.12 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 12:09.47 | *** join/#storm jdobrien (~john@184.88.18.24) |
| 12:50.10 | *** join/#storm wallyworld (~quassel@27-33-46-253.static.tpgi.com.au) |
| 13:52.16 | *** join/#storm danilos (~danilo@cable-89-216-144-44.dynamic.sbb.rs) |
| 13:52.16 | *** join/#storm danilos (~danilo@canonical/launchpad/danilos) |
| 15:35.01 | *** join/#storm lvh (~lvh@python/psf/lvh) |
| 15:35.03 | lvh | Hi! |
| 15:35.31 | lvh | I'm considering using Storm with Twisted. Should I use lp:~therve/storm/twisted-integration? |
| 15:35.52 | lvh | I'm guessing just using a sqlite store is okay, since it typically doens't block for very long |
| 15:36.07 | jkakar | lvh: It's considered experimental. |
| 15:36.39 | jkakar | lvh: I believe others are using it successfully, though, but it isn't really maintained as far as I know. |
| 15:36.54 | lvh | Right, that's what I thought |
| 15:37.06 | lvh | I'll just do the Axiom thing, use sqlite, and hope blocking isn't very bad |
| 15:37.06 | jkakar | lvh: The way we've used Storm in Twisted is to use deferToThread and do all Storm object interactions in the thread. It works well for our use case. |
| 15:37.17 | lvh | Ah, yes, that could work too |
| 15:37.20 | lvh | Thanks :) |
| 15:37.29 | jkakar | lvh: Anytime. :) |
| 15:38.29 | *** part/#storm bac (~bac@canonical/launchpad/bac) |
| 15:58.46 | *** join/#storm sidnei (~sidnei@200-203-14-142.cslce701.dsl.brasiltelecom.net.br) |
| 16:24.51 | *** join/#storm magmatt (~Matt@74-92-245-181-Utah.hfc.comcastbusiness.net) |
| 16:49.24 | *** part/#storm magmatt (~Matt@74-92-245-181-Utah.hfc.comcastbusiness.net) |
| 16:58.43 | *** join/#storm keppla1 (~keppla@i577B14E1.versanet.de) |
| 17:38.56 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 17:39.44 | jtv | jkakar: thought this might interest you⦠https://code.launchpad.net/~jtv/storm/profile-fetches/+merge/43323 |
| 17:42.20 | jkakar | looks |
| 17:46.59 | jkakar | jtv: I'm scared. |
| 17:47.18 | jtv | jkakar: that is⦠somewhat unexpected. |
| 17:47.24 | jkakar | jtv: :) |
| 17:47.47 | jtv | Care to elaborate? |
| 17:48.21 | jkakar | jtv: Yeah, sure. |
| 17:48.34 | jkakar | jtv: Storm was initially designed to be explicit. |
| 17:48.51 | jkakar | jtv: Adding this kind of implicit "behind your back" behaviour is kind of going against that grain. |
| 17:48.52 | jtv | Yes, I don't actually want to mess with that. |
| 17:49.02 | jtv | But it might be very easy to layer on top. |
| 17:49.12 | jkakar | jtv: Also, we have a tracer. You can use it to see which statements are being run, and when they are run, if you care to do profiling. |
| 17:49.22 | jkakar | jtv: So I guess I don't really understand how/when this will be used. |
| 17:50.05 | jtv | That's not the same kind of profiling though. |
| 17:50.33 | jtv | When we trace queries, we don't know what triggers a particular query and AFAIK we don't have any support for aggregating by context. |
| 17:50.59 | jkakar | jtv: You could trivially write a tracer to emit/store a traceback whenever a statement was executed. |
| 17:51.13 | jkakar | Or even extend the debug tracer that already exists to optionally do that. |
| 17:51.34 | jtv | But that doesn't expose the dependencies between queries, does it? |
| 17:51.48 | jkakar | jtv: My concern is that this is another kind of tracer, but we already have a tracing system. |
| 17:52.08 | jkakar | jtv: Well, you can see that lots of queries are being run from the same callsite (as in a loop) with tracebacks...? |
| 17:52.48 | jtv | What I'm doing here is not tracing. It's profiling of dependencies between queries. |
| 17:52.52 | jkakar | jtv: Anyway, I don't mean to be negative, I don't really understand your use case, so it's entirely likely I'm prejudiced here. |
| 17:53.06 | jtv | No worries! |
| 17:54.12 | jkakar | I'm also slightly concerned about the speed impact this will have profiling isn't wanted. |
| 17:55.01 | jtv | Profiling is still disabled by default, of course. |
| 17:55.08 | jtv | Anyway, to explain my use case further: |
| 17:55.47 | jtv | The first step is that I want to expose how one query leads to another in the application. The data dependencies. |
| 17:56.28 | jtv | The second is that I want to be able to eliminate queries without fiddly tweaks across abstraction barriers in the app. |
| 17:57.00 | jtv | This is why the example in the MP has the "derived query" in a function. I think that's realistic: |
| 18:00.10 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 18:01.16 | jtv | jkakar: I think I'm back. What was the last you saw of me? |
| 18:04.07 | jkakar | jtv: Hi! |
| 18:04.09 | jkakar | jtv: I saw: |
| 18:04.16 | jkakar | This is why the example in the MP has the "derived query" in a function. I think that's realistic: |
| 18:04.42 | *** join/#storm jdobrien (~john@24.18.88.184.cfl.res.rr.com) |
| 18:06.19 | jtv | jkakar: thanks, that's where I got my connection error. |
| 18:06.27 | jtv | So to resume there: |
| 18:07.20 | jtv | I see a lot of queries being triggered by dereferencing object attributes, in a function that's logically separate from the one that fetched the objects whose attributes are being dereferenced. |
| 18:07.59 | jkakar | Yep, makes sense. |
| 18:08.08 | jtv | For situations like that, I see maintenance hazards in apps trying to map out and pre-fetch the objects they need. |
| 18:08.21 | jtv | Not to mention cost, of course. |
| 18:08.52 | jtv | So I would like to have some way to give the ORM a license to optimize queries, but leaving the details to dynamic decision-making. |
| 18:09.36 | jtv | That has its risks too, obviously, but insofar as we can generalize them, we can avoid them generally without having to revisit our old mistakes all over the app. |
| 18:10.27 | jkakar | jtv: I guess I'm not convinced about the approach. I mean, JITs are great, but they're also tricky and magical. |
| 18:11.07 | jkakar | jtv: We've had a good experience with the tracing facility in Landscape, for any page we can get the queries run and, if we want, a traceback for each one... it's easy to find problems and actually fix them. |
| 18:12.40 | jkakar | jtv: Anyway, I'm curious to see what kind of results you get with this work. I don't want to discount anything without evidence... I'm just really a big fan of KISS and this feels rather complex. |
| 18:12.59 | jtv | I don't think it's particularly complex, actually. |
| 18:14.36 | jtv | I used to believe in our current approach, but the architecture research and post-graduate courses gave me the feeling that the kind of approach I'm taking here can work out massively better in practice. |
| 18:15.50 | jkakar | jtv: Cool, It'd be create to have something massively better than we have now. Tracing performance problems is tricky and boring. |
| 18:16.10 | jkakar | jtv: Maybe I'm being pessimistic (I do have a flu right now, so it's entirely possible). |
| 18:16.14 | jtv | :-) |
| 18:16.35 | jtv | Well I'm not a big fan of tracing for ongoing optimization, for several reasons. |
| 18:17.08 | jtv | They're a _great_ learning tool, but how do you re-use results? |
| 18:18.03 | jtv | We currently do that through experience, discussion, documentation. Dynamic optimization turns it into code and puts it to use wherever it applies. You can also measure the "coverage" (applicability, basically) of individual optimizations. |
| 18:21.40 | jkakar | jtv: That's a good point, yes. |
| 18:21.59 | jtv | Thanks. Another, although I admit I can't quantify it, is stale optimization. |
| 18:22.00 | jkakar | jtv: But it comes at a cost to every operation. So the wins need to be greater than the overall cost. |
| 18:22.13 | jtv | Absolutely. I kept that in mind. |
| 18:22.54 | jtv | The research work I'm referring to was on stuff where a nanosecond can be a long time, so believe me, I've had that hammered into me. :) |
| 18:24.36 | jtv | About the stale optimization: when you've hand-tweaked code in one place to elide specific queries in another, it's hard to keep track of the tweaks' continued usefulness. |
| 18:24.53 | jtv | You can test for it, but I think that's one of those⦠|
| 18:25.11 | jtv | reminds me of something I used to say: "the tunnel is circular. We added the light to keep 'em walking." |
| 18:26.27 | jtv | It's often easy to add an individual prefetch. But no alarm bells go off when you stop needing it. Arguably, having an alarm bell go off is a harmful distraction to developers. |
| 18:27.47 | jtv | Then, quis custodiet ipsos custodies? The test may not be representativeâbarring costly high-level tests, you may not notice when some other piece of the code starts pre-fetching the same data you do. |
| 18:28.31 | jkakar | jtv: That's a good point. |
| 18:28.32 | jtv | Of course there comes a time when you do wish alarm bells had gone off: when you want to make a far-reaching change like we just did in Rosetta. |
| 18:28.46 | jtv | Now you have to worry aboutâ |
| 18:28.59 | jkakar | jtv: Also, I'm *really really* happy that you're taking an interest in Storm. :) |
| 18:29.00 | jtv | ° Should I update this optimization? |
| 18:29.13 | jtv | Hey, I've been fascinated by ORMs since 1998 or so. :) |
| 18:29.33 | jtv | ° If I update this optimization without first getting traces, will I do the right thing or will I just flail about? |
| 18:29.34 | jkakar | Yeah, I've looked your C++ code for PostgreSQL. It's very cool. |
| 18:29.41 | jtv | Thanks! ⺠|
| 18:29.58 | jtv | ° Is this optimization still needed? Maybe our change makes things so much better I can forget it anyway. |
| 18:30.09 | jtv | ° Will I be hitting the right indexes? |
| 18:31.34 | jtv | Then, there's the cost to re-use. For instance we routinely have methods that give rise to very different access patterns depending on whether they're invoked from the web-service API, the web UI, or a script. |
| 18:32.46 | jtv | So instead of trying to clean out the worst instances of naïve ORM-induced access patterns, I'd like to see if we can legitimize them. Keep the convenience, but fight the performance problems structurally. |
| 18:34.35 | jtv | My profiling patch allows you to catalog access patterns for a particular region of code that you've marked as a context, but also of that region in the isolated context of an individual call path leading to it. |
| 18:35.07 | jtv | (I'd like to add decorators so you can say "this method is a context, so profile it.") |
| 18:35.35 | jtv | All the cleverness has to be out of the main loop, of course. |
| 18:36.00 | jkakar | Yeah. |
| 18:36.01 | *** join/#storm jdobrien (~john@24.18.88.184.cfl.res.rr.com) |
| 18:36.25 | jkakar | jtv: Dinner is ready here, so I'll step away, but I'm curious to see where you get with this. It'd be great if Storm gave you good performance magically. |
| 18:36.39 | jtv | Thanks. Glad to have discussed it with you! |
| 18:36.43 | jkakar | Me too. :) |
| 18:36.51 | jtv | Bon appétit |
| 18:42.13 | jtv | jkakar: (historical note⦠actually I shouldn't have been so surprised at the fear you expressed; Tom Lane reacted much the same when I introduced automatic transaction replay for serialization errors in libpqxx ⺠) |
| 18:55.44 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 18:59.07 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 19:13.02 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 19:31.11 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 19:33.16 | *** join/#storm jtv (~jtv@27.130.84.36) |
| 21:10.21 | splashs | love KISS btw, hopefully storm stays that way |