| 00:00.44 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 00:08.45 | *** join/#storm dobee (n=dobee@85-124-200-100.static.xdsl-line.inode.at) |
| 00:32.17 | *** join/#storm reacocard (n=reacocar@rccy-06-1010.dsl.iowatelecom.net) |
| 00:34.00 | reacocard | hi all. I just started tinkering with storm and I was wondering if there was an easy way to get a list of all values for a certain attribute? eg if I have a store with a bunch of objects in it, and each object has the property "name", is there a wasy to list all the values of name without instantiating all the objects or using SQL? thanks. |
| 00:35.13 | jkakar | reacocard: Yep, there is. For example, consider a Person object with name, email, address, city, province and postal_code fields. |
| 00:35.51 | jkakar | reacocard: You could do store.find(Person, <criteria>).value(Person.postal_code, Person.city) to get just postal codes and cities for matching Person's. |
| 00:36.24 | jkakar | reacocard: Oops, that's store.find().values, with an S on the values. |
| 00:36.46 | reacocard | sweet, let me try that, thanks |
| 00:37.25 | jkakar | np |
| 00:44.12 | reacocard | sweet, works great |
| 00:45.54 | reacocard | as long as I'm in here, what's the best way to deal with sqlite and threading when using storm? right now i've got sqlite in its own thread and I queue tasks into the thread, but this gets somewhat awkward and prevents me from using the generators storm produces to their potential |
| 00:59.22 | reacocard | could I move the thread queue handling inside storm itself, by writing a new backend or something? would that work? is it even possible? |
| 01:53.50 | jamesh | reacocard: For the Zope apps we've developed, we use one store per thread |
| 01:54.32 | jamesh | combine that with Zope using a thread pool to serve requests and you get a fixed number of long lived connections. |
| 01:56.51 | reacocard | jamesh: well, I dont have a fixed number of threads... it's a desktop app, so most interaction with the db will be very short |
| 01:57.35 | reacocard | I'm investigating the backend method right now, it looks like there's less than a dozen methods what would need wrapping, so it may well be feasable |
| 02:04.46 | jamesh | reacocard: you are probably best extracting all the information you need from the database objects and passing that to the other thread, yes. |
| 02:05.15 | jamesh | even if sqlite threading wasn't an issue, transaction handling would be. |
| 02:05.41 | jamesh | operations inside a database transaction are inherently serialised |
| 02:39.48 | *** join/#storm kov (n=kov@debian/developer/kov) |
| 03:00.16 | reacocard | that was far easier than I'd expected. http://rafb.net/p/vnuY5K99.html |
| 03:25.50 | jamesh | reacocard: that is an interesting approach, but you probably don't want multiple threads using the same store at the same time. Having one thread perform a transaction with the store and then another thread do the same would be fine though |
| 03:26.44 | reacocard | jamesh: things is, since its a desktop application multiple threads need to be able to access it simultaneously. |
| 03:27.08 | reacocard | eg, it could be running one task in the background whle also responding to user input |
| 03:27.34 | jamesh | reacocard: well, you'll need to serialise that access at the transaction level |
| 03:27.37 | jamesh | rather than statement level |
| 03:27.58 | reacocard | ? |
| 03:29.02 | jamesh | reacocard: imagine you have one thread that wants to create two objects linked by a Reference() |
| 03:29.21 | jamesh | reacocard: and then another thread that deletes all the rows from one of those tables |
| 03:29.29 | reacocard | ah, I see |
| 03:29.35 | jamesh | you don't want to interleave those operations |
| 03:30.01 | reacocard | well, I can always just make more Store objects, this approach allows that just fine. |
| 03:30.18 | jamesh | so if you have multiple threads wanting to operate on the store, let the first thread create its objects and commit(), then let the second thread delete its rows and commit() |
| 03:31.45 | reacocard | so, for my background task, I should make a new Store for it to do its work in, and let the UI have it's own store for interactive use? |
| 03:34.40 | reacocard | the only problem comes when I have two independant tasks trying to make alterations on the same Store, if I'm understanding you correctly, so as long as all write tasks use their own Store, it shouldn't be an issue? |
| 03:34.44 | jamesh | I don't really know how you're structuring your app |
| 03:35.14 | jamesh | If you want to have concurrent access to a database, use multiple stores |
| 03:35.38 | reacocard | it's a media player, I'm using storm the hold the media DB. so I need to be able to scan for media in the background while the user interacts with it in the foreground |
| 03:35.53 | reacocard | so each of those needs its own Store? |
| 03:36.24 | jamesh | making the background task use its own store would make sense. |
| 03:36.31 | reacocard | ok, will do |
| 03:36.48 | jamesh | if the foreground stuff is read only, just rollback() the store after you've done your accesses |
| 03:37.22 | jamesh | otherwise you won't see any changes made elsewhere |
| 03:37.45 | reacocard | ok |
| 03:38.10 | reacocard | most of the time it'll be read only, and I can just make another Store for the few write cases |
| 03:39.06 | jamesh | just remember that you aren't using MyISAM tables on MySQL -- transactions matter. |
| 03:39.13 | reacocard | :) |
| 03:45.15 | *** part/#storm reacocard (n=reacocar@rccy-06-1010.dsl.iowatelecom.net) |
| 03:48.30 | *** join/#storm nullie (n=ilya@87.254.150.23) |
| 04:17.24 | *** join/#storm jrydberg_ (n=johan@212.112.160.94) [NETSPLIT VICTIM] |
| 04:17.25 | *** join/#storm vinsci (n=vinsci@unaffiliated/vinsci) [NETSPLIT VICTIM] |
| 04:17.25 | *** join/#storm weatherman (n=weatherm@S0106001b11b2d97a.vc.shawcable.net) [NETSPLIT VICTIM] |
| 04:19.01 | *** join/#storm nullguid (n=ilya@87.254.150.23) |
| 04:23.35 | *** join/#storm jrydberg_ (n=johan@212.112.160.94) [NETSPLIT VICTIM] |
| 04:25.30 | *** join/#storm jukart (n=jukart@85-124-221-45.static.xdsl-line.inode.at) |
| 04:28.53 | *** join/#storm vinsci (n=vinsci@unaffiliated/vinsci) [NETSPLIT VICTIM] |
| 04:28.53 | *** join/#storm weatherman (n=weatherm@S0106001b11b2d97a.vc.shawcable.net) [NETSPLIT VICTIM] |
| 05:31.02 | *** join/#storm dobee (n=dobee@85-124-200-100.static.xdsl-line.inode.at) |
| 05:46.29 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 06:57.53 | *** join/#storm dobee (n=dobee@lsfw01.lovelysystems.com) |
| 07:01.16 | *** join/#storm nullie (n=ilya@87.254.150.23) |
| 07:53.48 | *** join/#storm niemeyer (n=niemeyer@200-138-40-171.ctame705.dsl.brasiltelecom.net.br) |
| 08:07.03 | *** join/#storm mcella (n=mcella@ip-179-87.sn2.eutelia.it) |
| 08:35.22 | *** join/#storm goschtl (n=goschtl@p5B0BF7C3.dip.t-dialin.net) |
| 09:02.42 | *** join/#storm BradAllen (n=BradAlle@1f.83.d12c.cidr.airmail.net) |
| 09:03.31 | *** join/#storm lightyear (n=ben@nat/fluendo/x-96a390cf943d9350) |
| 09:04.16 | lightyear | therve: I want to commit it to my branch. What mail adress should I use for the --author parameter with your name? |
| 09:05.21 | lightyear | therve: btw... it works ;) |
| 09:31.01 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 09:31.43 | therve | lightyear: I think you can take the contribution for you :) |
| 09:31.59 | therve | lightyear: but my mail is thomas@canonical.com |
| 09:32.19 | lightyear | k |
| 10:06.02 | jamesh | therve: you should register that address to your LP account |
| 10:06.24 | jamesh | that'll link your commits to your account on e.g. https://code.edge.launchpad.net/~storm/storm/trunk |
| 10:09.00 | therve | jamesh: I think I already did that? |
| 10:09.52 | therve | oh no I did the other one, thanks |
| 10:12.59 | therve | jamesh: errrrrr |
| 10:13.11 | therve | jamesh: now we have a problem |
| 10:13.16 | therve | jamesh: I can't connect anymore |
| 10:14.37 | jamesh | therve: really? What error do you see? |
| 10:15.03 | therve | jamesh: I've been logout, and I'm unable to login anymore |
| 10:15.14 | therve | jamesh: I've requested a password reset to see |
| 10:16.14 | jamesh | therve: hmm. Assuming you're in the beta testers team, could it be that you logged in to launchpad.net, were redirected to edge.launchpad.net where you weren't logged in? |
| 10:16.23 | jamesh | there are separate session cookies for the two instances |
| 10:16.36 | therve | jamesh: no I was in edge |
| 10:17.06 | therve | jamesh: it tells me: "This account cannot be used." |
| 10:17.11 | lightyear | therve: even with the patch I still have a problem, when I try to compare the values... |
| 10:17.36 | therve | lightyear: do you have a test? |
| 10:17.48 | lightyear | ah... sorry.. It uses the wrong storm version ;) |
| 10:18.02 | therve | ok I prefer that :) |
| 10:26.24 | lightyear | hm... or maybe not... |
| 10:26.30 | lightyear | lemme check smth |
| 10:28.25 | lightyear | nope... even with my branch this error :( |
| 10:29.40 | lightyear | the problem seems to be that it goes to storm/properties.py:60:__get_ ... dunno yet, if I can have a test case ... |
| 10:30.05 | lightyear | I think it might be beacause it is not yet flushed/commited... |
| 10:30.06 | lightyear | +t |
| 10:30.09 | lightyear | lemme try |
| 10:33.12 | lightyear | hmpf... can't reproduce it there :( |
| 10:34.29 | lightyear | but maybe... there is something wrong in my code before... |
| 10:38.23 | lightyear | arg |
| 10:48.34 | *** join/#storm nullie (n=ilya@87.254.150.23) |
| 11:01.21 | lightyear | got it ... ha! |
| 11:03.41 | lightyear | huu... that is hard. |
| 11:05.30 | lightyear | therve: this test http://paste.ubuntu.com/24690/ , procudes this traceback: http://paste.ubuntu.com/24691/ |
| 11:05.38 | lightyear | -, |
| 11:06.12 | therve | lightyear: the test is missing a part |
| 11:06.26 | lightyear | oups |
| 11:07.01 | lightyear | there is the missing line http://paste.ubuntu.com/24692/ |
| 11:07.14 | therve | thanks |
| 11:08.38 | therve | hum yeah :/ |
| 11:08.43 | therve | it's mostly the same problem |
| 11:10.03 | lightyear | jop |
| 11:12.37 | niemeyer | therve: Interestingly enough, you probably can't login because of one issue we overlooked on the migration of Launchpad to Storm |
| 11:14.42 | therve | niemeyer: now I feel responsible :) |
| 11:19.30 | therve | niemeyer: unfortunately, this is not the only issue |
| 11:19.51 | jamesh | therve: sorry, was doing a bit of packing for my flight. The problem is a bad caching bug, and is probably transient |
| 11:20.10 | jamesh | therve: did you actually get logged out when registering a new email? |
| 11:20.14 | therve | jamesh: https://bugs.launchpad.net/launchpad/+bug/244499 |
| 11:20.16 | ubot3 | Malone bug 244499 in launchpad "can't login anymore on launchpad after switching several mail addresses" [Critical,In progress] |
| 11:20.37 | therve | jamesh: lifeless tries to look at it |
| 12:27.17 | therve | lightyear: so I've look at it, and I don't have any solution for now |
| 13:36.58 | *** join/#storm BradAllen (n=BradAlle@69.15.123.154) |
| 13:37.33 | *** join/#storm BradAllen (n=BradAlle@69.15.123.154) |
| 13:38.30 | lightyear | therve: bad :( |
| 13:49.39 | BradAllen | Wow, LaunchPad migrated to Storm? Cool! |
| 13:55.38 | radix | yep! |
| 13:55.55 | therve | wellll |
| 13:56.04 | therve | that's a good news, sure |
| 13:56.15 | radix | actually, I wonder if it's on production yet? |
| 13:56.27 | therve | it is |
| 13:57.08 | therve | radix: http://launchpadlibrarian.net/15753021/Capture.png |
| 13:57.51 | radix | oops |
| 13:57.59 | BradAllen | Good error ID code |
| 13:58.18 | BradAllen | Hopefully the transition to new ORM won't be too bumpy |
| 13:59.23 | BradAllen | For me it's helpful because my boss is skeptical about using Storm; I am putting it in some code planning to go into production in Oct |
| 14:00.02 | BradAllen | having LaunchPad go into production adds credibility |
| 14:00.34 | therve | landscape is in production too :) |
| 14:00.44 | BradAllen | yeah, I already mentioned that one |
| 14:00.51 | BradAllen | but LaunchPad is something we can see |
| 14:01.09 | therve | sure, and the volume is not the same |
| 14:02.35 | BradAllen | yeah, I guess it's pretty high volume |
| 14:19.12 | lightyear | therve: the 'resolve-lazy-value' signal is meant to be synchronous? |
| 14:20.15 | lightyear | means... who ever reacts changes the value before the emit is over...? |
| 14:23.19 | radix | I think eventually to have better twisted integration we're going to need a way to be more explicit in storm about what operations should hit the database |
| 14:25.42 | jrydberg_ | Storm doesn't interact well with Twisted at this point? |
| 14:27.20 | radix | jrydberg_: you can use it pretty fine if you run your own usercode in threads and make sure a store stays in one thread |
| 14:27.47 | radix | (or you could even use it in the reactor thread if you don't mind blocking, i.e. with sqlite) |
| 14:28.05 | radix | but therve has been working on a branch to actually create an asynchronous Store API |
| 14:28.17 | radix | which wouldn't require your user code to be in a non-reactor thread, and also wouldn't block |
| 14:28.38 | radix | my "eventually" comment only referred to that thing |
| 14:28.45 | jrydberg_ | radix: t.i.defer based? |
| 14:28.51 | radix | jrydberg_: of course :) |
| 14:29.04 | radix | so that all the database operations return Deferreds |
| 14:29.29 | jrydberg_ | but accessing fields are still direct? |
| 14:29.47 | radix | jrydberg_: simple column values, yes |
| 14:29.48 | jrydberg_ | i mean, foo.field won't be a deferred? :) |
| 14:30.02 | radix | jrydberg_: of course things like reference fields need to be fetched asynchronously |
| 14:31.01 | jrydberg_ | therve: what's the state of your branch then? |
| 14:31.16 | jrydberg_ | or maybe radix you could answer that one too. |
| 14:32.22 | radix | there are some issues related to lazy loading |
| 14:32.29 | radix | as lightyear discovered |
| 14:39.47 | lightyear | hu? |
| 14:41.50 | lightyear | therve: just discussed it with twi: we could create an async property that does the lazy retrieval with a deferToThread and returns a deferred |
| 14:43.12 | lightyear | otherwise I don't see a way we can make this be really none-blocking because the store is flushing internally when you try to retrieve a lazy value |
| 14:43.34 | lightyear | (not always, but could happen and could block) |
| 14:45.33 | lightyear | or does anyone have a better idea? |
| 14:49.35 | lightyear | but as the properties are also used internally, we might should add a new method get() that goes to the thread and does it asynchrounous and still have the old __get__ for the cases you are in thread or the user is sure that it is cached as he manages the commits in a certain way |
| 14:50.47 | lightyear | so we would have __get__ with the usual behaviour that might break on sqlite and blocks and we have get() that returns a deferred and does it asynchrounous |
| 14:51.02 | lightyear | then it would be: my_model.title.get() |
| 14:52.10 | lightyear | hmm... does that work with this kind of classes (that have the __get__ model)? |
| 14:56.49 | therve | lightyear: no it doesn't |
| 14:57.26 | lightyear | i just figured out ;) |
| 14:57.41 | lightyear | but is the __get__ used internally by the store? |
| 14:58.41 | therve | I'm not following you |
| 15:08.01 | lightyear | o.k. slightly different.. will present something later... |
| 15:08.10 | lightyear | then we can discuss it |
| 15:23.03 | *** join/#storm nullie (n=ilya@87.254.150.141) |
| 15:34.21 | *** join/#storm jkakar (n=jkakar@S0106001b11b2d97a.vc.shawcable.net) |
| 15:44.03 | lightyear | therve: system wise something like this: http://paste.ubuntu.com/24749/ |
| 15:44.35 | lightyear | you have the usual property and you have an async property that (some how) knows about the thread to use and returns a deferred instead |
| 15:45.58 | therve | lightyear: that's a solution indeed |
| 15:46.37 | lightyear | the ugly part of it is to know about the thread |
| 15:48.06 | lightyear | if we can have something nice and generic there, that would be awesome |
| 15:48.59 | therve | lightyear: I would prefer to find some way to do it during the commit() call |
| 15:49.12 | lightyear | what? |
| 15:49.21 | therve | tries something |
| 15:49.26 | lightyear | change the lazy-callback? |
| 15:49.33 | lightyear | 'll wait |
| 16:15.24 | therve | lightyear: http://paste.ubuntu.com/24754/ |
| 16:16.10 | lightyear | is it a problem of the commit? |
| 16:16.18 | therve | lightyear: wellll |
| 16:16.44 | therve | lightyear: the value is only available once the commit is done |
| 16:16.47 | lightyear | isn't it the problem that the get calls from the wrong thread? |
| 16:17.03 | lightyear | I'll try it |
| 16:17.05 | therve | lightyear: yeah, but this mod makes sure the get doesn't have to make a query |
| 16:18.36 | lightyear | so it is not doing a lazy query...? |
| 16:19.04 | therve | yep |
| 16:21.01 | lightyear | hm... o.k. that works... |
| 16:21.19 | therve | heh don't be so surprised :) |
| 16:21.22 | lightyear | but it means that it stores all that in memory now... |
| 16:21.38 | lightyear | I'm not surprised... but it looks like... not right to me ;) |
| 16:22.23 | lightyear | because this change just prevents code from beeing executed that is there for a reason ;) |
| 16:22.42 | therve | well sure |
| 16:23.12 | therve | but it's the same thing for all returning a list |
| 16:23.32 | therve | you have to make choice when going async |
| 16:25.16 | lightyear | your patch is faster and easier ti implement, so I'll go for it |
| 16:25.23 | radix | therve: |
| 16:25.24 | radix | [12:16] <therve> lightyear: the value is only available once the commit is done |
| 16:25.35 | radix | therve: I don't have very much context, but I expect it's when the _flush_ is done, not the commit |
| 16:25.44 | radix | therve: for example, SEQUENCE-typed columns |
| 16:25.50 | therve | radix: right |
| 16:26.04 | therve | radix: in this case, default value from the database |
| 16:26.42 | radix | therve: yeah, that's available after flush as well |
| 16:27.10 | therve | ok |
| 16:56.05 | BradAllen | Is the Storm EventSystem normally only used internal to Storm? Or do most Storm API users normally capture events such as 'changed' etc. |
| 16:56.09 | BradAllen | ? |
| 16:57.05 | BradAllen | Currently we are mostly working with 'schema' objects which only have attributes for Storm properties and references |
| 16:57.10 | BradAllen | no methods |
| 16:57.29 | BradAllen | I am starting to think about some kind of facade class |
| 16:58.04 | BradAllen | or else attaching the methods directly to the schema objects, and maybe registering methods as events |
| 16:58.44 | BradAllen | so that if a property changes value, a special method can be triggered |
| 17:00.17 | BradAllen | I am not sure if events are really well supported (no docs) |
| 17:00.44 | BradAllen | so maybe using a facade class would be more straightforward |
| 17:01.05 | BradAllen | any opinions? |
| 17:16.10 | lightyear | hm... I still have the issue... |
| 17:23.28 | lightyear | __get__ still breaks for me |
| 17:23.46 | therve | in which circumstances? |
| 17:24.19 | lightyear | but it seems to be a much more complex case than the test case we already have... |
| 17:24.49 | lightyear | I use the deferred store in the media scanner for elisa |
| 17:25.49 | lightyear | but maybe this is because of my code... let me check.. |
| 17:27.08 | lightyear | peng... |
| 17:28.29 | lightyear | have a test that fails... so, I'll see what is the problem... |
| 17:31.57 | therve | lightyear: want me to have a look? |
| 17:32.56 | lightyear | dunno... did you see the two little hacks I added in my branch? |
| 17:33.17 | therve | nope |
| 17:33.24 | lightyear | it is now this part that needs to be adapted to the new behaviour of get() on the defferred store |
| 17:33.57 | lightyear | so, I'll see myself ;) |
| 17:34.39 | therve | ah alright |
| 17:56.27 | *** join/#storm lightyear (n=ben@nat/fluendo/x-4c147c4ad41c73da) |
| 17:56.45 | lightyear | arg |
| 17:56.49 | lightyear | therve? did you see my last patch? |
| 17:56.51 | lightyear | http://paste.ubuntu.com/24765/ |
| 17:57.48 | therve | now I have |
| 17:57.50 | therve | :) |
| 17:58.07 | therve | oh |
| 17:58.09 | therve | hum crap |
| 17:59.11 | lightyear | jop |
| 17:59.29 | lightyear | because the internal flush sets the lazy query on it... |
| 18:01.36 | lightyear | therve: any idea? |
| 18:02.04 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 18:03.09 | therve | lightyear: always :) |
| 18:03.20 | therve | lightyear: not any great ones although for now |
| 18:03.49 | lightyear | well... as long as it fixes my problem it is o.k. for now ;) |
| 18:06.57 | lightyear | we can make it properly later |
| 18:12.32 | therve | lightyear: http://paste.ubuntu.com/24768/ |
| 18:12.42 | therve | I think it addresses the issue more at the root |
| 18:13.26 | lightyear | :)) I though about what you might do and I only found that solution for now but thought... "no, that is tooo much of a hack" ... :)) |
| 18:13.39 | therve | yay |
| 18:13.45 | therve | bbl, see you |
| 18:13.49 | lightyear | but yes... that should fix it |
| 18:13.55 | lightyear | seeo |
| 18:13.59 | lightyear | +you |
| 23:28.30 | *** join/#storm jkakar (n=jkakar@S0106001b11b2d97a.vc.shawcable.net) |
