| 00:00.44 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 05:30.07 | *** join/#storm reacocard (n=reacocar@rccy-06-1010.dsl.iowatelecom.net) |
| 05:46.58 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 07:20.09 | *** join/#storm goschtl (n=goschtl@p5B0BC2AB.dip.t-dialin.net) |
| 07:51.07 | *** join/#storm mcella (n=mcella@ip-179-200.sn2.eutelia.it) |
| 08:45.31 | *** join/#storm lightyear (n=ben@nat/fluendo/x-32137b2d5dc709ce) |
| 10:33.38 | tlynn | morning all. |
| 10:33.50 | therve | good morning |
| 10:34.39 | tlynn | I've been wondering about user-supplied validators to storm fields. something like http://paste.ubuntu.com/29268/ |
| 10:35.49 | tlynn | it only really works if it's in the base class for all properties though |
| 10:50.53 | jamesh | tlynn: if you grab the latest development version of storm, you'll have validators |
| 10:51.07 | jamesh | we need to make a new release some time ... |
| 10:52.04 | jamesh | tlynn: you can pass a validator keyword argument in the property definition, giving a function that takes arguments (obj, attr_name, new_value) and returns the value to set |
| 10:52.20 | jamesh | so it provides both validation (raise an exception on bad values) and value transformation (return something other than new_value) |
| 10:54.17 | tlynn | jamesh: excellent, when did that happen? |
| 10:54.45 | jamesh | tlynn: it was merged to trunk on 2008-04-25 |
| 10:55.16 | tlynn | hmm, strange I've not noticed/got it then. |
| 10:55.31 | jamesh | it is being used in a large code base (Launchpad), so has seen some real world testing :) |
| 10:56.04 | jamesh | tlynn: if you have r223 or newer, you should have it. |
| 10:57.28 | tlynn | is there a way to check from within Python what bzr rev it's running? |
| 10:57.44 | tlynn | I've been relying on sudo python setup.py install |
| 10:58.12 | jamesh | not easily. |
| 10:58.30 | jamesh | looks like we haven't updated the version string in the dev version since 0.12 either, which doesn't help. |
| 10:59.19 | tlynn | so what should I be looking for in the code for this feature? |
| 10:59.48 | jamesh | tlynn: searching for validator in storm/variables.py would be a good indication |
| 11:01.46 | tlynn | hmm, no, it hasn't updated. |
| 11:04.33 | tlynn | judicious rm -rf seems to have fixed it now |
| 11:05.38 | jamesh | it has been over half a year since the last release, so we really should try and get some of this out to users. |
| 11:06.24 | tlynn | I agree. this feature in particular makes it a lot better suited for building web apps on. |
| 11:12.05 | tlynn | I'd also like to see some of this stuff cleaned up and put in (Json variables and more UTF-8 encoding in the database): http://paste.ubuntu.com/29270/ |
| 11:28.32 | *** join/#storm fcorrea_ (n=fcorrea@c953f05a.virtua.com.br) |
| 11:42.54 | jamesh | I don't quite understand why you'd want to be storing text in multiple encodings in the database. Surely you'd be better off keeping it all in UTF-8/unicode and doing your encode/decode at the edge of the application? |
| 11:45.42 | jamesh | as for the JSON stuff, does the Pickle column type do what you want? |
| 11:46.22 | jamesh | [that said, I generally try to avoid mutable variable types] |
| 11:50.43 | fcorrea_ | jamesh, hi :) |
| 11:51.05 | fcorrea_ | Just checked out your branch and ran the tests. All passing. |
| 11:51.24 | jamesh | fcorrea_: thanks. |
| 11:51.49 | fcorrea_ | I mean, the branch you attached to the postgres version problem |
| 12:21.24 | tlynn | jamesh: I do only want UTF-8, but I want unicode values to "just work" too |
| 12:21.47 | tlynn | and re. JSON, it's more appropriate than pickle for values that will be shared with other languages |
| 12:23.46 | tlynn | on the unicode thing, if you store UTF-8 data normally, you have to use RawStr, which uses X'...' strings in the database, which are hard to work with |
| 12:41.25 | *** join/#storm niemeyer (n=niemeyer@201-89-10-24.ctame705.dsl.brasiltelecom.net.br) |
| 12:46.49 | niemeyer | Good morning! |
| 12:54.55 | *** join/#storm kov (n=kov@debian/developer/kov) |
| 13:29.11 | lightyear | hi guys |
| 13:29.24 | lightyear | is there a way to do DISTINCT in storm easily? |
| 13:29.39 | therve | hi lightyear |
| 13:29.50 | lightyear | like get only one item per each value in a column? |
| 13:30.03 | therve | ResultSet has config method |
| 13:30.04 | lightyear | therve: hi. doing cool UI stuff with storm for elisa... |
| 13:30.11 | therve | with a distinct argument |
| 13:30.37 | lightyear | ah. that sounds good. thank you! |
| 13:30.47 | therve | lightyear: cool! it would be nice if I could help you get the twisted-storm branch sync |
| 13:31.00 | lightyear | therve: sure. |
| 13:31.12 | lightyear | removed some crap recently |
| 13:31.22 | therve | lightyear: for the other problem with ReferenceSet, could you open a bug? |
| 13:31.37 | therve | I don't want it to be lost |
| 13:31.38 | lightyear | the problem is, that we need to have the patch for the reference set |
| 13:31.49 | lightyear | the script I send the other day about the mem leak? |
| 13:31.55 | therve | lightyear: yep |
| 13:32.02 | lightyear | sure |
| 13:32.06 | therve | thanks |
| 13:32.58 | lightyear | hm... need to find the script ;) |
| 13:37.38 | lightyear | therve: maybe you want to add your possible patch: https://bugs.launchpad.net/storm/+bug/250819 |
| 13:37.42 | ubot3 | Malone bug 250819 in storm "reference set leaks memory" [Undecided,New] |
| 13:37.53 | lightyear | ui--- |
| 13:37.53 | therve | lightyear: yep, thanks |
| 13:39.44 | lightyear | therve: beside... I want to discuss: the implicit flush every time we do a get is quiet some overhead here |
| 13:40.10 | lightyear | therve: that is a huge problem in the case of the wrapper, as it is looping over all the items every time we do a flush |
| 13:40.55 | radix | there are some ideas about improving it |
| 13:41.13 | therve | lightyear: right, I'm aware of the problems |
| 13:41.18 | lightyear | but the flush is only necessary in very few cases: when the user did an external execute, otherwise it is not necessary |
| 13:41.31 | radix | lightyear: not true |
| 13:41.50 | radix | lightyear: the flush is to get storm's own modifications to happen |
| 13:41.57 | lightyear | what we did for our case (elisa) is that we know that we are only using the storm models and not the execute directly, so we disabled the loop in the wrapper |
| 13:42.09 | radix | the thing that's unnecessary is the event emitting |
| 14:00.08 | *** join/#storm fcorrea_ (n=fcorrea@c953f05a.virtua.com.br) |
| 14:05.27 | *** join/#storm bigdo1 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 14:06.01 | tlynn | lightyear: what's elisa? |
| 14:07.10 | lightyear | tlynn: http://elisa.fluendo.com |
| 14:07.16 | lightyear | a open source media center |
| 14:07.18 | lightyear | +n |
| 14:07.36 | lightyear | we use storm now for the database (currently only for audio based media) |
| 14:07.54 | *** join/#storm bigdo2 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 14:08.07 | radix | lightyear: oh, you work for fluendo, cool |
| 14:08.53 | radix | lightyear: is the database always local? |
| 14:08.54 | lightyear | radix: yep |
| 14:09.12 | lightyear | radix: does not have to be anymore... as we use create_database from storm |
| 14:09.27 | lightyear | that means you could now use mysql or similar (but not yet tested) |
| 14:09.29 | radix | ok, but I'm just wondering what the application will use |
| 14:09.32 | radix | ok, I see |
| 14:09.53 | lightyear | but sharing the database between multiple application might be problem as it is file-centric atm |
| 14:09.56 | radix | i was just thinking that maybe you would be able to avoid a lot of pain if you just used storm in normal blocking operation, since your database is local |
| 14:10.33 | lightyear | radix: elisa is a twisted application and one very high goal is to always have a reactive UI, so blocking is no way ;) |
| 14:10.42 | lightyear | and I am quiet happy with the twisted wrapper :) |
| 14:10.57 | lightyear | and it is fun to develop on it ;) |
| 14:11.49 | radix | I mean, even given Twisted and requirements for reaction |
| 14:12.15 | radix | if you are actually clever with your schema and how you do queries, you don't need to block for significant times |
| 14:12.39 | radix | another option is to just have *one* database thread which you communicate with manually |
| 14:12.45 | radix | with an application API |
| 14:14.08 | lightyear | radix: well... the one thread thing is done in the twisted wrapper for us :) |
| 14:14.30 | radix | I'm proposing something a bit different from that |
| 14:14.31 | lightyear | and we call the application API "storm" :D |
| 14:14.40 | radix | but yeah, it'd be nice if it worked :) |
| 14:14.45 | lightyear | it does |
| 14:15.09 | lightyear | at least here. I am quiet happy with it |
| 14:15.12 | radix | oh, that's good |
| 14:15.19 | radix | all of the nasty problems are resolved for you? |
| 14:20.15 | lightyear | radix: well... the memory leak in storm itself was a problem at some point ;) |
| 14:21.13 | lightyear | and everything else, for e.g. all the problems I find in the wrapper need to be fixed anyway, no? |
| 14:21.28 | lightyear | and atm I don't have any big ones left... |
| 15:14.55 | *** join/#storm ubot3` (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 15:27.41 | *** join/#storm reacocard (n=reacocar@rccy-06-1010.dsl.iowatelecom.net) |
| 15:46.24 | *** join/#storm fcorrea (n=fcorrea@c953f05a.virtua.com.br) |
| 15:52.39 | *** join/#storm reacocard (n=reacocar@rccy-06-1010.dsl.iowatelecom.net) |
| 15:59.22 | reacocard | quick question: if I load an object out of one store, close the store, open a new store and add the object again, doe sit just overwrite the old data for the row with that id? |
| 16:11.27 | reacocard | anyone? |
| 16:17.57 | radix | reacocard: storm doesn't really ever expect you to use an object from one store in a different store |
| 16:18.01 | radix | so probably it'll blow up horribly |
| 16:18.14 | reacocard | radix: but I closed the first store first |
| 16:18.21 | radix | even then. |
| 16:18.27 | reacocard | bleh |
| 16:18.33 | radix | why not just keep it open>? |
| 16:19.21 | reacocard | because I'm using sqlite and have multiple tasks that will be doing stuff in different threads. I have the threading solved but DB locking is giving me problems |
| 16:20.39 | reacocard | eg I have two stores open with pending changes. I can't commit either because it says the db is locked |
| 16:22.16 | tlynn | you should probably add a clone method to your objects |
| 16:22.24 | reacocard | ? |
| 16:22.39 | tlynn | a method that copies their fields into a new object with no associated store |
| 16:22.59 | reacocard | ah, and then if I add this one back it will overwrite, assuming the id is the same? |
| 16:23.20 | tlynn | if overwriting is what you want, then yes |
| 16:23.29 | reacocard | sweet, thanks |
| 16:23.58 | tlynn | (probably) |
| 16:27.19 | reacocard | assuming I don't have anything fancy in my object, would python's normal copy.copy() method work? or do I actually have to write my own |
| 16:27.32 | tlynn | apparently adding it won't work, you need a clone method that changes a passed-in instance |
| 16:27.45 | reacocard | come again? |
| 16:28.24 | tlynn | as in instead of "def clone(self): ... -> obj" you want "def copy(self, other): ... -> sets fields of other" |
| 16:29.10 | reacocard | ew. that means I'd have to load the object I want to change from the db twice |
| 16:29.30 | reacocard | once to copy once to commit |
| 16:29.40 | tlynn | it probably won't load the fields of the target object |
| 16:29.54 | tlynn | since it knows you aren't reading them |
| 16:30.26 | tlynn | so you just pass in store.find(YourClass, YourClass.id=whatever).one() |
| 16:30.33 | tlynn | this is all theory, but it might work :-) |
| 16:31.15 | reacocard | but it is still querying the db again. when you're dealing with thousands of objects doubling the querys could be a bit of overhead |
| 16:32.52 | reacocard | also this is all really ugly do design for sinc eIm sloting it into an existing architechure. |
| 16:33.21 | reacocard | I lov ethe sorm API but how its interacting with sqlite is nto being nice |
| 16:34.10 | tlynn | well, if the worst comes to the worst, you can use raw SQL to do the update |
| 16:34.29 | reacocard | ew |
| 16:34.53 | tlynn | let's hope it doesn't come to that :-) |
| 16:35.05 | tlynn | I'm trying a mockup now |
| 16:35.09 | reacocard | wishes there was a good alernative to sqlite |
| 16:35.37 | tlynn | there's zodb3 if you don't need a relational db |
| 16:35.53 | reacocard | its nto up to date in ubuntu packages |
| 16:36.37 | reacocard | well no it is.... |
| 16:37.29 | reacocard | oh thats why, its nto compiled against python 2.5, onyl 2.4 |
| 16:37.33 | reacocard | so no good |
| 16:38.07 | reacocard | otherwise i'd have tried it before storm |
| 16:38.25 | reacocard | spent a while looking into this |
| 16:39.36 | reacocard | really I jsut need a good transactional way to store objects, but aside from zodb and storm I couldnt find much and we're using python2.5 in ubuntu so that put zodb out |
| 16:41.30 | tlynn | memcached over something like mysql might be good enough for you |
| 16:41.51 | reacocard | except we cant force use of mysql on desktop users |
| 16:42.15 | reacocard | I like mysql, works great, but end-users need something that works with no extra work |
| 16:42.24 | reacocard | hence sqlite |
| 16:42.32 | tlynn | sqlite isn't that bad |
| 16:42.46 | reacocard | no it isnt, it just has these stupid locking issues |
| 16:42.55 | tlynn | well, memcached would help there |
| 16:43.20 | tlynn | you could just have a single thread copying from memcached into sqlite |
| 16:43.31 | reacocard | hm |
| 16:43.47 | radix | you could do that just as well without memcached |
| 16:44.02 | reacocard | yeah memcached might be a bit overkill |
| 16:44.23 | radix | plus it's basically got the same deployment concerns as mysql, though to a lesser extent |
| 16:44.31 | tlynn | I agree, it's another dependency you'd probably be better off without |
| 16:44.53 | reacocard | really I have no problems wrapping stuff, I just want the API to be a simple to work with as possible and for end-users nto to have to configure anything more than absolutely necessary |
| 16:45.32 | tlynn | and to be fair to sqlite, I think these locking problems sound like they are really there in your app anyway |
| 16:45.33 | reacocard | I do want to have optional mysql support though, for users with more-intensive needs |
| 16:45.59 | reacocard | um, I dont need to commit in two places, I just want to make changes and hold them til theres a chance to commit |
| 16:46.11 | reacocard | I make these changes in the objects that hold the data |
| 16:46.12 | tlynn | it turns out that doing find(...).one() does load all the fields :-( |
| 16:46.18 | reacocard | I fail to see how this is wrong |
| 16:46.50 | reacocard | all I want is a way to tell storm "ok, dont touch the db with these changes until I say commit" |
| 16:46.53 | reacocard | so it wont lock |
| 16:47.49 | tlynn | well, make all your changes to objects without an associated store |
| 16:48.03 | tlynn | which is where we came in :-) |
| 16:48.20 | reacocard | which complicates the API, unless I do a copy for everything loaded from the DB which will increase ram use |
| 16:48.38 | tlynn | the increase in ram use is surely neither here nor there |
| 16:49.02 | reacocard | ..fine |
| 16:49.38 | reacocard | so, if I make a copy for everything, then I just need to re-add them back to the db to save, no? |
| 16:50.15 | tlynn | nope, you'll need to copy them into place using find().one() as discussed above |
| 16:50.21 | tlynn | adding creates new rows |
| 16:50.34 | reacocard | right but with that one staement yes |
| 16:51.15 | reacocard | so what I need are two methods on my object: get_copy() and save_data(obj) |
| 16:52.01 | tlynn | try something like: http://paste.ubuntu.com/29348/ |
| 16:54.18 | tlynn | the final paragraph of code could be converted into the body of a save_data method |
| 16:54.26 | reacocard | I see |
| 16:54.58 | tlynn | but as you noted, that does involve an unnecessary database read of the old values |
| 16:55.19 | reacocard | Im really beginning to think using one store for all may be the best way.... Its not any worse than what we were going to do before I switched to storm |
| 16:56.46 | tlynn | I'd ask again a few hours earlier in the day when therve and niemeyer are around, they might have better ideas |
| 16:57.10 | reacocard | I few hours earlier mean i'd be asleep :) |
| 16:57.23 | tlynn | but tbh the model you want seems to be "load old values, mess around in memory, save" |
| 16:57.29 | reacocard | right |
| 16:57.44 | tlynn | so you might want to copy at the start to objects with no associated store |
| 16:57.52 | tlynn | change them, then copy back to the real store |
| 16:58.40 | tlynn | anyway, I have to go now. sorry I don't have a great solution for you, but hope the discussion helped |
| 16:58.58 | reacocard | it gave me a lot to think about at least. thanks a lot :) |
| 16:59.14 | tlynn | np cya |
| 17:27.34 | *** join/#storm kov (n=kov@debian/developer/kov) |
| 17:47.59 | *** join/#storm niemeyer (n=niemeyer@201-89-10-24.ctame705.dsl.brasiltelecom.net.br) |
| 17:51.43 | *** join/#storm ubot3 (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 18:33.20 | *** join/#storm fcorrea (n=fcorrea@c953f05a.virtua.com.br) |
| 19:27.57 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 19:32.08 | *** join/#storm ubot3 (n=ubot3@unaffiliated/nalioth/bot/ubot3) |
| 22:45.15 | *** join/#storm bdmurray (n=bdmurray@mylar.outflux.net) |
| 23:08.00 | *** join/#storm kov (n=kov@debian/developer/kov) |
| 23:17.58 | *** join/#storm fcorrea (n=fcorrea@c953f05a.virtua.com.br) [NETSPLIT VICTIM] |
| 23:17.58 | *** join/#storm niemeyer (n=niemeyer@201-89-10-24.ctame705.dsl.brasiltelecom.net.br) [NETSPLIT VICTIM] |
| 23:17.58 | *** join/#storm rreis (n=rreis@201-13-2-99.dsl.telesp.net.br) [NETSPLIT VICTIM] |
| 23:17.58 | *** join/#storm rockstar (n=rockstar@75-171-130-208.hlrn.qwest.net) [NETSPLIT VICTIM] |