| 00:07.46 | *** join/#storm jamesh_ (n=james@canonical/launchpad/jamesh) |
| 01:29.56 | *** join/#storm maxx (n=fircuser@149.254.48.66) |
| 02:54.32 | *** join/#storm shaunm (n=shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 04:47.49 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 04:59.39 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 05:44.32 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 06:34.09 | *** join/#storm _thumper_ (n=quassel@125-236-193-95.adsl.xtra.co.nz) |
| 06:48.05 | *** join/#storm thumper (n=quassel@canonical/launchpad/thumper) [NETSPLIT VICTIM] |
| 08:54.34 | *** join/#storm ivan` (n=ivan`@ludios.net) |
| 08:55.48 | *** join/#storm jml (n=jml@69.56.173.200) |
| 09:51.34 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 12:21.41 | *** join/#storm vvinet (n=vince@modemcable101.133-202-24.mc.videotron.ca) |
| 12:31.04 | *** join/#storm niemeyer (n=niemeyer@200-138-37-48.ctame705.dsl.brasiltelecom.net.br) |
| 12:54.41 | *** join/#storm bigdog1 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 13:22.58 | *** join/#storm mcayer (n=chatzill@modemcable250.87-70-69.static.videotron.ca) |
| 13:24.52 | mcayer | hi! |
| 13:25.11 | mcayer | could somebody help me with commit within thread? |
| 13:31.32 | radix | mcayer: hi |
| 13:31.36 | radix | store.commit()? |
| 13:31.38 | mcayer | hi! |
| 13:31.41 | mcayer | yep |
| 13:32.08 | *** join/#storm vvinet (n=vince@132.210.76.208) |
| 13:32.53 | mcayer | my problem is the following |
| 13:33.17 | mcayer | i want to do two commit, one at the beginning, one at the end of the thread |
| 13:33.36 | mcayer | however, db is updated only on the second commit, not on the first |
| 13:39.33 | radix | mcayer: what's the problem? |
| 13:39.56 | radix | oh, you mean you expect it to be updated on the first commit but it's not? |
| 13:40.03 | mcayer | exactly |
| 13:40.09 | radix | what kind of changes are you making to the objects? are you making sure you're using the store and all its objects in a thread-safe manner? |
| 13:40.25 | radix | (i.e. make sure you *do not* use a store or any objects associated with a store from more than one thread during its lifetime) |
| 13:40.47 | mcayer | from the documentation, it is said to create a new store from the same database inside a thread |
| 13:40.55 | radix | yes |
| 13:40.55 | mcayer | that's what I do |
| 13:40.58 | radix | okay cool |
| 13:41.14 | mcayer | simply said => store = Store(database) :) |
| 13:42.13 | radix | mcayer: okay, so what changes are you making to objects that you would expect to be committed? |
| 13:42.30 | mcayer | in the db there is a table called demand |
| 13:42.55 | mcayer | i got three bool() used to determine a state of the demand |
| 13:43.06 | mcayer | calculating, answered and first |
| 13:43.45 | mcayer | at the beginning of the thread I need to set to True the field calculating so the demand won't be answered twice |
| 13:44.13 | mcayer | because a demand is answered with a thread started each minute by crond |
| 13:44.28 | mcayer | if calculating is set to true, the demand is not started again |
| 13:44.34 | mcayer | that's what I want to do |
| 13:44.57 | radix | so you're basically doing "obj = store.find(whatever); obj.attrs = blah; store.commit()", but you're not seeing those changes in the database? |
| 13:48.20 | mcayer | basically yes |
| 13:48.40 | mcayer | but I use an object Client, for which the demand is attached, to make my modification |
| 13:48.56 | mcayer | like that => client.demand.one().calculating = True |
| 13:49.06 | radix | well, I don't know why it wouldn't be getting written. maybe you can enable tracing (by calling storm.tracer.debug(True)) to see what SQL is being executed |
| 13:49.15 | mcayer | k |
| 13:49.59 | mcayer | will this output to stdout? |
| 13:50.10 | radix | mcayer: yes, but debug() also takes a 'stream' argument |
| 13:50.17 | mcayer | k |
| 13:50.21 | radix | err, actually, the default is stderr, not stdout |
| 13:50.26 | mcayer | k |
| 13:50.44 | mcayer | doesn't bother me as long as I can see the result on command line |
| 13:54.29 | therve | mcayer: how do you check it's not written? |
| 13:54.41 | mcayer | what do you mean? |
| 13:55.15 | therve | mcayer: how do you know it's not written to the database? |
| 13:55.24 | therve | how do you read it? |
| 13:55.30 | mcayer | I check with phpMyAdmin |
| 13:55.48 | therve | ok |
| 13:56.05 | mcayer | and the crond job start the script again it restart the same demand |
| 13:56.45 | mcayer | okay start the script with tracer |
| 13:56.49 | mcayer | the update is done |
| 13:56.58 | mcayer | but no change in phpMyAdmin |
| 13:57.16 | mcayer | however, in the thread the value of the field changed |
| 14:04.47 | mcayer | now the value changed in phpMyAdmin, but after the end of the execution |
| 14:04.49 | mcayer | :( |
| 14:13.27 | mcayer | any ideas? |
| 14:28.39 | radix | mcayer: are you sure that you're not just missing the change happening earlier? is the entire transaction fast? |
| 14:30.30 | mcayer | well, at the beginning of the script there is a few transactions done except the update, mostly select |
| 14:30.53 | mcayer | after there is a program executed which take a lot of CPU |
| 14:31.02 | mcayer | and then multiple insert at end end |
| 14:31.33 | mcayer | so the update command for the demand is done at the beginning before the execution of the program |
| 14:32.04 | mcayer | but wouldn't this change be visible? |
| 14:32.38 | radix | as long as you also commit right after it |
| 14:33.47 | mcayer | that's what I do |
| 14:34.03 | mcayer | want to see the beginning of my code for the thread? |
| 14:39.40 | *** join/#storm vvinet (n=vince@132.210.76.208) |
| 14:41.58 | radix | mcayer: sure |
| 14:42.07 | radix | mcayer: in fact, if you can provide a minimal script which reproduces the problem, that'd be excellent |
| 14:42.19 | radix | (with a schema to go along with it) |
| 14:42.41 | mcayer | the schema isn't done |
| 14:43.07 | mcayer | could you give me the link to paste my code please? |
| 14:43.55 | mcayer | i'm not doing this often ;) |
| 14:44.27 | radix | http://rafb.net/paste |
| 14:48.20 | mcayer | there you go : http://rafb.net/p/uPzuL821.html |
| 14:48.36 | mcayer | please ask if you have questions |
| 14:49.07 | therve | os.fork... hu |
| 14:49.56 | mcayer | yep to create a new process |
| 14:50.10 | therve | mcayer: you can't do that |
| 14:50.17 | mcayer | why that? |
| 14:51.00 | radix | fork! oh deary |
| 14:51.21 | radix | uh |
| 14:51.34 | radix | it looks like you're trying to use a queue to communicate between processes? that won't work |
| 14:51.44 | radix | wait |
| 14:51.49 | mcayer | nop I don't use it for that |
| 14:51.51 | radix | mcayer: fork() doesn't create threads, it creates processes |
| 14:51.58 | mcayer | err yes |
| 14:52.05 | mcayer | was wring |
| 14:52.06 | mcayer | wroing |
| 14:52.08 | mcayer | wrong |
| 14:52.32 | mcayer | my fault |
| 14:52.34 | radix | ok |
| 14:52.53 | mcayer | a little bit more explanations would help |
| 14:53.07 | therve | also python behavior in fork is not really something you can rely on |
| 14:53.20 | radix | basically, yeah, I recommend strongly against using fork to run non-trivial code |
| 14:53.45 | radix | that is, fork() without exec() immediately after it |
| 14:53.59 | mcayer | k |
| 14:54.08 | mcayer | here is the code that fill the queue : http://rafb.net/p/mANa7610.html |
| 14:54.17 | radix | I'm not really sure why you're using processes here at all |
| 14:55.50 | mcayer | i start a number of thread, say 3. those thread take one element from the queue, and create to process (fork()) to calculate it, after the process is done, the thread start a new one |
| 14:56.05 | mcayer | so you got at all time 3 process running in parallel |
| 14:57.14 | radix | mcayer: so, in this particular case, it looks like the main problem is that you're sharing storm objects between *processes* |
| 14:57.20 | radix | which is basically even crazier than sharing them between threads :-) |
| 14:57.47 | mcayer | even if I create a store object for each process? |
| 14:58.01 | mcayer | using the same database object of course |
| 14:58.05 | radix | mcayer: yes. it's fine that you're creating a store, but I'm talking about the actual object |
| 14:58.19 | radix | mcayer: "client.demand.one()", whatever that is |
| 14:58.30 | radix | that's coming in from the other process, not from the store |
| 14:58.31 | mcayer | :O |
| 14:58.35 | mcayer | i see |
| 14:58.44 | radix | if you're starting a process or thread, you need to get all your objects anew from the store object |
| 14:58.50 | mcayer | :) |
| 14:58.57 | mcayer | that's clearer |
| 14:59.21 | mcayer | like passing the id of the client instead of the client object instead |
| 15:00.45 | mcayer | I can still pass the database ? |
| 15:00.53 | mcayer | database object that is |
| 15:01.04 | radix | mcayer: yeah, I don't think there's any particular problem with that |
| 15:01.13 | mcayer | great |
| 15:01.20 | mcayer | I think this will help |
| 15:01.39 | mcayer | thanks for your help radix and therve |
| 15:01.44 | radix | sure |
| 15:01.46 | mcayer | really appreciated |
| 15:03.23 | mcayer | that leaves me some work to do, but in the end the result will work :) |
| 15:09.17 | *** join/#storm munin (n=munin@209.9.227.50) [NETSPLIT VICTIM] |
| 15:37.29 | mcayer | IT WORKED! |
| 15:37.33 | mcayer | yahoo! |
| 15:39.03 | radix | yay |
| 15:39.41 | mcayer | now the calculating field is correctly updated |
| 15:39.49 | mcayer | and it shows in phpMyAdmin! :D |
| 15:44.28 | mcayer | in french we have an expression saying two heads worth more than one. that's definitely the case here! thanks very much for your help radix and therve |
| 15:44.58 | therve | mcayer: no problem! |
| 15:45.37 | radix | yeah, we say the same in English :) |
| 15:45.50 | radix | "two heads are better than one" |
| 15:46.28 | mcayer | that was my own humble traduction ;) I'll try to remember the correct wording :) |
| 15:48.04 | radix | :) |
| 15:49.04 | therve | mcayer: "translation" :) |
| 15:49.12 | mcayer | oups |
| 15:49.23 | mcayer | I need to take more English courses ;) |
| 15:53.20 | radix | I had to look up "traduction", but it's definitely valid |
| 15:53.36 | radix | hmm, although my dictionary does say it's obsolete :-) |
| 15:53.42 | radix | <PROTECTED> |
| 15:54.06 | mcayer | lol |
| 15:54.23 | mcayer | I'll go to bed this evening more smarter than yesterday :) |
| 15:58.01 | radix | :-) |
| 16:12.04 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 16:45.36 | *** join/#storm gord (n=gord@5acaca61.bb.sky.com) |
| 17:13.58 | *** join/#storm shaunm (n=shaunm@proxyserver.wolfram.com) |
| 18:18.03 | *** join/#storm bigdog1 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 18:27.27 | mcayer | see ya! |
| 19:08.04 | *** join/#storm bigdog (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 20:54.06 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 23:41.50 | thumper | hi |
| 23:42.02 | thumper | I have another wonderful question :) |
| 23:42.11 | thumper | how do I return a constant in a result set? |
| 23:42.32 | thumper | especially if that constant is a value of a column in an ORM object |
| 23:42.57 | thumper | what I'm trying to do is to use storm to build me a select statement |
| 23:43.12 | jkakar | thumper: You want do something like result.values(MyClass.foo, MyClass.bar, MyClass.CONSTANT)? |
| 23:43.14 | thumper | which I want to extract and use in an "insert into foo (...) select ..." |
| 23:43.46 | thumper | jkakar: hmm... |
| 23:43.47 | jkakar | Well, Storm doesn't support the INSERT INTO ... SELECT .. syntax. |
| 23:43.56 | thumper | jkakar: I know, |
| 23:43.57 | jkakar | So, you're going to have to use Store.execute to run that. |
| 23:44.04 | thumper | jkakar: I was going to use _get_select() |
| 23:44.19 | jkakar | thumper: You code hosting guys just love the private API, huh? ;b |
| 23:44.38 | thumper | jkakar: I was hoping to use the storm expressions to save myself some sql ickiness |
| 23:44.55 | thumper | I could hand craft the whole select/subselect jobbie |
| 23:45.04 | jkakar | thumper: I see. Have you tried to use expressions to run INSERT INTO .. SELECT .., at all? |
| 23:45.34 | thumper | jkakar: well, jamesh told me yesterday that storm doesn't support insert into .. select |
| 23:45.42 | thumper | jkakar: so I was just hoping to build the select bit |
| 23:45.47 | jkakar | thumper: Ah, I see. |
| 23:45.50 | thumper | jkakar: and shove that into a string |
| 23:45.55 | jkakar | thumper: Right. |
| 23:45.57 | thumper | jkakar: to execute manually |
| 23:46.08 | thumper | however if it is going to be much trouble |
| 23:46.11 | thumper | I'll build it manually |
| 23:46.25 | thumper | jkakar: I added a comment to the bug report about using result sets |
| 23:46.30 | jkakar | thumper: Yeah, I noticed that. |
| 23:46.43 | jkakar | thumper: Definitely something Storm should support. |
| 23:46.48 | thumper | :) |
| 23:46.50 | thumper | definitely |
| 23:47.06 | jkakar | thumper: I'd also like to see Storm support the batch INSERT INTO syntax where you can specify values for many rows in the same query. |
| 23:47.10 | thumper | what I'm doing is trying to have a single sql call to keep a denormalised cache table up to date |
| 23:47.38 | thumper | I have a sql call ready |
| 23:47.41 | jkakar | thumper: It would be cool to be able to call store.add(obj1, obj2, obj3, obj4) and have it run one query if all the objects are for the same table. |
| 23:47.57 | thumper | yes |
| 23:48.52 | thumper | I think it may just be easier to handcraft the whole sql statement |
| 23:49.00 | jkakar | thumper: I think it may, yes. |
| 23:49.04 | thumper | poos |
| 23:49.08 | jkakar | Aye. |
| 23:49.31 | thumper | thanks anyway |