| 00:00.19 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 00:16.15 | *** join/#storm gord (n=gord@5ac328be.bb.sky.com) |
| 01:20.59 | *** join/#storm thumper (n=tim@canonical/launchpad/thumper) |
| 01:27.07 | *** join/#storm thumper (n=tim@canonical/launchpad/thumper) |
| 01:40.25 | *** join/#storm thumper (n=tim@canonical/launchpad/thumper) |
| 01:43.16 | jamesh | darkhorse`: the way Storm is written, you can treat a disconnection the same way as you would treat a serialisation failure. |
| 01:43.31 | jamesh | after calling store.rollback(), Storm will try to reconnect and you can retry your transaction |
| 02:14.48 | darkhorse` | yea, i am learning that now |
| 02:15.27 | darkhorse` | my idea of running activity on the connections doesnt really seem to work all that well |
| 02:17.25 | darkhorse` | jamesh, there are many elements of our system that are readonly from the db, do even thoes sections need the rollback action? |
| 02:17.35 | darkhorse` | or are there other ways to fire a reconnect |
| 02:18.27 | jamesh | darkhorse`: yes. If other apps are writing to the database, then you might not see those updates if you don't rollback |
| 02:18.40 | jamesh | even if you don't write to the DB personally |
| 02:19.40 | darkhorse` | ah, ok |
| 02:20.09 | darkhorse` | the big issue i seem to have is when the app sits idle (which probably wont be an issue in production) the db connections die silently |
| 02:20.16 | darkhorse` | then when you make a new call, the call fails |
| 02:21.32 | darkhorse` | i thought i had this solved by looping across my Stores every 30 min or so and doing a "Select 1;" |
| 02:21.53 | darkhorse` | however i quickly found that all of thoes stores end up using the same connection to the db |
| 02:21.59 | darkhorse` | and all of the other connections sit idle |
| 02:24.48 | darkhorse | jamesh, you ever encounter that issue? and if so any workarrounds? |
| 02:25.20 | jamesh | darkhorse: some databases will disconnect idle connections. |
| 02:25.36 | jamesh | darkhorse: you should be prepared to retry your transactions |
| 02:26.53 | darkhorse | hmm, yea... im trying to think how one would best implement that "retry" |
| 02:26.55 | jamesh | for some projects, we've done so using a simple decorator that adds retry behaviour to a function along with transaction start/end |
| 02:27.04 | darkhorse | some apps just error to the user saying there was a db issue |
| 02:27.24 | jamesh | and for zope based web apps, we just use the framework's transaction retry policy |
| 02:27.51 | darkhorse | this is a twisted project |
| 02:28.10 | darkhorse | so i guess on a fault i could just fire off the request again and keep a retry counter somewhere |
| 02:34.54 | jamesh | for some batch jobs, that can be good enough if the next time they are run they pick up where they left off previously |
| 02:36.23 | darkhorse | yea, but for simple pages like get a list of users and print them to the page |
| 02:36.25 | *** join/#storm shaunm (n=shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 02:36.38 | darkhorse | that sort of thing might be simpler to just let fault and have the user retry the query |
| 02:36.44 | jamesh | so for that sort of thing, you probably want a retry policy in place |
| 02:37.09 | jamesh | I'm not sure how you'd easily do that for if you're using the twisted-integration bits. |
| 02:37.26 | jamesh | we've done some twisted apps without using that branch |
| 02:37.49 | jamesh | where we use deferToThread() to start off a full transaction |
| 02:37.52 | darkhorse | well the nice thing is if you get an errback the errback can check a retry counter and then retry from the brgining |
| 02:37.57 | darkhorse | yea |
| 02:38.07 | jamesh | that function begins and commits or rolls back the transaction |
| 02:38.16 | jamesh | and retries if necessary |
| 02:38.23 | darkhorse | that was kind of what we used to do with adbapi |
| 02:38.34 | darkhorse | but it was such a pain to maintain all the calls via native sql |
| 02:40.28 | darkhorse | i am still trying to figure out where the mapping of Store -> raw connection happens |
| 02:41.12 | darkhorse | when i fire of a ton of requests i end up with about 20 Stores and 20 connections to mysql |
| 02:41.25 | darkhorse | but it seems like any store can use any connection |
| 02:41.38 | darkhorse | there does not appear to be a 1-1 relation |
| 02:44.21 | jamesh | each store has its own connection |
| 02:44.36 | jamesh | there is a 1-1 relationship |
| 02:44.49 | jamesh | or at least there is in mainline branch |
| 02:46.53 | darkhorse | hmmm |
| 02:47.53 | darkhorse | this confuses me then |
| 02:48.07 | darkhorse | ok in my running debug right now i have 24 stores in the twisted pool |
| 02:48.25 | jamesh | in the twisted-integration branch, there is a StorePool class, but the stores it hands out each have their own connection |
| 02:48.28 | darkhorse | they all have a database reference to the same <storm.databases.mysql.MySQL object at 0x0304EDF0> |
| 02:49.05 | jamesh | the Database object is not a Connection |
| 02:49.06 | darkhorse | yea, i thought that was how it was supposed to work |
| 02:49.16 | jamesh | it is a connection factory |
| 02:49.22 | darkhorse | oooh... |
| 02:49.36 | darkhorse | i am looking at the wrong thing... |
| 02:53.12 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 02:55.01 | *** join/#storm gord (n=gord@5ac328be.bb.sky.com) |
| 02:56.29 | darkhorse | the thing that is still perplexing me is that given the 24 stores, i loop across them and execute a query in each one, but if you look at the connection on the mysql side, only one had its timer reset (the idle timer) so that leads me to think all 24 stores executed their query against the same db connection |
| 03:21.49 | darkhorse | jamesh, ok, i fixed the issue, it was an error on my part when executing the queries |
| 03:21.54 | darkhorse | it does act as intended |
| 04:04.52 | *** join/#storm darkhorse` (n=admins@hill-d-029.resnet.purdue.edu) |
| 04:32.25 | *** join/#storm darkhorse (n=admins@hill-d-029.resnet.purdue.edu) |
| 05:19.51 | *** join/#storm munga (n=munga@81.194.35.9) |
| 06:53.25 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 07:27.14 | *** join/#storm oubiwann (n=oubiwann@97-119-85-2.omah.qwest.net) |
| 09:19.06 | *** join/#storm niemeyer (n=niemeyer@nat/canonical/x-e7cb51ca976044bd) |
| 09:29.20 | *** join/#storm goschtl (n=goschtl@p5B0BFE73.dip.t-dialin.net) |
| 11:33.51 | *** join/#storm vvinet (n=vince@modemcable101.133-202-24.mc.videotron.ca) |
| 11:34.04 | *** join/#storm gord (n=gord@5ac7ead0.bb.sky.com) |
| 12:16.37 | *** join/#storm vvinet (n=vince@132.210.76.197) |
| 12:32.15 | *** join/#storm gord_ (n=gord@5ac7eacf.bb.sky.com) |
| 15:24.59 | *** join/#storm gord_ (n=gord@5ac7eace.bb.sky.com) |
| 15:52.51 | *** join/#storm shaunm (n=shaunm@proxyserver.wolfram.com) |
| 16:01.48 | *** join/#storm andrea-bs (n=andrea-b@ubuntu/member/beeseek.developer.andrea-bs) |
| 16:41.47 | *** join/#storm niemeyer (n=niemeyer@nat/canonical/x-e4fa2096a4ad6ccb) |
| 18:07.09 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 19:09.53 | shaunm | it's done! |
| 19:16.15 | therve | yay! |
| 19:16.28 | therve | what is done? |
| 19:20.15 | shaunm | converting pulse from django to storm |
| 19:24.30 | radix | shaunm: congrats :) |
| 19:25.14 | *** join/#storm jukart (n=jukart@d91-128-122-97.cust.tele2.at) |
| 20:31.56 | darkhorse | i am about 60% done converting my project from SQLObject / adbapi to Storm :) |
| 22:04.06 | *** part/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 22:04.10 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 22:28.47 | *** join/#storm niemeyer (n=niemeyer@89.206.137.245) |
| 23:40.05 | *** join/#storm cody-somerville (n=cody-som@ubuntu/member/somerville32) |
| 23:41.58 | cody-somerville | Is it a good idea to catch SIGTERM and do a store.commit() before exiting? |
| 23:43.33 | mzz | I don't think there's really a general answer to that |
| 23:43.46 | mzz | I'd usually expect the answer to be no, but there are always exceptions to that |
| 23:44.09 | mzz | (think about it, what if your app eats a sigterm halfway through a transaction that involves more than one query modifying the db?) |
| 23:49.04 | cody-somerville | nods. |
| 23:53.17 | *** join/#storm thumper (n=tim@canonical/launchpad/thumper) |
| 23:57.56 | *** join/#storm thumper (n=tim@125-236-193-95.adsl.xtra.co.nz) |