| 01:00.27 | *** join/#storm Cube-ness (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 01:00.58 | Cube-ness | ok.. what did i miss.. how do you select only specific column with Store.find()? |
| 01:03.21 | Cube-ness | nm.. i am being dumb |
| 01:04.13 | radix | Cube-ness: was the answer you were looking for .values? |
| 01:11.31 | leif | I've got kind of a related question. Is there a way to load objects and individual columns (e.g. from a different table) with one query? |
| 01:47.19 | radix | hmm |
| 01:47.26 | radix | there's a way to load multiple objects at once |
| 01:47.40 | radix | but I don't think there's a way to load objects from one table and simple values from another |
| 01:53.11 | leif | I think it would be nice if the interface for retrieving whole objects and individual columns was unified, so I could do store.find(Foo.name) or store.find((Foo, Bar.name)) or whatever. But I have very little clue about how this works internally, so |
| 01:53.20 | leif | I don't know if that's actually a good idea or not. |
| 02:54.03 | aa_ | well, with a bit of massaging, all the sqlite tests pass on windows |
| 02:57.36 | aa_ | is there any reason why TestHelper.tearDown is called before StoreTest.tearDown, and the same in the databases test? because for them to pass in windows, drop_database should be implemented, and the order of tearing things down reversed so that the connection/store are closed before the temp directory is deleted |
| 02:57.47 | aa_ | (all sqlite) |
| 03:00.00 | aa_ | (otherwise windows complains that a file is being used when trying to delete) |
| 03:01.56 | aa_ | 3 other store tests which create extra stores need explicit patching to close the extra store |
| 04:35.31 | *** join/#storm pyCube_ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 04:40.29 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 05:00.49 | radix | leif: that's a pretty interesting idea |
| 05:01.11 | radix | leif: store.find((Foo.a, Foo.b)) even seems like it might be a nicer syntax than store.find(Foo).values(Foo.a, Foo.b) |
| 05:53.23 | *** join/#storm jml_ (n=jml@ppp121-44-221-92.lns1.hba1.internode.on.net) |
| 06:50.34 | *** join/#storm dobee (n=dobeee@194.183.146.186) |
| 06:58.48 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 06:58.53 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 07:09.53 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 07:26.02 | *** join/#storm pyCube_ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 07:26.30 | *** join/#storm Cube-ness (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 08:08.01 | *** join/#storm dobee (n=dobeee@194.183.146.186) |
| 08:13.42 | *** join/#storm mcella (n=michele@ip-176-22.sn2.eutelia.it) |
| 08:56.05 | *** join/#storm dobee (n=dobeee@194.183.146.186) |
| 09:29.27 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 09:55.47 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 09:56.13 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 11:07.13 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 11:07.15 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 11:50.38 | *** join/#storm rafael (n=rafael@moinmoin/fan/rafael) |
| 12:27.52 | *** join/#storm niemeyer (n=niemeyer@200-103-134-216.ctame705.dsl.brasiltelecom.net.br) |
| 12:46.45 | richardb | Two quick questions: |
| 12:47.13 | richardb | 1. What's the policy for editing the Manual. Is it okay if I flesh out the section on "Working with multiple threads" |
| 12:47.21 | *** join/#storm rafael (n=rafael@moinmoin/fan/rafael) |
| 12:47.29 | richardb | since I've just gone through the process of working out how to do that myself. |
| 12:47.52 | richardb | And: 2. What license is the project's documentation under? |
| 12:48.07 | aa_ | richardb: perhaps make a wiki page which can later be merged into the manual? |
| 12:48.55 | richardb | aa_: That's a good idea - I'll do that. Then someone who knows can merge my text in if they think I've got it right. |
| 13:26.37 | niemeyer | richardb: You can hack on the Manual freely at this point, even more if it's an empty section. |
| 13:27.06 | niemeyer | richardb: We'll get notifications of changes, and will help you with any issues |
| 13:28.24 | niemeyer | richardb: We haven't discussed the manual license, but we'll most certainly put it under an open one. |
| 14:02.07 | mcella | niemeyer: http://www.aminus.org/blogs/index.php/fumanchu/2007/08/18/storm_sqlalchemy_and_geniusql |
| 14:02.09 | mcella | see this? |
| 14:02.43 | mcella | storm does perform pretty well, except in "Step 4" |
| 14:02.51 | mcella | s/see/seen/ |
| 14:06.08 | niemeyer | mcella: Yeah, I've seen it |
| 14:06.20 | niemeyer | mcella: I had already done some basic tests on my own as well |
| 14:06.56 | niemeyer | mcella: That step 4 is one thing we'll work on at some point in the future |
| 14:07.03 | mcella | niemeyer: cool :-) |
| 14:07.13 | niemeyer | mcella: I don't think he got the actual reason why it's slower, though |
| 14:07.37 | niemeyer | mcella: It's certainly not about expressions |
| 14:10.08 | mcella | great, I'm happy to hear that you've |
| 14:10.16 | mcella | seen it ;-) |
| 14:10.37 | mcella | anyway, storm does pretty well in every other test |
| 14:10.42 | mcella | congrats ;-) |
| 14:11.59 | niemeyer | mcella: One of the reasons it could happen, for instance, is because Storm won't keep strong references to the objects, so they may get deallocated after you do something like list(store.find(...)). |
| 14:12.08 | niemeyer | mcella: Thanks :) |
| 14:12.29 | niemeyer | mcella: (and don't use the result of list(store.find()), that is) |
| 14:20.20 | mcella | good to know, tnx again |
| 14:26.03 | richardb | Hmm, test_wb_create_database() doesn't ever get run for SQLite, because the SQLiteFileTest inherits from SQLiteMemoryTest, which disables the test |
| 14:26.21 | richardb | I've got a small patch to fix it - should I simply create a bug and attach the patch? |
| 14:26.57 | richardb | The procedure outlined in https://storm.canonical.com/DevelopmentProcedure seems a bit heavyweight for small changes like that. |
| 14:35.35 | niemeyer | richardb: Yes, that'd be fine in that case (and appreciated!) |
| 14:37.21 | richardb | Done. :) |
| 14:37.21 | niemeyer | richardb: Thanks! |
| 15:05.53 | mcella | niemeyer: I've seen that Robert |
| 15:06.00 | mcella | uses list(store |
| 15:06.03 | mcella | or better |
| 15:06.12 | mcella | len(list(store.*)) |
| 15:06.28 | mcella | to compute the resultset length |
| 15:06.34 | mcella | http://www.aminus.org/rbre/python/ormcompare/benchmark_storm.py |
| 15:07.27 | mcella | maybe it could be handy (not only for this benchmark) |
| 15:07.51 | mcella | to add a __len__ method to the ResultSet class |
| 15:11.29 | niemeyer | mcella: Nope.. he was actually testing the object load time, so he was explicitly not wanting to just count them |
| 15:11.30 | niemeyer | mcella: I think |
| 15:11.30 | niemeyer | mcella: We can't add __len__ because it has side effects |
| 15:11.45 | mcella | D'oh, you're right |
| 15:11.55 | mcella | __len__ side effects == hitting the db right? |
| 15:24.49 | richardb | Urk. I've just read http://www.sqlite.org/cvstrac/wiki?p=CorruptionFollowingBusyError |
| 15:25.20 | richardb | It would probably be good if Storm automatically rolled back the transaction if a SQLITE_BUSY error was raised by something other than a COMMIT |
| 15:25.33 | richardb | To avoid getting corruption due to the bug mentioned on that page. |
| 15:29.37 | richardb | Looks like that bug exists in the version of sqlite3 in feisty, at least. |
| 15:29.43 | richardb | Though not in gutsy. :) |
| 15:32.26 | niemeyer | mcella: No.. Python sometimes calls __len__ internally to get size hints from the object |
| 15:32.54 | mcella | niemeyer: uh, didn't know that, tnx for pointing it out ;-) |
| 15:32.58 | niemeyer | mcella: That would mean doing a very expensive operation for "optimization" :-) |
| 15:33.04 | mcella | yep, indeed :-D |
| 15:33.21 | richardb | I also wonder if it would be a good idea to introduce a storm.exceptions.DatabaseLockedError, to represent the special case of OperationalError('database is locked') being raised by a commit() call, since in that case (and only that case, I think) sqlite leaves the transaction open, instead of rolling it back. |
| 15:34.16 | niemeyer | richardb: I don't think so.. we should try to make SQLite behave well, similar to other databases |
| 15:34.32 | richardb | What does "behave well" mean? |
| 15:34.49 | richardb | ie, What state should the database be left in if commit() raises an exception? |
| 15:35.26 | richardb | I've been trying to look up what other databases do if part of a transaction fails, with little success. |
| 15:35.46 | niemeyer | richardb: Usually, the transaction is kept in an inconsistent state, which requires a rollback |
| 15:35.52 | richardb | Right. |
| 15:36.12 | richardb | Which is what SQLite does in all but the case of OperationalError('database is locked') |
| 15:36.22 | richardb | (ie, SQLITE_BUSY being returned) |
| 15:36.50 | richardb | However, it seems unhelpful to discard the transaction when it can validly be retried. |
| 15:37.04 | richardb | Or maybe that's a price worth paying for consistent behaviour. |
| 15:37.23 | richardb | We could easily detect that case and issue an explicit ROLLBACK. |
| 15:37.47 | richardb | (After retrying for the length of the timeout.) |
| 15:38.50 | richardb | I think that could be most helpful, actually. And easiest to document (which is a good sign). |
| 15:38.50 | niemeyer | richardb: Why should we be rolling it back, instead of leaving it up to the user to do so? |
| 15:39.11 | niemeyer | richardb: If an exception is raised, it has to move up the stack |
| 15:39.26 | niemeyer | richardb: We can't just rollback and pretend that nothing happened, right? |
| 15:39.45 | richardb | No, but we can rollback and pass up the exception. |
| 15:40.17 | niemeyer | richardb: So that's what I meant.. in postgres and MySQL, when an exception is raised, the user has to rollback the transaction |
| 15:40.23 | richardb | Oh, I see. |
| 15:40.34 | richardb | In SQLite, in most cases the rollback is done automatically. |
| 15:40.38 | niemeyer | richardb: And that's important because we have to rollback the Store to reflect that things went wrong, and we need a new transaction. |
| 15:40.45 | richardb | (But an explicit one won't cause a problem.) |
| 15:41.24 | richardb | In SQLite though, in the specific case of OperationalError('database is locked'), a rollback isn't required. |
| 15:41.41 | richardb | Though is required for an OperationalError() with different text. |
| 15:42.02 | richardb | Or for any OperationalError from anything other than a commit(). |
| 15:42.17 | richardb | So, I thought that changing such exceptions to a different type might be useful for the user. |
| 15:42.29 | niemeyer | richardb: I don't think this is the case.. |
| 15:42.33 | richardb | So that they can catch DatabaseLocked |
| 15:42.39 | richardb | Which bit? |
| 15:42.57 | niemeyer | richardb: In some cases, when an operation fails in sqlite3, it allows the transaction to go on, as if that transaction hadn't been executed |
| 15:43.05 | niemeyer | richardb: Which is different from rolling back the transaction |
| 15:43.15 | niemeyer | Erm.. sorry |
| 15:43.21 | niemeyer | richardb: In some cases, when an operation fails in sqlite3, it allows the transaction to go on, as if the failing statement hadn't been executed |
| 15:43.41 | niemeyer | E.g. |
| 15:43.46 | niemeyer | (flood ahead, sorry) |
| 15:43.48 | niemeyer | sqlite> begin; |
| 15:43.48 | niemeyer | sqlite> insert into foo values (1); |
| 15:43.48 | niemeyer | SQL error: PRIMARY KEY must be unique |
| 15:43.48 | niemeyer | sqlite> insert into foo values (2); |
| 15:43.48 | niemeyer | sqlite> insert into foo values (1); |
| 15:43.49 | niemeyer | SQL error: PRIMARY KEY must be unique |
| 15:43.51 | niemeyer | sqlite> insert into foo values (3); |
| 15:43.53 | niemeyer | sqlite> rollback; |
| 15:43.55 | niemeyer | sqlite> select * from foo; |
| 15:43.57 | niemeyer | 1 |
| 15:44.30 | niemeyer | Notice that the value (2) isn't present after the rollback. |
| 15:44.30 | richardb | Yes, that's true. But I don't see that that's what I'm talking about. |
| 15:44.46 | niemeyer | Now.. |
| 15:45.09 | niemeyer | sqlite> begin; |
| 15:45.09 | niemeyer | sqlite> insert into foo values (2); |
| 15:45.09 | niemeyer | sqlite> insert into foo values (1); |
| 15:45.09 | niemeyer | SQL error: PRIMARY KEY must be unique |
| 15:45.09 | niemeyer | sqlite> commit; |
| 15:45.09 | niemeyer | sqlite> select * from foo; |
| 15:45.09 | niemeyer | 1 |
| 15:45.09 | niemeyer | 2 |
| 15:45.09 | richardb | If the documented behaviour of storm is "when you get an exception, you're in an inconsistent state, so you have to rollback", that covers that case. |
| 15:45.16 | niemeyer | richardb: Actually, it's more like |
| 15:46.01 | niemeyer | When you get an error, whatever happens in the dbapi module is the state you are in. We're trying not to introduce magic behavior when possible. |
| 15:46.27 | richardb | Okay. Hmm. |
| 15:47.05 | niemeyer | Which is to say, I don't see a reason to modify the database behavior regarding transactions |
| 15:47.14 | richardb | So, if users want to not have to worry about what the dbapi module does, they can just call rollback. |
| 15:48.52 | richardb | So, my two suggestions were: |
| 15:48.52 | niemeyer | SQLite is in fact one case where we try really hard to do so.. we try to workaround the weird transactional behavior of pysqlite2 to get closer to what sqlite3 actually exposes. |
| 15:48.52 | richardb | Indeed. |
| 15:48.53 | richardb | 1. change the exception type to make it easier to detect SQLITE_BUSY return codes |
| 15:49.14 | richardb | ie, undo the work that pysqlite2 does which merges that code into OperationalError, which I believe can be produced by other return codes. |
| 15:49.56 | radix | I wonder if it includes the error code as an attribute of its OperationalError? |
| 15:49.57 | richardb | This isn't particularly important, but I thought it might be a useful help to users. |
| 15:50.08 | richardb | radix: probably not - that would be helpful. |
| 15:50.12 | radix | heh heh :) |
| 15:50.12 | richardb | I'll check |
| 15:51.17 | niemeyer | I agree that pysqlite2 could do a better job at providing sqlite3 related information in the exception, but we can't just provide another exception.. as much as we want to respect sqlite3, we also want to respect dbapi, and don't want to introduce compromises we'll have to maintain forever when the lower api isn't helping us. |
| 15:52.26 | radix | niemeyer: raising a subclass of OperationalError shouldn't be a problem |
| 15:52.27 | niemeyer | radix: That's true |
| 15:52.27 | niemeyer | At the same time.. the same check we'd have to introduce might be done at user land. |
| 15:52.30 | niemeyer | It's ugly, of course. |
| 15:52.42 | niemeyer | Because it's comparing with a string, which could change. |
| 15:52.52 | niemeyer | and that's precisely the reason why I'm trying to avoid it ;) |
| 15:53.50 | radix | yeah |
| 15:53.50 | richardb | OperationalError definitely has no useful attributes attached to it, for what it's worth. |
| 15:54.20 | richardb | The other suggestion I had was to perform the rollback automatically if the SQLITE_BUSY situation happened for anything other than a commit() |
| 15:54.21 | niemeyer | Yeah.. it'd be good to have a sqlite_error to any exceptions coming from sqlite |
| 15:54.52 | richardb | This suggestion is simply to ensure that the bug in sqlite versions 3.3.0 to 3.3.17 doesn't bite our users. |
| 15:55.18 | richardb | Since that bug is quite severe (ie, dataloss), and users are quite likely not to know about it. |
| 15:55.48 | richardb | Alternatively, it could just be mentioned in our documentation. :) |
| 15:57.45 | niemeyer | richardb: Definitely +1 for the later, in any case (IOW, whatever we implement) |
| 15:58.00 | richardb | yep |
| 15:58.03 | niemeyer | richardb: Can you describe the data loss situation? |
| 15:58.35 | richardb | http://www.sqlite.org/cvstrac/wiki?p=CorruptionFollowingBusyError has the details, but briefly... |
| 15:58.57 | richardb | The problem appears when an SQLITE_BUSY error occurs in the middle of an explicit transaction |
| 15:59.22 | richardb | and the user then attempts to COMMIT |
| 15:59.46 | richardb | Or any other operation other than ROLLBACK, actually. |
| 16:00.40 | richardb | the problem is that the internal cache in sqlite gets into an inconsistent state, so if committed to disk, will generally cause corruption. |
| 16:01.00 | niemeyer | Ouch |
| 16:01.08 | richardb | However, it's probably fairly rare, unless you have large transactions: |
| 16:01.12 | richardb | " The only time you will ever get an SQLITE_BUSY error for a non-COMMIT command within a transaction is if your database change has become so big that it will no longer fit entirely in cache. If you increase the cache size from the default 2000 pages, you might reduce the chance of this ever happening. " |
| 16:06.24 | *** join/#storm EnTeQuAk (n=EnTeQuAk@p57B9BCB6.dip0.t-ipconnect.de) |
| 16:07.23 | richardb | The disadvantage of automatically rolling back is that, if the user _knows_ that they're not using a version of sqlite with the bug, they don't get the option to continue with the transaction. |
| 16:08.08 | richardb | I suppose it's a question of how much cotton wool Storm wants to wrap the user in. |
| 16:08.08 | niemeyer | Right.. I think that it's unlikely that any workarounds that we find won't present disadvantages regarding magical behavior |
| 16:08.35 | niemeyer | Anyway.. I'll step out to have lunch and be back in a bit |
| 16:09.44 | richardb | Considering the discussion, my view is that we shouldn't do any automatic rollback magic or suchlike, but should add a subclass of OperationalError which is raised in the case of SQLite raising an OperationalError('database is locked') from a commit() transaction. |
| 16:10.01 | richardb | Then the documentation can advise users that, if they get this subclass of OperationalError, they are safe to retry the COMMIT |
| 16:11.14 | richardb | If the user gets any other OperationalError, the documentation can advise that what they can safely do depends on the version of sqlite they have, and they should be aware of the bug. |
| 16:11.25 | richardb | But that it is always safe to ROLLBACK in this situation. |
| 16:31.49 | niemeyer | richardb: The simple fact that they should be aware of the bug, makes it sound like we shouldn't be handling it in Storm. |
| 16:32.05 | niemeyer | richardb: If they are aware about it, and want to handle it, they can simple catch OperationError and handle it |
| 16:33.36 | richardb | So, we're agreed that a magic ROLLBACK shouldn't happen. |
| 16:33.45 | niemeyer | Cool |
| 16:34.17 | richardb | But do you mean you think that a subclass of OperationalError for the case where we know it's safe is also undesirable |
| 16:34.30 | richardb | ? (Because it's not necessary) |
| 16:34.53 | richardb | I currently think it would be a useful help, but don't feel very strongly about it. |
| 16:35.18 | richardb | I just pushed an implementation of it to my sqlite-locked-recovery branch on launchpad though - very easy to do it. |
| 16:35.54 | richardb | I'll get on with documenting all this in the manual shortly. |
| 16:37.40 | richardb | (Note that there's currently a bug in trunk with this - once a commit() raises an OperationalError('database is locked'), the store gives an error for all subsequent operations |
| 16:37.51 | richardb | Bug report and fix at https://bugs.launchpad.net/storm/+bug/136771 |
| 16:37.52 | ubot3 | Malone bug 136771 in storm "If SQLite commit() fails, transaction status gets incorrectly cleared" [Undecided,New] |
| 16:38.40 | richardb | I'm ignoring this bug for the purposes of the manual though, since I imagine it'll go away soon. |
| 16:41.28 | niemeyer | richardb: Yes, I think even the subclass would be undesirable.. if someone is aware about the bug, and want to workaround it, they can very easily catch the OperationError exception and run is_busy_error() on it, or whatever |
| 16:42.36 | richardb | Fair enough. I'll update the documentation accordingly. |
| 16:42.56 | niemeyer | richardb: Thanks a lot for that |
| 16:43.16 | richardb | least I can do after taking up so much of your time with this issue. ;-) |
| 16:57.17 | *** join/#storm dobee (n=dobeee@81-223-53-162.dornbirn.xdsl-line.inode.at) |
| 17:41.18 | richardb | Manual updated - hopefully that all makes sense. |
| 17:48.34 | niemeyer | richardb: Hmm |
| 17:48.57 | niemeyer | richardb: I think there's a detail which isn't being taken in consideration |
| 17:49.18 | niemeyer | richardb: The normal "database locked" error is returned to the user after a timeout has already passed |
| 17:49.38 | niemeyer | richardb: So I'm not sure if it makes sense for the user to just try to commit again |
| 17:49.43 | niemeyer | richardb: There's most probably something else wrong |
| 18:02.05 | richardb | The usual case that I've had where "database locked" is returned, is where there are two connections trying to perform modifications |
| 18:02.20 | richardb | In that situation, the user can stop one of those connections, and retry. |
| 18:02.59 | richardb | The commit on each one can't happen until the other times out. |
| 18:03.35 | richardb | (Actually, until the other is rolled back) |
| 18:04.36 | richardb | So, the user can handle this condition by picking one to roll back, and retrying the other. |
| 18:05.37 | niemeyer | richardb: I've never seen this being done manually |
| 18:06.06 | richardb | I've never seen it being handled... |
| 18:07.18 | richardb | How can it be done anything other than manually? |
| 18:07.42 | richardb | (Other than just giving up on the changes.) |
| 18:08.41 | richardb | I suppose closing the connection that gave "database locked", sleeping a random number of seconds (to avoid repeating the same contention), and then retrying the modification would work. |
| 18:08.51 | richardb | For some value of "work". |
| 18:08.59 | niemeyer | richardb: Precisely.. the "database locked" error will go up, and will naturally rollback because it's an exception |
| 18:09.16 | niemeyer | richardb: Then the second thread will commit because it's still inside the timeout, thus retrying |
| 18:09.31 | richardb | yep |
| 18:09.46 | niemeyer | richardb: So the user doesn't actually have to do anything, and so we don't have to document it as "you should retry" |
| 18:10.11 | richardb | Okay. You just handle it as you would any other exception. |
| 18:10.12 | niemeyer | richardb: The timeout feature is already done by Storm and sqlite |
| 18:10.16 | niemeyer | Right |
| 18:10.23 | richardb | Oh, I see. |
| 18:10.38 | niemeyer | (by Storm on older revisions, because there was a bug, and by sqlite on newer ones) |
| 18:10.42 | richardb | Yes, I wasn't meaning to imply in the docs that the user should just blindly retry... |
| 18:11.26 | niemeyer | richardb: I think one can understand that in "However, in the specific case of a call which returns an OperationalError with the message "database is locked", the Store is left in a well defined state, such that you can simply retry the commit() call" |
| 18:11.47 | richardb | However, the user _can_, if they want, handle the exception instead of passing it up. |
| 18:12.06 | richardb | Actually, no they can't - because if they don't rollback, the other connection will also timeout. |
| 18:12.06 | niemeyer | Right |
| 18:12.18 | niemeyer | Well "handle" as in "do whatever they want" :) |
| 18:12.43 | richardb | ie, they'll have to be very clever about how they handle it, and the simplest approach is probably to let it pass up, but they might be able to do something more cunning. |
| 18:14.22 | richardb | Incidentally, currently if sqlite is recent enough that it _does_ handle the timeout itself, I believe the timeout will be doubled.... |
| 18:14.45 | richardb | because Storm doesn't start its timeout counter until after the first time that "database is locked" is thrown. |
| 18:15.47 | niemeyer | It won't |
| 18:15.59 | niemeyer | Well.. at least I remember thinking about it |
| 18:16.02 | niemeyer | Let me look at the logic |
| 18:17.07 | niemeyer | richardb: Crap! |
| 18:17.43 | richardb | I put the fix that I think is needed in https://code.launchpad.net/~richardboulton/storm/sqlite-locked-recovery |
| 18:17.47 | richardb | revision 175 |
| 18:18.04 | niemeyer | richardb: I probably "optimized" it by moving the _started to the except, and broke it :( |
| 18:18.11 | richardb | :) |
| 18:18.17 | niemeyer | richardb: Thanks for pointing it out |
| 18:18.48 | niemeyer | This should be tested, to avoid breaking again.. but I don't know how to do it, since it depends on sqlite behavior, which is either one or the other |
| 18:19.15 | niemeyer | Maybe mocking sqlite itself.. but sounds bad at first sight |
| 18:20.06 | richardb | Ouch, indeed. |
| 18:20.24 | richardb | Anyway - I've got my dinner ready to eat, so back later. |
| 18:20.32 | niemeyer | richardb: Enjoy it! |
| 18:20.38 | niemeyer | richardb: And thanks for your help |
| 18:21.07 | *** join/#storm BleSS (n=BleSS@13.Red-88-27-146.staticIP.rima-tde.net) |
| 18:21.50 | *** join/#storm pyCube_ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 18:22.07 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 18:33.37 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 18:33.49 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 18:34.58 | BleSS | hi! |
| 18:35.18 | BleSS | is there a shard implementation on Storm? |
| 18:35.39 | radix | nope |
| 18:39.26 | *** part/#storm BleSS (n=BleSS@13.Red-88-27-146.staticIP.rima-tde.net) |
| 18:52.57 | niemeyer | Dude |
| 18:58.58 | niemeyer | It really depends on what he thinks "shard" means |
| 19:02.31 | niemeyer | IOW, by the definition of wikipedia, it certainly supports sharding. |
| 19:03.57 | *** join/#storm BleSS (n=BleSS@13.Red-88-27-146.staticIP.rima-tde.net) |
| 19:05.08 | BleSS | with shard I mean if is possible that certain records are in certain database. E.g. users 1 to 1,000,000 , or news/articles/information stored by year. |
| 19:23.23 | *** part/#storm BleSS (n=BleSS@13.Red-88-27-146.staticIP.rima-tde.net) |
| 19:28.27 | niemeyer | BleSS: Bye.. |
| 19:29.43 | radix | well, I made a distinction between supporting sharding and having a sharding implementation |
| 19:30.30 | niemeyer | radix: Ok, cool |
| 19:31.07 | niemeyer | He seems to be pretty harsh, so it's not worth even trying to question or explain the distinction.. |
| 19:40.45 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 19:40.55 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 20:56.42 | *** join/#storm dobee (n=dobeee@81-223-53-162.dornbirn.xdsl-line.inode.at) |
| 21:20.34 | *** join/#storm dobee (n=dobeee@81-223-53-162.dornbirn.xdsl-line.inode.at) |
| 21:41.39 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 21:41.39 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 22:22.14 | tjs_ | morning |
| 22:37.34 | *** join/#storm pyCube (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 22:37.37 | *** join/#storm pyCube__ (n=jmayfiel@ip70-162-117-219.ph.ph.cox.net) |
| 22:40.02 | niemeyer | tjs: Morning! |
| 22:56.47 | *** join/#storm jml (n=jml@ppp121-44-221-92.lns1.hba1.internode.on.net) |
| 22:57.48 | *** join/#storm jml_ (n=jml@ppp121-44-221-92.lns1.hba1.internode.on.net) |
| 23:25.19 | tjs | branches for storm are getting a little confusing |
| 23:25.33 | tjs | there are two 'trunk' branches for a start ;) |
| 23:25.48 | tjs | and merged ones still marked as new |
| 23:52.50 | *** join/#storm wallflower (n=wallflow@216.113.246.205) |