| 00:43.51 | jkakar | gabriel_: Am back in case you have questions. |
| 01:13.17 | *** join/#storm artista_frustrad (n=artista_@201-24-206-206.ctame704.dsl.brasiltelecom.net.br) |
| 01:21.50 | *** join/#storm WebMaven (n=webmaven@68.35.24.218) |
| 01:32.49 | *** join/#storm stub (n=stub@ppp-58-8-17-47.revip2.asianet.co.th) |
| 01:55.37 | *** join/#storm shaunm (n=shaunm@c-98-212-133-244.hsd1.il.comcast.net) |
| 02:15.50 | *** join/#storm fcorrea (n=fcorrea@189-54-36-208-nd.cpe.vivax.com.br) |
| 02:20.46 | *** join/#storm mwhudson (n=mwh@canonical/launchpad/mwhudson) |
| 02:21.10 | mwhudson | should i be scared of iterating over a result set and committing transactions while i do it? |
| 02:25.52 | jkakar | mwhudson: Hmm... good question. |
| 02:26.53 | jkakar | mwhudson: I don't think anything obviously bad will happen, since we do this in Landscape and haven't had issue. |
| 02:27.04 | jkakar | mwhudson: I do wonder what version of the data you'll be retrieving from the result set, though. |
| 02:27.48 | mwhudson | jkakar: i don't think it matters that much in my case |
| 02:28.02 | jkakar | mwhudson: I don't think you need to worry then. |
| 02:28.12 | mwhudson | jkakar: cool, thanks |
| 02:28.24 | jkakar | mwhudson: We do this in cases where committing every N objects results in dramatically better performance. |
| 02:28.55 | jkakar | mwhudson: 40 or 50 objects changed/commit seems to be a magic range, with default PostgreSQL settings. |
| 02:29.29 | mwhudson | jkakar: hmm |
| 02:29.44 | mwhudson | jkakar: in my case i need to do things on disk in relation to rows i insert |
| 02:30.07 | mwhudson | mangling the disk before the transaction is committed sounds like a really bad idea |
| 02:30.45 | mwhudson | (the context is a script to create official source branches for all packages in lucid after karmic is released) |
| 02:31.15 | jkakar | Ah, ha, I see. |
| 02:31.35 | jkakar | mwhudson: Wouldn't you ideally want to mangle the disk, then if all is well, commit, otherwise rollback? |
| 02:31.36 | mwhudson | has spent the last 2.5 days being extremely paranoid |
| 02:31.48 | jkakar | mwhudson: Is there an equivalent of sync that you can call from Python? :) |
| 02:31.53 | mwhudson | jkakar: no, because the mangling is not especially reversible |
| 02:32.04 | jkakar | mwhudson: Ah ha, that's tricky. |
| 02:32.38 | jkakar | mwhudson: I guess you could mangle a copy, make sure everything is fine and then copy it into place... but maybe the performance characteristics of that kind of approach are unnacceptable. |
| 02:32.45 | mwhudson | jkakar: well, if the power goes and the ups attached to crowberry fails, then there's always restoring from tape :) |
| 02:32.47 | jkakar | I'm guessing you'll have really a lot of source branches. |
| 02:33.02 | mwhudson | jkakar: right, copying rather than moving would be preferable somehow |
| 02:33.05 | mwhudson | jkakar: about 16000 |
| 02:33.13 | mwhudson | jkakar: so performance is certainly an issue |
| 02:33.13 | jkakar | mwhudson: You need geographically distributed transactions! |
| 02:33.18 | mwhudson | as is disk space |
| 02:33.22 | jkakar | Right. |
| 02:33.45 | jkakar | Surely disk space is cheap? |
| 02:34.28 | jkakar | I need to get dinner before my stomach incinerates itself, back in a bit. |
| 02:37.24 | mwhudson | jkakar: spoken like a true developer |
| 03:02.40 | jkakar | :) |
| 03:26.19 | *** join/#storm srichter (n=quassel@c-76-19-135-180.hsd1.ma.comcast.net) |
| 04:31.06 | gabriel_ | jkakar: not sure if you're still around, but it seems as though I (finally) got the setup sorted out, although a few of the tests are still failing |
| 04:33.58 | gabriel_ | jkakar: I also wrote up instructions for the testing setup process so i can remember how I did it. Shall I post them on the wiki to make it a bit easier for anyone else who wants to help out with the project? |
| 04:36.21 | gabriel_ | i've posted the results of the test suite at http://pastebin.ubuntu.com/288302/ if you want to take a look |
| 04:39.15 | gabriel_ | a bunch of the errors say the problem is "no password supplied", however there is no password set on the DB, so this shouldn't be a problem. Most of the other errors are the "broken pipe"s. I'm not sure what to make of these... |
| 04:42.42 | jkakar | gabriel_: Heya! |
| 04:42.57 | jkakar | gabriel_: Those errors, I think, mean you're missing some PostgreSQL specific configuration. |
| 04:44.11 | jkakar | gabriel_: If you could paste your instructions into a bug report that'd be really helpful. |
| 04:44.27 | jkakar | gabriel_: I think they should go in the README file, ideally, not the wiki. |
| 04:45.16 | jkakar | gabriel_: Give me a sec and I'll figure out what special incantations you need for those tests to pass with PostgreSQL. |
| 04:45.35 | gabriel_ | jkakar: ok, great |
| 04:59.30 | jkakar | gabriel_: Okay, I think you need to make sure you have a line like this in /etc/postgresql/8.3/main/pg_hba.conf: |
| 04:59.34 | jkakar | host all all 127.0.0.1/32 ident |
| 04:59.51 | jkakar | In my case, with PostgreSQL 8.4, that involved changing 'md5' to 'ident'. |
| 05:00.14 | gabriel_ | ...checking... |
| 05:09.18 | jkakar | gabriel_: Hm, I think that wasn't enough. |
| 05:09.31 | jkakar | gabriel_: I'm trying to set itup on a fresh machine and I've just seen test failures. :/ |
| 05:10.49 | gabriel_ | i'll let you know when the tests finish (i guess having to wait for tests to finish is the downside of a massive test suite :) |
| 05:12.10 | jkakar | gabriel_: I've just realized I can trigger the failure faster with: STORM_POSTGRES_HOST_URI=postgres://localhost/storm_test ./test |
| 05:14.01 | jkakar | gabriel_: If you change 'ident' to 'trust' the tests pass. |
| 05:14.04 | gabriel_ | so altering that config file doesn't seem to have changed anything in the test output |
| 05:14.14 | gabriel_ | ok |
| 05:14.19 | jkakar | gabriel_: Also, restart postgres: /etc/init.d/postgresql-8.3 restart |
| 05:14.32 | jkakar | runs the entire test suite to make sure everything works |
| 05:15.00 | *** join/#storm artista_frustrad (n=artista_@201-24-207-135.ctame704.dsl.brasiltelecom.net.br) |
| 05:15.09 | jkakar | gabriel_: What timezone are you in, out of curiosity? |
| 05:15.27 | gabriel_ | EST, you? |
| 05:15.38 | jkakar | gabriel_: Ooh, it's late for you. :) |
| 05:15.39 | gabriel_ | (yes, its pretty late here) |
| 05:15.44 | jkakar | gabriel_: I'm in PST, I live in Vancouver, Canada. |
| 05:15.58 | gabriel_ | oh, i'm in Hamilton :) |
| 05:16.04 | jkakar | Oh, nice. :) |
| 05:16.22 | jkakar | gabriel_: We're neighbours. ;) |
| 05:17.02 | jkakar | Watching the test suite run without C extensions makes me really appreciate them. :) |
| 05:17.03 | gabriel_ | right, almost as close as my friends from Florida :) |
| 05:17.07 | jkakar | Heh |
| 05:18.38 | gabriel_ | does 'make check' use the C extensions? |
| 05:19.05 | gabriel_ | that's what i just ran |
| 05:19.31 | gabriel_ | ah, from Makefile: |
| 05:19.43 | gabriel_ | Run the tests once with cextensions and once without them |
| 05:20.24 | gabriel_ | jkakar: so do you work for canonical? |
| 05:20.36 | jkakar | gabriel_: I do, yeah. |
| 05:20.44 | jkakar | gabriel_: I work on the Landscape team. |
| 05:21.56 | gabriel_ | jkakar: they don't have an office in Vancouver, do they? |
| 05:22.32 | jkakar | gabriel_: Nope. I work from home. We have an office in Montréal, primarily for our support team. |
| 05:23.40 | gabriel_ | jkakar: ok, that's what I thought. I guess they understand that distributed development can work, eh? |
| 05:24.31 | jkakar | gabriel_: Yeah. The majority of Canonical employees (particularly ones with technical roles) are distributed across the world and work from wherever they happen to live. |
| 05:25.12 | gabriel_ | jkakar: that's pretty cool |
| 05:25.19 | jkakar | gabriel_: It is, yeah. |
| 05:25.28 | gabriel_ | ok, the test suite just finished, but still appears to be broken |
| 05:25.29 | jkakar | gabriel_: We have regular sprints where we meet up in person. |
| 05:26.13 | jkakar | gabriel_: I have one test failure now, and it's MySQL related: AssertionError: u'C??ng' != u'C\u1eeb\u01a1ng' (ensure your database was created with CREATE DATABASE ... CHARACTER SET utf8) |
| 05:26.24 | jkakar | Which doesn't make sense since I specified 'utf8' when I created the database. |
| 05:26.37 | gabriel_ | jkakar: i didn't know they did that for all the teams, but i was at part of the bzr sprint at pycon a couple years ago |
| 05:28.09 | jkakar | gabriel_: Cool! The different teams have different sprint schedules, depending project needs, people involved, etc. But generally speaking, all the distributed teams have sprints: there is a lot of value in having personal time together, and also a number of tasks that benefit hugely from the high bandwidth you get by being in the same room with others. |
| 05:28.27 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 05:28.51 | jkakar | jukart: Heya. I reviewed a branch that therve pushed with a fix for the problem you reported the other day. |
| 05:28.55 | gabriel_ | jkakar: yea, absolutely. That was the first properly organized sprint i'd ever attended |
| 05:29.02 | jkakar | jukart: If you're able/willing to provide a second review we can get it landed. |
| 05:29.21 | jukart | jkakar: sure |
| 05:29.33 | jukart | I already have that branch in production :) |
| 05:29.44 | jkakar | jukart: Sweet, thanks. The merge proposal is here: https://code.edge.launchpad.net/~therve/storm/mutable-variables-flush-leak/+merge/12827 |
| 05:29.48 | jkakar | jukart: Ah, heh, nice. :) |
| 05:29.54 | jkakar | jukart: Nothing like real world testing. ;) |
| 05:30.09 | gabriel_ | jkakar: arg, just realized that I forgot to restart the server after updating the config file again...test suite is running again |
| 05:30.17 | jkakar | gabriel_: :) |
| 05:31.20 | jukart | jkakar: there is no change to the branch, right? |
| 05:31.56 | jkakar | jukart: Not since Oct 3 as far as I can see. |
| 05:32.00 | jukart | That means that it is OK, because my production system runs like charm since this patch |
| 05:32.25 | gabriel_ | jkakar: ok, so now i just have the unicode test failing, same as you |
| 05:32.26 | jkakar | jukart: Okay, if you could add an 'Approve' review to the merge proposal with that information, it'd be useful. |
| 05:32.38 | jkakar | gabriel_: I used mysqladmin to create the database before. |
| 05:33.04 | jkakar | gabriel_: I've just dropped it and created it manually with 'CREATE DATABASE storm_test CHARACTER SET utf8' and am rerunning the tests. |
| 05:33.17 | jkakar | gabriel_: Perhaps that will solve the problem. |
| 05:34.21 | gabriel_ | ok, i didn't explicitly set the encoding, i just ran 'mysqladmin create storm_test', so that may be the problem |
| 05:34.29 | jkakar | I wonder how much speedup we'd get by running the MySQL and PostgreSQL tests in parallel. |
| 05:34.36 | jkakar | gabriel_: Yeah, that's a problem. |
| 05:34.51 | jkakar | gabriel_: I tried fixing it by adding --default-character-set utf8 but that didn't help. |
| 05:35.19 | jkakar | I set all this up on my main development machine so long ago that I don't even remember jumping through these hoops. |
| 05:35.28 | jkakar | Will make sure to add all this crap to the README this time. |
| 05:35.45 | gabriel_ | that being said, I've been using storm with unicode on a MySQL database without explitly setting the encoding, and haven't had any problems so far... |
| 05:36.00 | jukart | jkakar: done |
| 05:36.06 | jkakar | jukart: Thanks! |
| 05:37.03 | jukart | jkakar: btw, is there support for sharding going on somewhere ? |
| 05:38.13 | jkakar | jukart: What kind of support? |
| 05:38.32 | jukart | I mean development for sharding |
| 05:39.33 | jkakar | jukart: Well, we're already using Storm for sharding... I guess I don't understand the question still. |
| 05:40.00 | jukart | jkakar: some automatic server selection in a store |
| 05:42.12 | jkakar | jukart: Ah, I see. Nothing is going on to provide that kind of thing, as far as I know. |
| 05:42.24 | jukart | jkakar: OK, thanks |
| 05:42.57 | jkakar | jukart: That said, Storm has been designed from day one to handle connections to many databases, with the same or different schemas. |
| 05:43.26 | jkakar | jukart: We have many physical databases in Landscape that we access with Storm simultaneously and it works great. |
| 05:44.26 | jukart | jkakar: I do that too and I guess it wouldn't be a big thing to provide a "Store" with some automatic selection of a database |
| 05:45.04 | jukart | we may have this requirent in the near future |
| 05:45.47 | jkakar | jukart: Right. Do you use something like ZStorm or the Django middleware? |
| 05:45.58 | jukart | I use ZStore |
| 05:46.20 | jkakar | jukart: Cool. I'd guess extending ZStorm to provide that kind of functionality would be the natural way to deal with the issue. |
| 05:46.29 | jukart | We use storm with Zope3 and with tornado |
| 05:46.37 | jkakar | jukart: Oh wait, Lovely systems... of course you use Zope 3. :) |
| 05:56.21 | jkakar | gabriel_: Okay, all tests pass here, so it does appear you need to run a CREATE DATABASE statement. |
| 05:56.27 | jkakar | For MySQL that is. |
| 05:57.18 | gabriel_ | jkakar: ok... 'CREATE DATABASE storm_test CHARACTER SET utf8'? |
| 05:57.40 | jkakar | gabriel_: Yep, exactly. |
| 06:16.33 | gabriel_ | jkakar: ok, the test suite just finished running again, and there aren't any errors. Yay! |
| 06:16.56 | jkakar | high-fives gabriel_! |
| 06:17.09 | jkakar | gabriel_: Now we need to write this down so no one ever has to go through this again. :) |
| 06:17.25 | jkakar | gabriel_: I noticed you created StartingDevelopment, which is a good first step, |
| 06:21.09 | gabriel_ | did you see the content, or was i still editing it when you had a look? |
| 06:21.31 | gabriel_ | jkakar: in any case, there should be stuff there now |
| 06:21.51 | jkakar | gabriel_: I'm subscribed to all changes in the wiki, so I got an email about it. :) |
| 06:22.15 | jkakar | gabriel_: Sweet, these instructions are great. |
| 06:22.33 | gabriel_ | like i said, its just a writeup of what you explained to me today and a walkthrough of the configuration steps that i had to sort out |
| 06:22.53 | jkakar | gabriel_: I'll probably move them to the README file, if you don't mind, since I think they make more sense there than in the wiki. |
| 06:22.58 | gabriel_ | jkakar: thanks for spending so much time to explain it all to me :) |
| 06:23.07 | jkakar | gabriel_: My pleasure. Thanks for sticking it out. :) |
| 06:24.39 | gabriel_ | jkakar: whatever makes sense. my personal preference when checking out a new project is to have the info both in the README as well as on the wiki, but as long as other people don't have to spend an afternoon wrestling with this, i'll be happy :) |
| 06:25.00 | jkakar | gabriel_: Aye. :) |
| 06:25.05 | gabriel_ | no, lets see if i can get that code review done :) |
| 06:25.10 | jkakar | YAY! |
| 06:25.10 | gabriel_ | *now |
| 06:25.16 | jkakar | cheers gabriel_ on! |
| 07:14.25 | mup | storm/trunk r330 committed by thomas@canonical.com |
| 07:14.25 | mup | Merge mutable-variables-flush-leak [r=jkakar,jukart] |
| 07:14.25 | mup | Remove a leak when mutable variables (ListVariable or PickleVariable instances) are |
| 07:14.25 | mup | collected before store.flush, leaving hooks behind them. |
| 07:15.15 | jukart | mup: thanks, thats a very important fix for us |
| 07:15.43 | mup | storm/trunk r331 committed by thomas@canonical.com |
| 07:15.43 | mup | Add a note in the NEWS file about r330 change. |
| 07:16.32 | jkakar | jukart: mup is a bot. ;) |
| 07:16.43 | jukart | jkakar: just figured this out :) |
| 07:17.26 | jkakar | jukart: There's a couple of other bits I want to land, but will try and release 0.16 in the next couple of weeks so that you can guys can use something "official". |
| 07:17.57 | *** topic/#storm by jkakar -> The Storm Python ORM - http://storm.canonical.com/ - 0.15 released! || Review branches: https://code.launchpad.net/storm/+activereviews || IRC logs: http://ibot.rikers.org/#storm/ |
| 07:26.30 | mup | storm/trunk r332 committed by jkakar@kakar.ca |
| 07:26.31 | mup | - The README file now includes information about Storm's license, |
| 07:26.31 | mup | <PROTECTED> |
| 07:26.31 | mup | <PROTECTED> |
| 07:26.31 | mup | <PROTECTED> |
| 07:35.43 | *** join/#storm zmijunkie (n=chatzill@i59F5D91A.versanet.de) |
| 07:37.20 | *** topic/#storm by jkakar -> The Storm Python ORM - http://storm.canonical.com/ - 0.15 released! || Review branches: https://code.launchpad.net/storm/+activereviews || IRC logs: http://ibot.rikers.org/#storm/ || API documentation: http://people.canonical.com/~jkakar/storm/ |
| 07:42.13 | gabriel_ | jkakar: thanks for the shout-out. I'm looking forward to seeing .16 - hopefully now that I've got the environment set up, I'll be able to help with some more reviews to make it happen |
| 07:43.25 | gabriel_ | jkakar: I just commented on the nested set branch (bug# 242813), and I'm off to bed now |
| 07:43.26 | mup | Bug #242813: ResultSet set expression API limits size of unions and intersections <Storm:New for jamesh> <https://launchpad.net/bugs/242813> |
| 07:43.37 | gabriel_ | jkakar: g'night! |
| 07:43.51 | jamesh | looks |
| 07:43.59 | jamesh | gabriel_: thanks for the input, btw. |
| 07:44.15 | gabriel_ | jamesh: oh, didn't realize you were here |
| 07:44.17 | jkakar | gabriel_: Have a good one! |
| 07:44.22 | jkakar | I'm off to bed too. |
| 07:44.22 | gabriel_ | jamesh: np |
| 07:44.26 | jkakar | => zzz |
| 07:45.24 | gabriel_ | jamesh: as i said to jkakar earlier today, i don't have much experience with code reviews, so i hope i was looking at the right things (and not asking too many stupid questions :) |
| 08:01.51 | *** join/#storm elmom_ (n=elmom@vallila-gw.hupnet.helsinki.fi) |
| 08:03.19 | jamesh | gabriel_: just having a second person look over a piece of code is useful, and something that any programmer can do. And the questions in your comment are good ones. |
| 08:04.42 | gabriel_ | jamesh: well, glad i could be of assistance then |
| 09:12.45 | *** join/#storm artista_frustrad (n=artista_@201-15-219-136.ctame704.dsl.brasiltelecom.net.br) |
| 09:27.31 | *** join/#storm zmijunkie_ (n=chatzill@i53870DFD.versanet.de) |
| 09:56.56 | *** join/#storm seiflotfy2 (n=seif@P5118.pallas.wh.tu-darmstadt.de) |
| 11:19.01 | *** join/#storm elmom_ (n=elmom@cs181200016.pp.htv.fi) |
| 11:52.47 | *** join/#storm fcorrea (n=fcorrea@189-54-36-208-nd.cpe.vivax.com.br) |
| 12:02.11 | *** join/#storm niemeyer (n=niemeyer@200-103-139-122.ctame705.dsl.brasiltelecom.net.br) |
| 12:44.34 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 13:07.05 | *** join/#storm fcorrea (n=fcorrea@187.3.193.8) |
| 13:09.34 | *** join/#storm artista_frustrad (n=artista_@201.35.54.137) |
| 14:23.33 | *** join/#storm stub (n=stub@58.8.17.245) |
| 14:46.51 | *** join/#storm drudi (n=drudi@189.65.243.145) |
| 15:33.04 | *** join/#storm jukart (n=jukart@d86-32-163-51.cust.tele2.at) |
| 16:03.17 | *** join/#storm elmom_ (n=elmom@hoasnet-ff05dd00-13.dhcp.inet.fi) |
| 16:17.08 | gabriel_ | some files in the test suite report 0 test run, despite appearing to contain tests (tests/databases/sqlite.py and tests/django/stores.py etc.) |
| 16:17.26 | gabriel_ | should all the tests not execute? |
| 16:45.12 | *** join/#storm jkakar (n=jkakar@S0106001ee57a879c.vc.shawcable.net) |
| 17:04.29 | jkakar | gabriel_: Good morning. :) |
| 17:04.40 | jkakar | gabriel_: Some of the tests run conditionally, depending on what's available on your system. |
| 17:04.57 | jkakar | gabriel_: So, for example, if you don't have MySQL installed the MySQL-related tests won't run. |
| 17:07.53 | gabriel_ | jkakar: good morning! |
| 17:08.19 | gabriel_ | jkakar: that makes sense. So I suppose that I need to have django installed for those tests to run |
| 17:08.40 | gabriel_ | jkakar: but what about the sqlite test? |
| 17:10.31 | gabriel_ | jkakar: also, tests/databases/mysql.py reports that it contains one test, but that it ran none |
| 17:12.21 | jkakar | gabriel_: I'm not sure about the SQLite or MySQL tests. You'll have to look at the individual test and figure out what it's is_supported method is doing, I suspect. |
| 17:12.52 | jkakar | And yeah, the Django tests run for me and I have it installed, so I suspect that's the issue there. |
| 17:18.14 | *** join/#storm artista_frustrad (n=artista_@201-24-227-169.ctame704.dsl.brasiltelecom.net.br) |
| 17:18.22 | *** join/#storm drudi (n=drudi@189.65.33.141) |
| 17:24.17 | gabriel_ | jkakar: woops! i think i was reading the test results wrong (associating test results with the filename below, rather than above :). The only ones that are failing make sense (django and zope related) |
| 17:24.29 | jkakar | gabriel_: Sweet. :) |
| 17:24.42 | gabriel_ | failing -> not running |
| 17:24.54 | gabriel_ | thanks for the patience |
| 17:25.14 | jkakar | None was required, please keep asking questions. :) |
| 19:14.16 | *** join/#storm jukart (n=jukart@vpn3.hotsplots.net) |
| 20:40.39 | *** join/#storm artista-frustrad (n=artista_@201-15-250-200.ctame704.dsl.brasiltelecom.net.br) |