| 04:02.19 | *** join/#storm dinho (n=dinho@201.79.137.196) |
| 04:35.18 | *** join/#storm bigdo1 (n=scmikes@49-66-14-216-arpa.cust.cinci.current.net) |
| 06:27.37 | *** join/#storm Fujitsu (n=fujitsu@ubuntu/member/fujitsu) |
| 07:24.51 | *** join/#storm dinho (n=dinho@201.79.136.71) |
| 07:28.41 | *** join/#storm dobee (n=dobeee@194.183.146.186) |
| 07:39.15 | *** join/#storm rafael (n=rafael@moinmoin/fan/rafael) |
| 07:39.34 | rafael | good morning |
| 09:59.02 | *** join/#storm rafael_ (n=rafael@p5089C899.dip.t-dialin.net) |
| 10:14.02 | *** part/#storm soren (n=sh@ubuntu/member/shawarma) |
| 11:24.31 | *** join/#storm rafael_ (n=rafael@p5089E610.dip.t-dialin.net) |
| 12:16.11 | *** join/#storm niemeyer (n=niemeyer@201-14-57-210.ctame705.dsl.brasiltelecom.net.br) |
| 13:14.22 | *** join/#storm moshe (n=moshe@ztn-c-1566b.adsl.wanadoo.nl) |
| 13:47.06 | *** join/#storm rafael_ (n=rafael@p5089FC97.dip.t-dialin.net) |
| 13:52.40 | *** join/#storm desmaj (n=user@d65h163.public.uconn.edu) |
| 14:45.44 | *** join/#storm nephish_ (n=piv@63.98.191.58) |
| 14:50.51 | *** join/#storm dobee (n=dobeee@81-223-53-162.dornbirn.xdsl-line.inode.at) |
| 15:13.27 | *** join/#storm dobee (n=dobeee@81-223-53-162.dornbirn.xdsl-line.inode.at) |
| 17:12.59 | *** join/#storm rafael (n=rafael@moinmoin/fan/rafael) |
| 17:56.20 | rafael | isn't there any way to create a db without the store.execute(CREATE TABLE...) statement? i think this part sucks |
| 17:56.41 | radix | rafael: storm does not define schemas for you |
| 17:56.53 | rafael | why? |
| 17:57.15 | radix | rafael: it's tricky :) |
| 17:57.39 | rafael | to implent it? |
| 17:57.50 | radix | rafael: well, to implement it and to have an API that's sensible and flexible |
| 17:57.55 | jkakar | rafael: Storm doesn't try to hide SQL from you. That was an explicit design decision. |
| 17:58.15 | rafael | but then you must write it twice |
| 17:58.24 | radix | rafael: except you're not really writing the same thing twice. |
| 17:58.26 | jkakar | rafael: Also, is it really so hard to define your schema manually? |
| 17:58.39 | rafael | no but i think it could be nicer |
| 17:58.49 | jkakar | I actually prefer it, personally. |
| 17:59.02 | radix | I can understand it. |
| 17:59.22 | jkakar | Perhaps I'm missing the benefits of automating that kind of thing. |
| 17:59.47 | jkakar | I can see the "target my schema at another database easily" as being the primary benefit but I doubt that use case actually comes up very often. |
| 17:59.53 | rafael | for example: text = Unicode(sql_field='text VARCHAR') but the default is 'text TEXT' |
| 17:59.56 | radix | jkakar: they're pretty obvious: if you have automatic schema generation, you can define your database schema and your python object schema at once |
| 18:00.01 | rafael | and you can pack this in an optional module.. |
| 18:00.14 | radix | rafael: that's an interesting idea |
| 18:00.51 | rafael | i think it isn't so difficult to impolent^^ |
| 18:01.02 | rafael | *implent |
| 18:01.04 | radix | rafael: indeed it's not, but it's pretty different from what most people want with automatic schema generation |
| 18:01.18 | rafael | what do they want? |
| 18:01.36 | radix | rafael: anyway, I think it'd be worth starting a discussion on this |
| 18:01.56 | radix | rafael: If you want, it'd be pretty cool if you started a wiki page on storm.canonical.com specifying your ideas. |
| 18:02.21 | rafael | oki |
| 18:02.40 | radix | we could even start a blueprint on launchpad :) |
| 18:03.52 | rafael | blueprint? ehm the way is called "schemas", isn't it? |
| 18:04.01 | radix | huh? |
| 18:04.19 | radix | blueprints are a launchpad thing that are meant to track feature development with specification documentations. |
| 18:04.28 | radix | rafael: AutomaticSchemaGeneration, perhaps? :) |
| 18:04.37 | rafael | yeah, thanks |
| 18:08.24 | rafael | some time ago there was a discussion about the Char field.. how it is named now? |
| 18:11.25 | radix | rafael: RawStr, but that's only in trunk, we need to do a 0.10 release soonish |
| 18:11.33 | radix | rafael: but Char will continue to work for backwards compatibility |
| 18:38.33 | rafael | radix: http://dpaste.com/14892/ i have no launchpad acc... can you please add it in the wiki? |
| 18:39.33 | radix | rafael: sure thing. you'll probably want to get a launchpad account if you want to interact much with the Storm project though, it's pretty easy to register |
| 18:40.11 | rafael | thanks :-) maybe there are some english mistakes... i'm german |
| 18:44.01 | rafael | oh, there is a "ERROR: EOF in multi-line statement at the end :/ |
| 18:46.49 | niemeyer | rafael: FWIW, I don't agree that it's being defined twice |
| 18:47.21 | niemeyer | rafael: One of the ways to prove that is that your proposal includes having things like sql_field="..." |
| 18:47.45 | niemeyer | rafael: In general, additional information is needed to define any non-trivial schema |
| 18:48.42 | niemeyer | rafael: and that ends up being defined in the Python class or in a Python-like representation of each detail of the database, so that more complex schemas may be defined |
| 18:49.15 | niemeyer | rafael: In Storm, we explicitly decided to move the Python class into the Python side |
| 18:49.54 | niemeyer | rafael: All Storm properties map exactly to Python types |
| 18:50.38 | niemeyer | rafael: We do have things like allow_none attributes in these properties, but these are only defined when there's a benefit for the programming logic, rather than for schema definition. |
| 18:51.01 | niemeyer | rafael: We're not likely to have SmallInt/LongInt/etc, as a counter example |
| 18:51.28 | radix | niemeyer: well, his proposal allows Integer(sql_field="SMALLINT") |
| 18:52.05 | niemeyer | radix: Right.. I'm not even evaluating positively or negatively the idea here (more discussion is needed) |
| 18:52.11 | radix | ok |
| 18:52.11 | radix | yeah |
| 18:52.13 | niemeyer | radix: I'm just pointing out that there isn't *duplication* |
| 18:52.18 | radix | I agree |
| 18:52.28 | radix | rafael: but anyway, thank you very much for kicking this discussion off in a structured way |
| 18:52.46 | radix | rafael: we had many random comments about schema generation, but nobody actually structuring their comments, so thank you for writing that wiki page :) |
| 18:52.48 | rafael | no problem :-) |
| 18:52.59 | niemeyer | That's very true |
| 18:53.05 | niemeyer | and it's a neat idea as well.. |
| 19:33.38 | *** join/#storm Moshe (n=moshe@ztn-c-1566b.adsl.wanadoo.nl) |
| 19:34.50 | Moshe | How do dynamically create "SELECT 1, 2" the way Select(1) creates "SELECT 1"? |
| 19:36.17 | radix | Select([1, 2]) seems to work |
| 19:41.46 | Moshe | Ok. I'm working on a firebird backend, that one might need some special handling |
| 19:43.13 | Moshe | "SELECT ? FROM foo" does not work, in firebird it needs to be "SELECT CAST(? AS INT) FROM foo" |
| 19:45.46 | radix | huh, I didn't even know that worked in other databases :) |
| 19:45.52 | radix | does Storm actually generate queries like that? |
| 19:46.01 | radix | ... |
| 19:46.27 | *** join/#storm TilAshi (n=moshe@ztn-c-1566b.adsl.wanadoo.nl) |
| 19:46.32 | TilAshi | oops |
| 19:47.07 | TilAshi | storm doesn't build the CAST, but firebird needs it ;-) |
| 19:47.21 | radix | I mean, I didn't even know it generated "SELECT ? FROM foo" |
| 19:47.48 | TilAshi | It does in the test_execute_expression testcase |
| 19:49.42 | TilAshi | there isn't a testcase for Select([1,2]) yet, but I suppose the backend needs to handle that kind of queries |
| 19:54.30 | *** join/#storm EnTeQuAk (n=EnTeQuAk@p54BD855A.dip0.t-ipconnect.de) |
| 20:02.38 | nephish_ | i am just curious, in the above paste there are {{{ and }}} brackets, what do these signify ? |
| 20:03.34 | TilAshi | I still have 2 major problems remaining, the datetime tests lose their microseconds (probably something wrong with the python module for firebird), and get_insert_identity needs to be implemented |
| 20:06.45 | rafael | nephish_: what paste? |
| 20:08.19 | radix | nephish_: it's moinmoin wiki markup |
| 20:08.22 | radix | nephish_: it means "pre-formatted text" |
| 20:08.37 | *** part/#storm EnTeQuAk (n=EnTeQuAk@p54BD855A.dip0.t-ipconnect.de) |
| 20:08.59 | radix | TilAshi: thanks a lot for working on a firebird backend |
| 20:13.12 | nephish_ | oh, i see, thanks |
| 20:13.22 | TilAshi | (tests=22, failures=5, errors=0) << those 5 are the hard ones ;-) |
| 20:13.25 | radix | :-) |
| 20:28.10 | TilAshi | ehh, there is a double test_execute_empty_params inside tests/databases/base.py ;-) |
| 20:29.14 | radix | I think someone just filed a bug about that |
| 20:29.25 | TilAshi | ah, ok |
| 20:40.19 | TilAshi | datetime.datetime(1977, 4, 5, 12, 34, 56, 78) is too much precision for firebird, it only holds datetime values on a 100th millisecond scale |
| 20:41.19 | TilAshi | should I override the test to use less precision? |
| 20:41.53 | TilAshi | or is it better to change the tests inside tests/databases/base.py? |
| 20:43.52 | radix | TilAshi: override the test specifically for firebird, probably |
| 20:44.01 | TilAshi | ok |
| 20:44.34 | radix | and note in the docstring why it's being overridden |
| 20:45.40 | TilAshi | sure |
| 20:46.13 | TilAshi | I've already overridden the tests with static SQL |
| 20:46.46 | TilAshi | because they use SELECT without FROM |
| 20:47.22 | radix | ah, that's like Oracle |
| 20:48.19 | TilAshi | indeed |
| 20:49.41 | TilAshi | I've read on the mailinglist that oracle has the DUAL table, firebird has RDB$DATABASE |
| 21:58.15 | Moshe | What exactly should get_insert_identity do? |
| 21:59.18 | Moshe | I see it builds an expression that can be used to identify the last insert, but what are those arguments for? |
| 22:00.31 | radix | Moshe: well, you don't have to use them if you don't need to |
| 22:00.38 | radix | Moshe: sqlite's database backend doesn't need to, for example |
| 22:00.41 | radix | <PROTECTED> |
| 22:00.42 | radix | <PROTECTED> |
| 22:01.34 | radix | I'm trying to figure out exactly how it's used now |
| 22:01.48 | Moshe | well, those 2 tests call it with different arguments, what use is that, if it should return the same result? |
| 22:02.38 | radix | Moshe: which tests? |
| 22:02.50 | radix | it probably *does* exercise different codepaths in certain databases |
| 22:02.52 | radix | like postgres and mysqol |
| 22:03.07 | Moshe | test_get_insert_identity and test_get_insert_identity_composed |
| 22:03.20 | radix | see databases/postgres.py's implementation of get_insert_identity, it uses both parameters |
| 22:03.38 | radix | by constructing a comparison that matches the primary keys to the values of those primary keys |
| 22:04.18 | Moshe | I see, but the result should not be different, should it? |
| 22:05.03 | Moshe | or should it be possible to get the identity for the previous insert or something? |
| 22:05.12 | radix | well one test tests when you just have a single primary key, and the other one tests that it works for multi-primary-keys |
| 22:05.17 | radix | Moshe: what? that's exactly what the method is for |
| 22:05.22 | radix | to get the identity of the previous insert... |
| 22:06.00 | Moshe | I mean the insert before the last |
| 22:06.43 | radix | nope... |
| 22:07.13 | radix | I think I've figured out what it's all about now, but now I'm not understanding you |
| 22:08.00 | radix | you have to implement get_insert_identity so it works whether you just inserted a row with one or multiple primary keys (and sometimes you don't even have the values for all the primary keys) |
| 22:08.31 | radix | hrm. primary variables, not primary values |
| 22:08.56 | radix | yeah, ok that's right |
| 22:09.23 | Moshe | well, what use is test_get_insert_identity_composed, title is no primary key as far as I can see |
| 22:10.01 | Moshe | anyway, I'm having a hard time thinking of something, the way it should be done for firebird is first generating a value, and then inserting with that value |
| 22:10.37 | Moshe | but that's not really possible |
| 22:10.48 | radix | hmm, yeah, I'm not sure why it's using a non-primary-key in the test |
| 22:12.27 | radix | Moshe: does the firebird python interface expose something like .lastrowid, like mysql? or does firebird have some sort of global row-IDs like sqlite and oracle do? |
| 22:12.37 | Moshe | no, it doesn't |
| 22:12.51 | radix | Moshe: the postgres backend takes advantage of the implicitly-created tables for sequence IDs |
| 22:13.16 | radix | which postgres creates |
| 22:13.21 | Moshe | well, I can use the generators from firebird, but the are not implicit |
| 22:13.54 | radix | ah, so it'd require you to define your schema in a particular way/ |
| 22:13.58 | Moshe | so the user makes up a name for the generator |
| 22:14.08 | Moshe | it does |
| 22:14.39 | Moshe | you have to create a before insert trigger |
| 22:15.14 | radix | aye |
| 22:15.18 | Moshe | but if you use a trigger, you can't know the value (according to the docs) |
| 22:16.06 | Moshe | the way to know the value is SELECTing the value before the INSERT |
| 22:16.19 | Moshe | as far as the docs tell, its the only way |
| 22:17.29 | Moshe | I figured, the arguments for get_insert_identity might make it possible to just make an expression based on the inserted values |
| 22:17.35 | radix | well, it does |
| 22:17.35 | Moshe | but that's ugly |
| 22:17.48 | radix | it is? that's exactly how the mysql and postgres implementations work |
| 22:18.02 | radix | the problem is that sometimes you've got a primary key column that gets its value populated by the database |
| 22:18.12 | radix | so you don't know the value to query for |
| 22:18.22 | Moshe | they don't use the lastrowid stuff? |
| 22:18.26 | radix | not in postgres |
| 22:18.27 | radix | it doesn't have one |
| 22:19.12 | radix | the only kind of implicit-value-primary-key that the postgres backend supports is sequence columns (which are by far the common case) |
| 22:20.05 | radix | actually that may also work for UUID columns, it uses the built-in postgres function currval('table_column_seq') |
| 22:20.29 | Moshe | hmm, I remembered psycopg2 to have an .lastrowid, but that is the OID |
| 22:20.55 | radix | hm |
| 22:20.59 | radix | maybe it does :) |
| 22:21.03 | radix | this code was mostly written for psycopg1 |
| 22:21.09 | radix | it was only ported to psycopg2 right before the public release |
| 22:21.20 | Moshe | ah, ok |
| 22:21.42 | Moshe | but it doesn't work anyway with WITHOUT OID ;-) |
| 22:22.46 | radix | is that a clause in CREATE TABLE? |
| 22:23.32 | radix | oops, the psycopg2 FAQ says that OIDs are on their way out anyway |
| 22:25.14 | Moshe | I haven't used postgres in some time, I have to use firebird at my work, at the moment |
| 22:25.40 | Moshe | currval is the good solution for postgres |
| 22:25.49 | radix | yeah |
| 22:26.30 | radix | Moshe: so hrm |
| 22:26.40 | radix | Moshe: I wonder why they say you have to get the value before the insert |
| 22:26.46 | radix | Moshe: is it not an incrementing counter? |
| 22:27.14 | Moshe | it is, but gen_next(generatorname, 0) is not multi-user safe |
| 22:27.27 | radix | even in a SERIALIZABLE transaction? |
| 22:27.44 | Moshe | it never is |
| 22:27.48 | radix | :\ |
| 22:29.40 | Moshe | would it be possible to somehow let the user define a callback that provides the id? |
| 22:30.18 | Moshe | the user knows the id if it did an gen_next(foo, 1) before the insert, storm doesn't |
| 22:30.47 | radix | by "the user" to you mean "the database backend"? why can't the database backend call that itself? |
| 22:31.18 | Moshe | by the user I mean, the one creating the application using storm |
| 22:31.40 | radix | Couldn't you avoid that by having the database backend get call the gen_next before the insert? |
| 22:32.00 | radix | I'm assuming some arbitrary hacking to storm itself to allow this to work |
| 22:32.08 | Moshe | well, the names for the generators are arbitrary, I can't really make a wild guess |
| 22:32.39 | radix | Moshe: you could perhaps allow the user to specify that in a backend-specific way |
| 22:34.13 | radix | just so the user can be responsible for the minimum necessary |
| 22:35.34 | radix | I need to get back to some horrible ancient DBUS hackery |
| 22:35.51 | radix | actually I think I need to quit for the day, this crap is making me tear my hair out |
| 22:36.17 | Moshe | DBUS :-) |