| 00:00.29 | *** join/#storm _mup_ (~mup@li37-90.members.linode.com) |
| 00:19.31 | *** join/#storm kr4d10 (~kr4d10@76.73.0.203) |
| 00:29.20 | *** join/#storm magmatt (~Matt@63-235-131-194.dia.static.qwest.net) |
| 01:08.50 | *** join/#storm dangayle (~dangayle@67-5-82-199.spok.qwest.net) |
| 01:09.03 | *** part/#storm dangayle (~dangayle@67-5-82-199.spok.qwest.net) |
| 05:04.45 | *** join/#storm stub (~stub@canonical/launchpad/stub) |
| 12:17.35 | *** join/#storm niemeyer (~niemeyer@201-35-138-159.pltce701.dsl.brasiltelecom.net.br) |
| 12:17.35 | *** mode/#storm [+o niemeyer] by ChanServ |
| 13:14.56 | *** join/#storm awarner (awarner@nat/theladders/x-ofujxpuxxsmrihoc) |
| 20:31.47 | *** join/#storm phyatt (~phyatt@65.242.92.120) |
| 20:31.55 | *** part/#storm phyatt (~phyatt@65.242.92.120) |
| 21:08.27 | *** join/#storm lifeless (~robertc@122.58.129.196) |
| 21:21.09 | *** join/#storm lifeless (~robertc@122.58.129.196) |
| 21:28.12 | *** join/#storm lifeless_ (~robertc@122.58.129.196) |
| 21:41.54 | magmatt | I'm having trouble doing Max() on a numeric -- storm is doing something I can't figure out: https://gist.github.com/1887584 |
| 21:43.05 | magmatt | anyone know why storm max is 6.2 and sqlite's is 10? |
| 21:48.33 | magmatt | it looks like storm just unicodes Decimals before sending them to the database |
| 21:50.33 | magmatt | am I reading that correctly? http://bazaar.launchpad.net/~storm/storm/trunk/view/head:/storm/variables.py#L372 |
| 22:01.45 | magmatt | Is there a way to see the actual value being sent to sqlite? |
| 22:05.35 | magmatt | is anyone here? |
| 22:10.35 | marienz | it's entirely possible the data lands in the sqlite db as strings, and you're getting the lexicographical latest (with 6 sorting after 1) |
| 22:10.46 | marienz | I've not used decimals with storm and sqlite, I think |
| 22:11.14 | marienz | with sqlite not having that kind of datatype I don't think you're going to be able to use sqlite's functions on stored decimals sensibly |
| 22:12.03 | magmatt | marienz: how do I figure out what storm is doing to the data before it gets to sqlite? I'd have a better shot if I could discover the magic |
| 22:13.27 | marienz | well, with sqlite not having this datatype the best storm can do is store them as strings. If it stored them as floats you'd lose precision. |
| 22:13.44 | marienz | and I don't think sqlite has useful functions for working with strings that are actually decimal numbers |
| 22:13.53 | magmatt | marienz: how do I duplicate what storm is doing? When I pass in unicode strings, it converts them to numbers (I updated the paste) |
| 22:20.35 | marienz | mmm, that's a little odd |
| 22:23.02 | marienz | I can't quite tell what's going on (there's different cases of what I think is breakage in both) |
| 22:23.54 | marienz | still, keep in mind sqlite just does not have a decimal datatype, so if you're going to do comparisons on stored decimals fancier than equality/inequality it just cannot work, unless there's some sqlite function I've not used for working with decimal numbers stored as strings |
| 22:24.32 | marienz | so if you want that Max call to actually do what you want you're just out of luck, I'm pretty sure |
| 22:25.34 | marienz | (your non-storm example looks like it's working, but I'm fairly certain your decimal's being stored as a float, which rather defeats the purpose of using decimals in the first place) |
| 22:25.38 | marienz | attempts to verify that assumption |
| 22:26.16 | marienz | certainly looks like it |
| 22:27.13 | magmatt | marienz: so the conclusion is... rats |
| 22:27.49 | marienz | I mean, see the "storage classes and datatypes" section of http://www.sqlite.org/datatype3.html |
| 22:28.20 | marienz | if all you're trying to do is store decimals and pull them back out (not use them in queries) storing them as strings seems somewhat sane, and afaik that's what storm is trying to do |
| 22:28.39 | magmatt | yeah, that's what storm does |
| 22:29.04 | magmatt | I would feel better, though, if I could duplicate storm's behavior |
| 22:29.05 | marienz | if you want to use them in queries in any nontrivial way you have to store them differently (I'd expect multiplying with a fixed constant and storing as ints to work for many use cases) |
| 22:29.29 | marienz | you could probably cook up a storm property that does that conversion |
| 22:29.49 | magmatt | well... we use sqlite for testing and postgres in production -- so that seems like an unwanted hack |
| 22:29.55 | marienz | agreed |
| 22:30.15 | marienz | but I'm assuming postgres has an actual decimal datatype you're using for this |
| 22:30.26 | magmatt | right |
| 22:32.43 | magmatt | sqlite does let you register functions -- it would be cool if there was a simple way to say, "use this for sqlite databases and don't for everything else" |
| 22:32.50 | magmatt | marienz: thanks for your help, as always |
| 22:33.29 | marienz | oh, some of the confusion actually makes sense |
| 22:33.52 | marienz | err, no. Hmm. |
| 22:36.22 | *** join/#storm kr4d10 (~kr4d10@2001:49f0:a01d::203) |
| 22:41.00 | marienz | weird, what I get from the storm tracer is EXECUTE: u'INSERT INTO foo (val) VALUES (?)', (Decimal('6.2'),), I cannot tell at which point that Decimal instance gets converted to anything else, and throwing a Decimal at an sqlite3 connection by hand doesn't work |
| 22:41.45 | magmatt | marienz: yes, I saw the same thing. The tracer calls variable.get() when I think it should call variable.get(to_db=True) |
| 22:43.15 | marienz | ahh, I didn't look inside the tracer, so I was assuming I was just seeing reprs, with nothing storm-specific going on |
| 22:44.10 | magmatt | marienz: so I'm wondering about using this: http://docs.python.org/library/sqlite3.html#sqlite3.Connection.create_aggregate |
| 22:45.22 | marienz | magmatt: as what, a replacement for max()? |
| 22:45.27 | magmatt | marienz: yeah |
| 22:45.33 | marienz | hrm |
| 22:45.58 | marienz | I guess that could work, ifff decimals are ending up in the db as strings properly |
| 22:46.11 | marienz | kinda bleh, but I can't think of anything less bleh :) |
| 22:46.29 | magmatt | marienz: right... so how can we find out what the actual type is in the database? |
| 22:46.51 | magmatt | as you noted, you can't pass a Decimal() to sqlite, so it has to be a string |
| 22:47.01 | marienz | I'd save a test db disk and have the sqlite3 tool dump it |
| 22:47.08 | marienz | s/disk/to disk/ |
| 22:47.37 | marienz | infobot: why thank you |
| 22:47.37 | infobot | why not? |
| 22:47.53 | marienz | resists the temptation to mess with the bot |
| 22:50.26 | magmatt | marienz: so this duplicates what storm is doing: http://paste.pocoo.org/show/555243/ |
| 22:52.02 | marienz | I knew it'd buffer()ify strings, but I didn't see a path from decimals to strings |
| 23:48.24 | *** join/#storm mgedmin_ (~mg@office.pov.lt) |
| 23:53.29 | *** join/#storm bigkevmcd_ (~kevin@87-194-156-112.bethere.co.uk) |