| 00:00.44 | *** join/#storm wallflower (n=wallflow@ip-205-246-113-216.pool.grokthis.net) |
| 03:02.09 | *** join/#storm niemeyer (n=niemeyer@201-25-35-29.pltce701.dsl.brasiltelecom.net.br) |
| 04:50.54 | *** join/#storm jukart (n=jukart@85-124-221-45.static.xdsl-line.inode.at) |
| 05:08.36 | *** join/#storm bigdo1 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 05:46.50 | *** join/#storm jukart (n=jukart@lsfw01.lovelysystems.com) |
| 05:53.39 | *** join/#storm bigdo1 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 06:57.14 | *** join/#storm dobee (n=dobee@lsfw01.lovelysystems.com) |
| 07:48.42 | *** join/#storm mcella (n=mcella@ip-176-85.sn2.eutelia.it) |
| 08:28.14 | *** join/#storm lightyear (n=ben@nat/fluendo/x-6dce6d710783a916) |
| 09:19.37 | *** join/#storm mcella (n=mcella@ip-176-85.sn2.eutelia.it) |
| 11:32.15 | *** join/#storm bigdo2 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 11:44.38 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 11:50.20 | *** part/#storm bigdo2 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 12:03.30 | *** join/#storm niemeyer (n=niemeyer@201-25-35-29.pltce701.dsl.brasiltelecom.net.br) |
| 12:56.08 | *** join/#storm kov (n=kov@debian/developer/kov) |
| 13:31.22 | *** join/#storm goschtl (n=goschtl@88.128.80.137) |
| 14:16.50 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 15:02.00 | *** join/#storm bigdo2 (n=scmikes@8.8.197.72) |
| 15:15.02 | *** join/#storm lgs (n=lgs@217.216.91.110.dyn.user.ono.com) |
| 15:25.46 | *** join/#storm goschtl (n=goschtl@80.187.210.51) |
| 15:37.23 | *** join/#storm goschtl (n=goschtl@80.187.210.51) |
| 15:40.50 | *** join/#storm goschtl (n=goschtl@80.187.210.51) |
| 15:44.49 | *** join/#storm dobee (n=dobee@85-124-200-100.static.xdsl-line.inode.at) |
| 15:45.54 | *** join/#storm dobee (n=dobee@85-124-200-100.static.xdsl-line.inode.at) |
| 15:52.02 | *** join/#storm niemeyer (n=niemeyer@201-11-214-174.pltce701.dsl.brasiltelecom.net.br) |
| 17:53.29 | *** join/#storm jukart (n=jukart@85-124-221-45.static.xdsl-line.inode.at) |
| 17:53.43 | *** join/#storm jamesh (n=james@canonical/launchpad/jamesh) |
| 18:29.10 | *** join/#storm jamesh_ (n=james@canonical/launchpad/jamesh) |
| 18:44.35 | *** join/#storm mshooshtari (n=mshoosht@216.169.139.2) |
| 18:44.58 | mshooshtari | Hello, I was wondering if there was any support for mssql currently or planned? |
| 18:47.43 | jkakar | mshooshtari: I believe there is a branch with some initial work in that direction. |
| 18:48.20 | mshooshtari | I tried sql alchemy and I can query on tables and work with data there just fine, but I'm not getting anything with views, it just hangs. |
| 18:48.32 | jkakar | mshooshtari: See https://bugs.edge.launchpad.net/storm/+bug/145317 |
| 18:48.35 | ubot3 | Malone bug 145317 in storm "Storm should support MS SQLServer databases" [Undecided,New] |
| 18:49.07 | mshooshtari | There's nothing about that published. I did a trace using sql profiler and it's being handled differently, but I hit a wall there. |
| 18:49.39 | jkakar | mshooshtari: Is there a special kind of syntax for dealing with views in MSSQL? If not, and it's just like querying any other table, then you shouldn't have a problem with Storm, provided the mssql support in the branch linked to that bug works. |
| 18:54.17 | mshooshtari | it should work. I'm going to give that branch a try |
| 18:57.31 | jkakar | mshooshtari: Cool! If you run into problems please add comments to the bug so that they can be dealt with. |
| 18:58.10 | reacocard | hi all, I've noticied creating objects from storm is fairly slow, what's the best way to speed it up? |
| 18:58.49 | mshooshtari | what tool should I use to download this branch? bzr branch lp:~sidnei/storm/mssql-support |
| 18:59.14 | mshooshtari | Do I need an svn client? |
| 18:59.17 | jkakar | reacocard: If you're just creating objects for the sake of inserting data into rows, then you might consider running INSERT statements directly, as that will avoid the overhead of creating and caching objects that are just going to be thrown away. |
| 18:59.26 | jkakar | mshooshtari: You need Bazaar. |
| 18:59.55 | jkakar | mshooshtari: 'apt-get install bzr' and then 'bzr branch lp:...' will get you the branch. |
| 18:59.55 | reacocard | jkakar: no, I need the actual object to use in the rest of the program. they'll stick around for a while, but I have to load a bunch at once so I want to speed it up. |
| 19:00.11 | jkakar | reacocard: Ah, I see. |
| 19:02.38 | mshooshtari | looks good, not fond of learning new classes for tables but canonical has a good reputation |
| 19:02.48 | mshooshtari | for community and product development |
| 19:03.26 | jkakar | mshooshtari: What do you mean "learning new classes"? Are you talking about learning about the Storm API, or about how you need to change your own code to use Storm? |
| 19:03.45 | mshooshtari | changing code to use storm |
| 19:03.47 | jkakar | mshooshtari: Mostly I'm wondering if there's something we can do to make that learning curve easier. |
| 19:04.08 | jkakar | mshooshtari: Ah, I see. I think (hope) you'll find Storm is pretty easy to integrate. |
| 19:04.21 | jkakar | mshooshtari: Have you seen the tutorial? |
| 19:04.52 | mshooshtari | I can pastebin something that I had to do for sqlalchemy |
| 19:05.09 | mshooshtari | it's a class that defines a table and you can use it and the columns ad objects from there |
| 19:06.30 | mshooshtari | http://www.pastebin.ca/1074609 |
| 19:06.35 | jkakar | Ah, that's one big difference between something like SQLAlchemy and Storm. By design, Storm doesn't do schema migration or handle table creation. There's some threads on the mailing list that go into detali about why. |
| 19:07.12 | jkakar | mshooshtari: That looks very similar to the kind of thing you need to do with Storm. |
| 19:07.45 | mshooshtari | I'm just looking for a row = table.select(where x = foo) |
| 19:07.50 | mshooshtari | then row.item |
| 19:10.40 | jkakar | mshooshtari: I see. This is easy with Storm, but instead of querying the table you use a Store. |
| 19:10.57 | jkakar | mshooshtari: Store.find(Table, Table.x == foo) for example. |
| 19:11.30 | jkakar | mshooshtari: Have you seen this: https://storm.canonical.com/Tutorial |
| 19:12.08 | mshooshtari | I have briefly, but didn't look too much because I didn't see anything on the main page for mssql |
| 19:12.51 | mshooshtari | I actually wanted to go with Storm first, but wasn't sure about the branch |
| 19:13.12 | jkakar | mshooshtari: I see. Well, Storm usage is mostly independent of the backend. I recommend you go through it since the things it describes are mostly backend-agnostic and will give you a feel for the API. |
| 19:14.47 | mshooshtari | good news no error now with create_database when using mssql:// |
| 19:15.31 | jkakar | Nice. :) |
| 19:20.09 | mshooshtari | having some trouble putting the pieces togeather |
| 19:20.22 | mshooshtari | the toutorial seems geared towards new databases |
| 19:20.48 | mshooshtari | I have created a class for the table, have a database with create_database |
| 19:22.19 | mshooshtari | maybe just help with defining the store |
| 19:22.54 | mshooshtari | lol found it. |
| 19:23.18 | jkakar | :) |
| 19:27.02 | mshooshtari | result = store.get(ad_creative, ad_creative.ad_creative_id == 35124) |
| 19:27.30 | mshooshtari | have headline = Unicode in my ad_creative class, but doing result.headline gives errors |
| 19:27.39 | radix | mshooshtari: it's not really that geared towards new databases. all you have to do is leave out the CREATE TABLE statements :) |
| 19:31.45 | mshooshtari | print store.get(ad_creative, 35124).headline |
| 19:32.05 | mshooshtari | so the 35124 is the ad_creative_id (primary key) |
| 19:32.59 | radix | mshooshtari: it would be helpful to know what the errors are |
| 19:33.11 | mshooshtari | pymssql.DatabaseError: internal error: None |
| 19:33.42 | *** join/#storm rreis (n=rreis@c953f05a.virtua.com.br) |
| 19:33.51 | mshooshtari | it's not very helpful I know |
| 19:36.31 | *** join/#storm fcorrea (n=fcorrea@c953f05a.virtua.com.br) |
| 19:44.41 | mshooshtari | http://www.pastebin.ca/1074649 |
| 19:44.54 | mshooshtari | Here is the code I have |
| 19:52.52 | mshooshtari | I think I can fix it but it's a problem where the query is generated |
| 19:59.39 | radix | hmm |
| 19:59.51 | mshooshtari | have the raw stuff |
| 20:16.07 | mshooshtari | ha! |
| 20:16.13 | mshooshtari | Internal error my ass |
| 20:16.45 | mshooshtari | there was this crap raise DatabaseError, "internal error: %s" % self.__source.errmsg() |
| 20:16.49 | mshooshtari | which hid the real error |
| 20:16.55 | mshooshtari | TypeError: not all arguments converted during string formatting |
| 20:17.15 | mshooshtari | why raise an exception when it's less descriptive than the real error |
| 20:18.14 | radix | where was that? |
| 20:19.07 | mshooshtari | pymysql line 134-152 |
| 20:19.22 | mshooshtari | I think it goes back to types though |
| 20:21.10 | mshooshtari | the parameters have an extra comma |
| 20:21.31 | mshooshtari | return s % params params = (35124,) |
| 20:21.45 | mshooshtari | that number is the primary key used in the get |
| 20:25.07 | mshooshtari | <PROTECTED> |
| 20:25.07 | mshooshtari | <PROTECTED> |
| 20:25.48 | mshooshtari | Have any Idea why that would be that way? |
| 20:25.59 | mshooshtari | I'm just trying to trace the problemb back, and it looks like that's the source there |
| 20:26.34 | *** part/#storm lgs (n=lgs@217.216.91.110.dyn.user.ono.com) |
| 20:28.44 | mshooshtari | niemeyer, Do you have time to explain some things about store.py? |
| 20:31.05 | mshooshtari | composed keys...mmm |
| 20:32.06 | mshooshtari | it looks like Storm expects keys to be tuples, if it's not it converts them, maybe the ms part does not expect that. |
| 20:34.14 | mshooshtari | in fact it appears to be that way |
| 20:34.23 | mshooshtari | pymssql does not expect primary keys to be tuples |
| 20:34.37 | mshooshtari | so the handoff needs to change it |
| 20:41.03 | niemeyer | mshooshtari: Sure, what's up? |
| 20:41.43 | mshooshtari | I was wondering why the primary key was changed to a tuple, and at what point it would be appropriate to cange it back for the purposes of pymssql |
| 20:42.33 | mshooshtari | I thought primary keys could only be one value |
| 20:44.48 | niemeyer | mshooshtari: Sorry, I'm missing some context |
| 20:45.11 | niemeyer | mshooshtari: Primary keys may be one value or multiple values, just like on the actual database |
| 20:45.35 | mshooshtari | <PROTECTED> |
| 20:45.35 | mshooshtari | <PROTECTED> |
| 20:46.32 | mshooshtari | return s % params |
| 20:46.56 | mshooshtari | the first two lines are yours from store.py the last is from pymssql.py |
| 20:48.21 | mshooshtari | going to pastebin the specific area from pymysql |
| 20:48.27 | mshooshtari | It's above me |
| 20:48.50 | mshooshtari | http://www.pastebin.ca/1074726 |
| 20:49.16 | mshooshtari | this code of mine is failing because of those conflicts. store.get(ad_creative, 35124).headline |
| 20:53.23 | niemeyer | mshooshtari: I'm having a hard time reaching your intentions |
| 20:53.51 | niemeyer | mshooshtari: You've asked about primary key, then there's a paste of some quote function, and also some params, then there's a conflict about something |
| 20:54.19 | niemeyer | mshooshtari: Would you be able to assume that I know nothing about Storm and then describe what you're trying to achieve? |
| 20:55.25 | mshooshtari | I have a good assumption that you are the original author for store.py and database.py |
| 20:56.02 | mshooshtari | I know you did not have much to do with the work done to support mssql. |
| 20:56.32 | radix | mshooshtari: the problem is you're not communicating your problem very clearly |
| 20:56.51 | mshooshtari | I apologize, I'm having a hard time with this too. |
| 20:58.02 | mshooshtari | pymysql is trying to parse the params provided by storm and is trowing a TypeError because the tuple has 2 parameters (25124,) and it is trying to only parse out 1. |
| 20:58.13 | mshooshtari | s % params |
| 20:59.00 | radix | pymysql does string interpolation for bind parameters? |
| 20:59.10 | radix | or wait, do you mean pymssql? |
| 20:59.18 | mshooshtari | pymysql does this |
| 20:59.52 | radix | mshooshtari: so, that _quoteparams function is from pymysql source code? |
| 20:59.54 | mshooshtari | The function I pasted was where it does the interpolation |
| 20:59.58 | radix | ok |
| 20:59.59 | mshooshtari | yes |
| 21:00.16 | radix | hm |
| 21:00.24 | mshooshtari | I don't understand that interpolation, I think it may be wrong, but I dont know enough to say that. |
| 21:00.45 | radix | well, unless I'm mistaken about this code, my recommendation would be to stay far, far away from pymysql |
| 21:01.07 | radix | because string interpolation is a terrible idea for SQL |
| 21:02.11 | mshooshtari | What about ODBC? do you think I might be able to use that here? |
| 21:02.33 | mshooshtari | I think I saw an ODBC driver that worked with python. |
| 21:02.50 | radix | mshooshtari: Why not use MySQLdb, which is what Storm already supports? |
| 21:03.13 | mshooshtari | I have a legacy mssql database :( |
| 21:03.27 | mshooshtari | I was thinking about this one. http://pyodbc.sourceforge.net/ |
| 21:03.32 | radix | wait, mssql? |
| 21:03.35 | radix | I'm really confused, Iguess |
| 21:03.48 | radix | Is the module "pymysql" actually an interface for MS SQL? |
| 21:04.25 | mshooshtari | pymysql is an interface to mysql, pymssql is an interface for MS SQL |
| 21:04.46 | radix | I don't understand why we're talking about pymysql at all then |
| 21:04.52 | radix | if you're trying to integrate with an MS SQL database |
| 21:05.58 | mshooshtari | I have confused you mshooshtari: so, that _quoteparams function is from pymysql source code --- Corection, for this _quoteparams is a function of pymssql |
| 21:06.10 | mshooshtari | I think that's where I got you confused |
| 21:06.37 | mshooshtari | _quoteparams is a function of pyMSsql |
| 21:06.49 | mshooshtari | that's what uses string interpolation |
| 21:07.10 | mshooshtari | I agree that it's a good Idea to stay away from pymssql if it does that. |
| 21:08.21 | *** join/#storm bigdo2 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 21:08.38 | radix | I see |
| 21:08.44 | radix | ok |
| 21:09.10 | radix | mshooshtari: so are you one of the developers of the MS SQL backend for storm? |
| 21:09.41 | mshooshtari | I'm a developer trying to use storm and it's MS SQL backend |
| 21:09.53 | mshooshtari | I was going to change the driver from pymssql to pyodbc though |
| 21:10.04 | radix | Which MS SQL backend, in particular? there's none in trunk. |
| 21:10.32 | mshooshtari | https://code.launchpad.net/~sidnei/storm/mssql-support |
| 21:10.36 | radix | and in fact, right now there are two different branches which are trying to implement a MS SQL backend |
| 21:10.38 | radix | ok |
| 21:11.10 | radix | so, there has been some discussion about these two branches on the mailing list recently |
| 21:11.12 | radix | have you followed it? |
| 21:12.24 | radix | mshooshtari: the other branch supports PyODBC |
| 21:12.38 | radix | https://code.edge.launchpad.net/~zeomega/storm/storm_ms_sql_2005 |
| 21:17.41 | mshooshtari | adodbapi |
| 21:17.46 | mshooshtari | seems to be recomended too |
| 21:18.35 | mshooshtari | but not platform independent |
| 21:57.50 | *** part/#storm bigdo2 (n=scmikes@72-197-8-8-arpa.cust.cinci.current.net) |
| 22:57.12 | *** join/#storm synic (n=squish@pdpc/supporter/student/synic) |
| 22:58.13 | synic | I have a question. store.get(SomeObject, 8) seems horribly slow, especially if I have to get, say, 15 different objects from a list of 5000 |
| 22:58.25 | synic | it takes 10 seconds or so just to get these 15 objects |
| 23:06.13 | jkakar | synic: Are these 5000 objects in memory already when you call store.get? |
| 23:07.52 | synic | unlikely |
| 23:11.08 | jkakar | synic: Is it possible that the queries are slow because of table scans or something? |
| 23:11.31 | synic | I'm not sure, seems like it should always be fast on the primary key |
| 23:11.34 | synic | there's only one table |
| 23:12.13 | jkakar | synic: Are you using trunk? |
| 23:12.27 | synic | no |
| 23:13.04 | jkakar | synic: With trunk you can call storm.tracer.debug(True) to cause Storm to write out executed statements (with timing) to stdout. This may help determine if the issue is on the Python side or on the database side. |
| 23:13.19 | synic | ok |
| 23:13.22 | jkakar | synic: But yeah, the assumption that lookup by primary key should be fast is correct I think. |
| 23:13.41 | radix | on older versions you can use storm.database.DEBUG = True to turn on statement printing |
| 23:14.07 | radix | synic: and are you sure that time is spent in store.get, and not in some other part of your application? |
| 23:14.15 | synic | yeah |
| 23:23.33 | jkakar | synic: If you can produce a reasonably self-contained test case that demonstrates this behaviour, that'd be very helpful. |
| 23:23.40 | synic | k |
| 23:24.04 | jkakar | Ta! |