| 00:20.21 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 00:26.26 | *** join/#storm sidnei (n=sidnei@plone/dreamcatcher) |
| 03:46.08 | *** join/#storm WebMaven (n=webmaven@c-68-35-136-143.hsd1.nm.comcast.net) |
| 05:56.23 | *** join/#storm seiflotfy (n=seif@41.234.24.162) |
| 11:20.56 | *** join/#storm kamstrup (n=mikkel@0x3e42da90.adsl.cybercity.dk) |
| 11:36.04 | *** join/#storm thekorn (n=markus@a89-182-192-85.net-htp.de) |
| 11:42.58 | kamstrup | I am seeing Storm perform ~10 times slower than raw Sql for inserting objects... Is that usual? I have a very simple test case here : http://pastebin.com/m4d40f5a3 |
| 11:43.47 | *** join/#storm RainCT (n=RainCT@ubuntu/member/rainct) |
| 11:45.19 | therve | kamstrup: it's probably the order of magnitude |
| 11:45.29 | therve | don't forget you're creating object, vs creating strings |
| 11:45.44 | therve | you may want to try the c extension too, can reduce that gap a little bit |
| 11:50.07 | kamstrup | therve: Creating 1000 objects in Python should not be a big deal i suspect... |
| 11:50.55 | *** join/#storm seiflotfy1 (n=seif@41.234.24.162) |
| 11:53.27 | kamstrup | therve: This updated example proves that I am right: http://pastebin.com/m44debadb |
| 11:54.35 | *** join/#storm niemeyer (n=niemeyer@200-103-135-47.ctame705.dsl.brasiltelecom.net.br) |
| 12:01.22 | kamstrup | The example here with tracing enabled also shows exactly why Storm is so slow: |
| 12:01.23 | kamstrup | http://pastebin.com/m326f2cf9 |
| 12:04.31 | kamstrup | And here is the output: http://pastebin.com/m42a97fa7 |
| 12:05.45 | kamstrup | (storm does SELECT/INSERT on each op. against doing raw INSERT) |
| 12:20.44 | therve | kamstrup: the updated example shows that the difference is much less |
| 12:20.58 | therve | I didn't say that the whole difference comes from creating objects |
| 12:25.11 | *** join/#storm RainCT_ (n=RainCT@62.175.149.58.dyn.user.ono.com) |
| 12:27.10 | kamstrup | The difference is x10 for ~1000 insert, if we go to 100,000 insert I don't even want to wait for storm... |
| 12:28.13 | kamstrup | The point being: If you want to insert a lot of items, and do it fast, then it is bonkers to do a SELECT for each INSERT |
| 12:28.51 | kamstrup | If you are on a SSD then it might be a huge problem (but still a perf hit), but on a rotating disk it really nails you down |
| 12:34.13 | niemeyer | kamstrup: Which database is this? |
| 12:34.37 | kamstrup | niemeyer: sqlite |
| 12:35.35 | niemeyer | kamstrup: Storm needs a way to map the inserted row with the in memory object. Current versions of Postgres and MySQL won't do the INSERT/SELECT because there are alternatives like INSERT * RETURNING which allow querying the inserted unique id |
| 12:35.45 | niemeyer | kamstrup: The sqlite backend isn't there yet |
| 12:36.40 | niemeyer | kamstrup: Nevertheless, that's a requirement for any mapper.. not retrieving the inserted primary key somehow would mean not knowing how to address the inserted object |
| 12:36.59 | niemeyer | kamstrup: The alternative would be to create a primary key on the Python side, but we've opted not to do it |
| 12:37.13 | kamstrup | niemeyer: uh, I see... |
| 12:38.12 | niemeyer | kamstrup: Maybe there is a trick like INSERT/RETURNING in newer sqlites.. if/when that's the case, it's easy for us to update the backend to support it |
| 12:38.55 | kamstrup | niemeyer: I have only found http://www.sqlite.org/c3ref/last_insert_rowid.html |
| 12:39.08 | kamstrup | I don't know if that function is bound into Python |
| 12:40.10 | niemeyer | kamstrup: Hmmm, maybe it is |
| 12:40.35 | niemeyer | kamstrup: We must look into this again, thanks for bringing it up |
| 12:40.51 | therve | note that it won't solve everything |
| 12:41.02 | therve | storm will still me a lot slower than raw execute |
| 12:41.28 | niemeyer | Well, true.. A+B will always be slower than A alone |
| 12:42.38 | therve | yeah, we're just trying to measure B :) |
| 12:43.19 | kamstrup | niemeyer: Here you go: http://oss.itsystementwicklung.de/download/pysqlite/doc/sqlite3.html#sqlite3.Cursor.lastrowid |
| 12:45.36 | niemeyer | kamstrup: Cool, I'm pretty sure we can use that to prevent the SELECT |
| 12:45.48 | kamstrup | niemeyer: woohoo! |
| 12:49.02 | *** join/#storm andrea-bs (n=andrea@ubuntu/member/beeseek.developer.andrea-bs) |
| 12:52.54 | *** join/#storm vvinet (n=vince@smtpin.revolutionlinux.com) |
| 13:41.34 | *** join/#storm shaunm (n=shaunm@140.177.205.91) |
| 14:00.22 | *** join/#storm oubiwann (n=oubiwann@97-119-81-240.omah.qwest.net) |
| 14:34.16 | *** join/#storm artista_frustrad (n=artista_@201-35-2-242.ctame704.dsl.brasiltelecom.net.br) |
| 14:38.55 | *** join/#storm RainCT (n=RainCT@ubuntu/member/rainct) |
| 14:41.19 | *** join/#storm jukart (i=lovely@81.189.156.94) |
| 16:38.50 | *** join/#storm pyCube (n=jmayfiel@12.229.94.3) |
| 17:48.16 | *** join/#storm WebMaven (n=webmaven@c-68-35-136-143.hsd1.nm.comcast.net) |
| 17:59.16 | *** join/#storm rocksta`` (n=rockstar@starscream.eventuallyanyway.com) |
| 18:29.47 | *** join/#storm jukart (n=jukart@d91-128-122-26.cust.tele2.at) |
| 19:52.58 | *** join/#storm pyCube_ (n=jmayfiel@12.229.94.3) |
| 20:32.07 | *** join/#storm seiflotfy2 (n=seif@41.234.24.162) |
| 22:47.27 | *** join/#storm Busata (n=Dries_De@dD5776105.access.telenet.be) |