| 02:27.37 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 03:02.39 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 03:16.19 | *** join/#asterisk-dev Devon_ (~Devon_@2601:586:8003:6bc3:9170:b7ab:3ed8:3d9e) |
| 03:22.11 | *** join/#asterisk-dev Devon_ (~Devon_@2601:586:8003:6bc3:9170:b7ab:3ed8:3d9e) |
| 03:27.37 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@wsip-98-179-138-226.ri.ri.cox.net) |
| 05:05.23 | *** join/#asterisk-dev babak (uid19622@gateway/web/irccloud.com/x-itebsiqgwtvdnega) |
| 05:47.00 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-179-40-172.cust.bezeqint.net) |
| 05:47.00 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 07:11.54 | *** join/#asterisk-dev pchero (~pchero@109.70.54.56) |
| 07:28.46 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 07:38.23 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 07:38.23 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 07:45.30 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 07:45.30 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 08:12.21 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 08:12.21 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 08:30.10 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 08:30.10 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 08:46.42 | oej | Good morning folks! |
| 08:59.25 | *** join/#asterisk-dev wdoekes (~walter@wjd.osso.nl) |
| 09:04.56 | *** join/#asterisk-dev pchero_work (~pchero@109.70.54.56) |
| 12:45.04 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 12:45.04 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 13:01.34 | *** join/#asterisk-dev Marquis42 (~mrbmw@d192-24-112-5.try.wideopenwest.com) |
| 13:15.04 | *** join/#asterisk-dev [TK]D-Fender (~chatzilla@216-191-106-163.dedicated.allstream.net) |
| 13:48.36 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-jfbbymzshbtadkzp) |
| 14:43.10 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-tcnmlgkiocgirdii) |
| 15:32.41 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-welxzeiglvwmrlbp) |
| 15:49.07 | *** join/#asterisk-dev moy (sid47040@gateway/web/irccloud.com/x-ddkatyghdbkeasjz) |
| 16:20.29 | *** join/#asterisk-dev Stefan26 (Stefan27@195-67-255-105-no207.tbcn.telia.com) |
| 16:26.21 | Kobaz | sooo |
| 16:26.29 | Kobaz | happy monday |
| 16:27.02 | file | it is Monday, this is true |
| 16:32.17 | Stefan26 | I disconnected yesterday, and I cant access irc logs; did anyone respond to my question about pj_ice_sess_destroy possibly racing with pj_ice_sess_on_rx_pkt by any chance? |
| 16:32.47 | file | negative |
| 16:32.56 | Kobaz | mm |
| 16:32.56 | Kobaz | nope |
| 16:32.58 | WIMPy | NOpe |
| 16:33.01 | Kobaz | don't know the pj internals |
| 16:33.09 | Kobaz | did anyone ponder my ast_db_put issues? |
| 16:33.23 | Stefan26 | my goal is to have a stable temporary solution until some ast_developer finds a permenant solution, much like the dtls issue from some months back |
| 16:34.17 | Kobaz | i'm going to go play with replacing astdb with an in-memory hashtable |
| 16:34.30 | Kobaz | and then leave the sqlite part for storing to disk |
| 16:36.59 | Stefan26 | Mark Michelson seems to be the assignee on the issue which is related to my issue, does he use IRC? I'm curious who wrote the comment "Will this work?" in res_rtp_asterisk before calling ice_session_reset. I'm thinking about testing removing that ice_session_call altogether, or adding a mutex so pj_ice_sess_destroy possibly can't be executed at the same time as pj_ice_sess_on_rx_pkt on |
| 16:36.59 | Stefan26 | same rtp->ice |
| 16:37.24 | file | you can do a git annotate to trace where it came from |
| 16:38.21 | Stefan26 | OK cool, I'll have to do that when I get to work tomorrow, I dont have any shell setup with git on this device |
| 16:44.04 | Stefan26 | according to most programming books, debugging by scattering output-logs throughout a program is a very bad debugging practice |
| 16:47.50 | Stefan26 | I can't find a better alternative though, trying to fully understand all possible interactions is too difficult and I can't reproduce it in development asterisks |
| 16:48.04 | file | gotta do what you gotta do |
| 16:54.21 | Kobaz | :( |
| 16:54.24 | Kobaz | no one cares |
| 16:54.26 | Kobaz | poor astdb |
| 16:54.35 | Marquis42 | lol |
| 16:54.45 | Marquis42 | I only use astdb when/if I have to. :) |
| 16:54.50 | Kobaz | well |
| 16:54.51 | Kobaz | hsh |
| 16:54.52 | Kobaz | heh |
| 16:54.56 | Kobaz | chan_sip uses it all the time |
| 16:55.06 | Kobaz | personally yeah, dialplan/whatever i use postgres for db |
| 16:55.20 | Kobaz | but i'm having major manor performance problems with 200 phones and astdb storing registrations |
| 16:55.21 | Marquis42 | s/astdb/chan_sip/ in my above statement. ;) |
| 16:55.23 | Kobaz | with -11 |
| 16:55.36 | Kobaz | whoops major major |
| 16:56.00 | Kobaz | my fix is working so far |
| 16:56.08 | Kobaz | and "fixed" the issue |
| 16:56.09 | Kobaz | [2015-11-02 11:54:16.137] ERROR[25333]: db.c:335 ast_db_put: ast_db_put() COULD NOT LOCK. Abandon. |
| 16:56.13 | Stefan26 | Yeah, I haven't touched 95% of the ast features, that's why I didn't even comment on your ast_db :) |
| 16:56.18 | Kobaz | hehe |
| 16:56.39 | file | astdb itself hasn't been touched in ages and I haven't seen any issues reported around that, so you'd have to understand why and what is going on |
| 16:57.52 | Stefan26 | Kobaz, are you a developer like file? |
| 16:58.57 | Kobaz | i used to be more involed, but now i gotta get back in since i'm moving production to 11 |
| 16:59.04 | Kobaz | i wrote a bunch of stuff a few years ago |
| 16:59.20 | Stefan26 | were you employed by Digium? |
| 17:00.37 | Kobaz | it's been super-super solid on 1.8 until now |
| 17:00.38 | Kobaz | nope |
| 17:00.49 | Kobaz | rustleb wanted to hire me |
| 17:00.49 | Kobaz | hehe |
| 17:02.19 | Kobaz | so weird, like i've had the same version of 1.8 on this one box since 2013 |
| 17:02.38 | Kobaz | rock solid, up until two weeks ago where it would just lock up three-four times a day |
| 17:03.21 | Kobaz | and i did what i could to keep things the same, or mostly the same |
| 17:03.39 | Kobaz | i bumped up to 1.8.32.3 |
| 17:03.57 | Kobaz | and then it still didn't go, so i'm like well, try the next thing, port all my stuff to asterisk-11 |
| 17:04.13 | Kobaz | that took two and a half days and that didn't fix the problem, db still freezing except much worse |
| 17:04.19 | Kobaz | so i did what i could and bandaided it |
| 17:04.31 | Kobaz | and now looking for the proper solution |
| 17:04.37 | file | have you fully checked out the system? |
| 17:04.47 | Stefan26 | I'm not really a part of the ast-dev community, I do most of my work outside asterisk, but often we want to tweak the behaviour of asterisk, and I sometimes discover bugs. It's just always a dilemma when i find bugs - I could create a jira and pray someone solves the issue within x weeks, or i can try attack it myself, but it's sometimes inefficient time spent, because some bugs are |
| 17:04.47 | Stefan26 | obscure and other bugs turn out not to be bugs just different behavior than what we need |
| 17:05.16 | Kobaz | file: fully checked out the what in the who where? |
| 17:05.22 | file | the system Asterisk is running on |
| 17:05.26 | Kobaz | oh |
| 17:05.30 | Kobaz | in terms of what, like OS |
| 17:05.34 | Kobaz | everything looks pretty good |
| 17:05.35 | file | hardware |
| 17:05.39 | file | stuff just doesn't stop working |
| 17:05.40 | Kobaz | i haven't done a reboot |
| 17:05.43 | Kobaz | i know |
| 17:05.48 | Kobaz | nothing else is crashing |
| 17:05.50 | Kobaz | no disk io errors |
| 17:06.01 | Kobaz | i recreated the astdb file on a different inode |
| 17:06.16 | Kobaz | kernel thread load looks all sane |
| 17:06.18 | file | I'd say it's something to do with the system tbh |
| 17:06.24 | Kobaz | yeah like |
| 17:06.26 | Stefan26 | do you have configuration under version control? |
| 17:06.38 | Kobaz | something is making the disk write take longer than it should |
| 17:06.42 | Kobaz | that's my only theory so far |
| 17:07.00 | Kobaz | when astdb background writer kicks off, it locks for a considerably long time |
| 17:07.06 | Kobaz | and therefore locks up all writers |
| 17:07.11 | file | unless you fix and identify that you're just moving the problem around |
| 17:07.42 | Kobaz | Stefan26: my local branch is in git |
| 17:07.59 | Kobaz | Stefan26: for source code, and yeah we version our db system and everything |
| 17:08.13 | Kobaz | none of that has changed since before/after the problem |
| 17:08.44 | Kobaz | file: but, the thing is, disk io should ideally never freeze up asterisk |
| 17:08.48 | Stefan26 | everyone version controls source code, some ppl forget to version control OS-conf changes |
| 17:09.02 | Kobaz | like, Monitor() for instance |
| 17:09.10 | file | in a subset of the world, yes |
| 17:09.20 | Kobaz | disk io definitely did affect calls with that, so MixMonitor came around |
| 17:10.09 | Kobaz | so whatever code i'm wrapping to protect ast_db_put, i think is ideal in the real world |
| 17:10.25 | Kobaz | because in the real world, disk io happens, thrashing happens, and we should protect as much as we can against it |
| 17:11.04 | file | be weary of licensing if you plan to contribute... |
| 17:11.19 | file | and you're incorporating outside stuff at all |
| 17:11.22 | Kobaz | yeah |
| 17:11.47 | Kobaz | i was looking at trying to find a gpl multithreaded lockless hash table |
| 17:12.10 | file | Asterisk is dual licensed, has to be more permissive |
| 17:12.19 | Kobaz | ah right |
| 17:12.57 | Kobaz | bsd license? |
| 17:13.15 | Kobaz | and then maybe the author can also offer a gpl so that part's compatible? |
| 17:13.32 | Kobaz | not sure how that works, never did a 3rd party thing with ast |
| 17:13.46 | file | BSD is fine, but I reiterate that while this is useful you REALLY need to find the cause |
| 17:13.53 | Kobaz | yeah |
| 17:13.56 | Kobaz | i agree |
| 17:15.02 | Kobaz | === ---> Lock #0 (db.c): MUTEX 1029 db_sync_thread &dblock 0x8214f60 (1) |
| 17:15.09 | Kobaz | that lock is held for at least a second |
| 17:15.31 | Kobaz | and then when you have like 30 sip registrations coming in, they all want the lock |
| 17:15.40 | Kobaz | and then the sip thread doesn't do anything else in the meantime |
| 17:15.44 | file | yes. |
| 17:16.05 | Kobaz | and then you wind up with an out of control load issue |
| 17:16.22 | Kobaz | then the db locks again after maybe like 2 regs squeek in |
| 17:16.40 | Kobaz | an then more build up, and then everything slows down to the point nothing finishes and then you just can't make any calls anymore |
| 17:17.00 | file | yes. |
| 17:17.22 | Kobaz | so i'm like 99% certain i've isolated it to that flow |
| 17:17.41 | file | those are symptoms, not a cause |
| 17:17.47 | Kobaz | well yeah |
| 17:18.08 | Kobaz | well it's a secondary cause |
| 17:20.38 | Kobaz | oh man |
| 17:21.00 | Kobaz | dblock being held for 20+ seconds |
| 17:21.06 | Kobaz | it has to be disk io |
| 17:23.29 | *** join/#asterisk-dev gtjoseph (~gtj0@unaffiliated/gtj) |
| 17:24.43 | Kobaz | there's not much going on |
| 17:24.53 | Kobaz | OS-wise |
| 19:15.44 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-rtpbuhpoxbupsqet) |
| 19:30.52 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 19:30.52 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 19:58.20 | *** join/#asterisk-dev pchero (~pchero@109.70.54.56) |
| 20:47.50 | *** join/#asterisk-dev Dovid (~dovid@static-173-63-105-210.nwrknj.fios.verizon.net) |
| 20:48.01 | Dovid | Hi. If we paid can we have this looked at sooner and worked? https://issues.asterisk.org/jira/browse/ASTERISK-25307 |
| 20:51.58 | file | that's up to who you find to pay |
| 21:03.24 | nbjoerg | it would be nice to find someone to look at 22920 again |
| 21:09.49 | Dovid | file: r u for hire ? ;) |
| 21:09.54 | file | no |
| 21:18.49 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-179-40-172.cust.bezeqint.net) |
| 21:18.49 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 22:36.29 | WIMPy | Dovid: You could always add some "arguments" to the issue. ;-) |
| 22:37.00 | WIMPy | I think there is a chance of that being a rather easy thing to fix. |
| 22:38.27 | WIMPy | And adding some configs to easily reproduce the issue would surely help. |
| 22:40.06 | Dovid | WIMPy: Sorry. been very busy |
| 22:41.11 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 23:31.39 | WIMPy | Oh, this is evil. |
| 23:32.15 | WIMPy | asterisk -x "pri set debug file ..." won't return until the file is being opened for reading by something. |
| 23:48.43 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-tcnmlgkiocgirdii) |