| 00:03.15 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 01:01.13 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 01:20.22 | *** join/#asterisk-dev tm1000 (sid6728@gateway/web/irccloud.com/x-exahspurbfzpvikv) |
| 01:22.23 | *** join/#asterisk-dev Synthase_ (uid63346@gateway/web/irccloud.com/x-gwsrkwhntfgazzkm) |
| 02:26.06 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 03:35.48 | *** join/#asterisk-dev snuff-work (~blatus@203.87.61.22) |
| 03:35.48 | *** mode/#asterisk-dev [+o snuff-work] by ChanServ |
| 03:40.16 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-tnzienedwbfhsvoo) |
| 03:40.16 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 04:16.22 | *** part/#asterisk-dev mjordan (mjordan@nat/digium/x-tnzienedwbfhsvoo) |
| 05:38.28 | drmessano | m |
| 06:16.04 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 06:16.04 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 06:22.00 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 06:22.00 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 06:24.00 | *** join/#asterisk-dev tm1000 (sid6728@gateway/web/irccloud.com/x-fjsdlcqwcartzzyg) |
| 06:51.29 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 06:51.29 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 07:09.42 | *** join/#asterisk-dev hehol (~hehol@2001:1438:1009:200:7c02:587e:414a:11c6) |
| 07:19.41 | *** join/#asterisk-dev pchero_work (~pchero@109.70.54.56) |
| 08:52.43 | *** join/#asterisk-dev WIMPy (~wimpy@e183095026.adsl.alicedsl.de) |
| 09:54.41 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 09:54.41 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 11:20.13 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 11:20.13 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 12:22.25 | *** join/#asterisk-dev seanbright (~sean@asterisk/contributor-and-bug-marshal/seanbright) |
| 12:22.25 | *** mode/#asterisk-dev [+o seanbright] by ChanServ |
| 12:23.26 | Greenlight | Hey there! I'm trying to further debug some ongoing stability issues we're getting in Asterisk 11, under rather heavy load. I seem to have two distinct issues, although I'm unsure if they're related or not. The first is a deadlock occuring on a seemingly daily basis now, and the second is all calls suddenly going crackly with asterisk "timing test" show horrid results. With the second |
| 12:23.26 | Greenlight | issue, in the backtrace I can see a LOT (200+) of threads at ast_waitfor_nandfds... any pointers as to what this could be indicative of? |
| 12:28.08 | file | that's used for waiting on 1 or 2 channels |
| 12:33.18 | Greenlight | Waiting for the channel do what? |
| 12:35.39 | file | waiting for the channel to do anything - provide audio, DTMF, hangup |
| 12:39.56 | Greenlight | I see. Thanks. Perhaps this is more another symptom of the issue than a possible cause then. I've got hundreds of threads, even just doing fast_originate "waiting" |
| 12:40.49 | file | you'll have 1 thread per channel, except for situations where you do a Dial() in which case 1 thread will handle 2 channels |
| 12:43.37 | Greenlight | Okay, so perhaps 600 threads with a load in that state isn't too unusual |
| 12:45.04 | Greenlight | "timing test" was reporting stuff like 1024 milliseconds for 14 ticks; and customers reported calls going very choppy |
| 12:45.09 | *** join/#asterisk-dev [TK]D-Fender (~chatzilla@216-191-106-163.dedicated.allstream.net) |
| 12:59.26 | file | yeah, if you have tons of channels you'll have tons of threads - that's normal |
| 13:15.23 | *** join/#asterisk-dev Marquis42 (~mrbmw@d14-69-246-193.try.wideopenwest.com) |
| 13:27.20 | *** join/#asterisk-dev snuff-home (~brad@203-219-29-182.tpgi.com.au) |
| 13:27.20 | *** mode/#asterisk-dev [+o snuff-home] by ChanServ |
| 13:28.28 | *** join/#asterisk-dev russellb (~russellb@redhat/russellb) |
| 13:28.28 | *** mode/#asterisk-dev [+o russellb] by ChanServ |
| 13:38.15 | *** join/#asterisk-dev newtonr (RustyNewto@nat/digium/x-ghqddgnarubfrirb) |
| 13:38.15 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 13:44.12 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-bdypyzbgsadrwwgk) |
| 13:49.04 | *** join/#asterisk-dev putnopvut (putnopvut@asterisk/master-of-queues/mmichelson) |
| 13:49.04 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 14:00.03 | tzafrir | https://git.asterisk.org/ gives an error here (the HTTP variant works fine) |
| 14:09.45 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-xidbtudtmyjjppaz) |
| 14:24.31 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 14:24.32 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 14:33.02 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-gruiljdsnltnwbig) |
| 14:33.02 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 14:36.21 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-vwchvgzspsjrwthn) |
| 14:37.04 | *** join/#asterisk-dev gtjoseph (~gtj0@unaffiliated/gtj) |
| 14:57.14 | *** join/#asterisk-dev sgriepentrog (sgriepentr@nat/digium/x-yddjucgydnhohzbx) |
| 15:01.56 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-edpretkebnnjohgs) |
| 15:33.16 | gtjoseph | hey file: is there a reason that permanent contacts get kept in aor->permanent_contacts instead of being added to sorcery? |
| 15:34.30 | file | no explicit reason except I didn't think they belonged there, it's not an option to state to add them to sorcery - it's that any queries for an AOR should also return them |
| 15:35.07 | file | you'd also need to reconcile the sorcery side if the config changed and you restarted |
| 15:36.36 | gtjoseph | ok, no biggie now, just requires some duplicated code. |
| 15:37.32 | gtjoseph | I.E they don't get handled by an apply handler or observers. |
| 15:44.31 | *** join/#asterisk-dev superscrat (asanders@nat/digium/x-xfvreptshcycfpmd) |
| 16:26.33 | Greenlight | So, I've kinda further tracked down an occurance of my ongoing deadlock issue (https://issues.asterisk.org/jira/browse/ASTERISK-25000). I've the devicestate thread holding the channel container lock (c), that is in turn locked against a channel. What's odd is the channel it's locked on is in the process of being bridged, and devicestate seems to think the channel tech is "Bridge", presumably |
| 16:26.33 | Greenlight | because of the channel name: "DEBUG[12268] devicestate.c: No provider found, checking channel drivers for Bridge - SIP/fep-ast-01". Is it normal behaviour for device state to be working out states for such channels? |
| 17:53.23 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.dynamic.se.alltele.net) |
| 17:53.23 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 17:58.41 | *** join/#asterisk-dev dustinm (~dustinm@105.ip-167-114-152.net) |
| 18:36.28 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 18:36.28 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Hell hath frozen over: Asterisk is moving to Git -=- Follow on Twitter at @AsteriskDev |
| 18:55.45 | *** join/#asterisk-dev superscrat (~asanders@173-17-133-2.client.mchsi.com) |
| 19:06.04 | *** join/#asterisk-dev newtonr (RustyNewto@nat/digium/x-fbbrbprehgerpvrq) |
| 19:06.04 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 19:13.25 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-osuqgujqusprffoz) |
| 19:32.51 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-218-28-58.cablep.bezeqint.net) |
| 19:32.51 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 20:01.39 | coreyfarrell | what's the best way to publish a WIP branch that is based on another gerrit review? I just posted https://wiki.asterisk.org/wiki/display/~coreyfarrell/Module+Loader+Ideas and have a 'functional' branch implementing it, but it's based on gerrit review 140 instead of master. do I need to just wait for review 140 to complete? |
| 20:19.23 | leifmadsen | you can link reviews to be dependent |
| 20:19.48 | leifmadsen | however it's been so long that I don't remember exactly how |
| 20:20.09 | Greenlight | I don't understand why in ast_do_masquerade we ao2_link the zombie channel to the channel container, surely we'd want to be unlinking it to ensure nothing touches it |
| 20:40.35 | DivideBy0 | leifmadsen: I read the 4th edition of your definitive guide on a plane this weekend. It's an EXCELLENT book. even by O'reilly standards. Thanks for making it |
| 20:40.59 | DivideBy0 | O'reilly usually has good stuff, but it was great |
| 20:41.03 | leifmadsen | thanks! glad you enjoyed it! I had a lot of help (I barely did any work) |
| 20:41.20 | leifmadsen | russellb, jsmith, and Jim van Meggelen did a lot :) |
| 20:41.50 | DivideBy0 | you're the only one I've actually talked to here on IRC, so you're the "front-man" in my mind |
| 20:42.00 | russellb | DivideBy0: glad you enjoyed it |
| 20:53.01 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 20:53.41 | rmudgett | Heh. Menuselect is broken. I get this when I do make menuselect: "The existing menuselect.makeopts file did not specify... Either run make menuselect or ..." |
| 21:01.36 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 21:02.22 | DivideBy0 | russellb: If Leif is the front man, you can be the drummer |
| 21:02.47 | russellb | Leif is the puppet |
| 21:02.50 | russellb | the rest of us did the real work |
| 21:03.48 | DivideBy0 | haha, so my rock band analogy is actually more accurate than I thought |
| 21:04.37 | leifmadsen | rmudgett, thought there was a known issue (at least last I saw) that you needed to checkout menuselect manually or something... ? |
| 21:04.42 | leifmadsen | thought I saw some comment about that |
| 21:04.51 | leifmadsen | russellb, DivideBy0 +1 |
| 21:05.35 | russellb | DivideBy0: fun fact, leifmadsen and I are both drummers IRL. |
| 21:05.41 | russellb | i'm better though |
| 21:05.44 | file | if you switch between branches menuselect can get unhappy because the menuselect.makeopts may contain stuff not relevant to the version |
| 21:07.39 | rmudgett | file: This was on master in a workspace that I only use for master. It was complaining about test_dns_query_set. We'll see if this happens again. :) |
| 21:08.53 | DivideBy0 | it's interesting to me how many a-level programmers are also musicians |
| 21:11.48 | Greenlight | NewtonR: In response to your reply to https://issues.asterisk.org/jira/browse/ASTERISK-25000, no unfortuntaly DEBUG_THREADS kills the server very quickly |
| 21:13.44 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 21:38.03 | coreyfarrell | rmudgett: any suggestion for the best tests to run for the astobj2 change? so far I ran the testsuite manager/original, pjsip/basic_calls/incoming/nominal all passed. if you want me to do a full run I will but since everything uses ao2 it shouldn't take too many tests to get good coverage. |
| 21:39.36 | rmudgett | coreyfarrell: There is the ao2 unit tests. |
| 21:41.37 | coreyfarrell | rmudgett: so those plus the stuff I did against the testsuite should be good? I'm not sure I'll be able to run the thrash tests, I've had issues with them in the past, but that was probably from having REF_DEBUG enabled. |
| 21:43.28 | rmudgett | Yes. That should be good enough. |
| 21:44.54 | *** join/#asterisk-dev putnopvut (putnopvut@asterisk/master-of-queues/mmichelson) |
| 21:44.55 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 21:45.03 | *** part/#asterisk-dev mjordan (mjordan@nat/digium/x-gruiljdsnltnwbig) |
| 22:24.24 | newtonr | Greenlight, ah okay, thanks |
| 22:25.12 | Greenlight | It may sound odd, but from my backtraces, it's like a thread is taking the lock, and then not releasing it, and ending. |
| 22:26.13 | Greenlight | I'm unsure if it's possible for __ao2_unlock to fail? |
| 22:26.45 | newtonr | Greenlight, I'm not a dev so I can't interpret the backtraces. I'll ask mjordan or someone else to take a look when they are able. |
| 22:26.59 | Greenlight | Okay, thanks! |
| 22:27.22 | newtonr | I just know the "core show locks" output is often most helpful in the case of locking issues |
| 22:28.19 | Greenlight | Yup, it would make things easier! I've been going over the backtraces and the code for the last 12 hours or so, and can't for the life of me see where it could be going wrong |
| 22:33.10 | *** join/#asterisk-dev superscrat (~asanders@173-17-133-2.client.mchsi.com) |
| 22:35.58 | rmudgett | coreyfarrell: Was there a reason you didn't change the REF_DEBUG flag bit to 20? |
| 22:37.14 | coreyfarrell | uh thought I did.. give me a minute I'll switch back to that branch and update |
| 22:38.49 | coreyfarrell | rmudgett: updated |
| 22:38.56 | rmudgett | k |
| 22:47.29 | rmudgett | coreyfarrell: +1'd the review |
| 22:49.09 | coreyfarrell | rmudgett: thanks! |
| 23:22.20 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-xidbtudtmyjjppaz) |