| 04:59.07 | Kobaz | insanity |
| 04:59.20 | Kobaz | i'm having more problems with ast_db_put |
| 04:59.23 | Kobaz | and it's not disk io anymore |
| 04:59.33 | Kobaz | i've removed the sync thread so it does not write to disk |
| 05:01.32 | Kobaz | http://pastebin.com/NcPwii99 |
| 05:01.41 | Kobaz | all of these locks are being held for more than 2 minutes |
| 05:01.45 | Kobaz | locks up all sip activity |
| 05:08.18 | Kobaz | i'm gonna switch back to 1.8 for tomorrow and use tmpfs for astdb storage. Hopefully that'll fix this problem for now |
| 05:59.27 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 05:59.27 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 07:54.25 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 08:27.39 | *** join/#asterisk-dev pchero_work (~pchero@109.70.54.56) |
| 13:07.15 | *** join/#asterisk-dev [TK]D-Fender (~chatzilla@216-191-106-163.dedicated.allstream.net) |
| 13:41.30 | file | moo |
| 14:04.13 | *** join/#asterisk-dev newtonr (RustyNewto@nat/digium/x-nolroznfbzrxjtoa) |
| 14:04.14 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 14:31.23 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-zuvsmqcttropffjf) |
| 14:38.46 | leifmadsen | mooo urns |
| 15:28.58 | *** join/#asterisk-dev superscrat (asanders@nat/digium/x-hfvulemflvuethwt) |
| 15:32.38 | Kobaz | ack, ugh, 1.8 locked up as well |
| 15:32.44 | Kobaz | even with running astdb on a ramdisk |
| 15:33.16 | seanbright | 1.8? WHAT YEAR IS THIS?! |
| 15:35.39 | Kobaz | it's the most solid asterisk i have |
| 15:35.46 | Kobaz | 11 has been dieing on me |
| 15:35.57 | Kobaz | and randomly drops sip registrations |
| 15:36.12 | Kobaz | although there's a pending patch for that now |
| 15:36.23 | Kobaz | i haven't ported my apps to 13 yet |
| 15:36.26 | Kobaz | i think that's next |
| 15:36.46 | Kobaz | i probably need to switch to pjsip to take advantage of all the multi threading |
| 15:37.25 | Kobaz | the single chan_sip signaling thread locks up and i'm screwed |
| 15:37.50 | coreyfarrell | Kobaz: do you actually run in production with DEBUG_THREADS enabled? |
| 15:38.11 | Kobaz | yes |
| 15:38.16 | Kobaz | i have to |
| 15:38.58 | Kobaz | when i get a lockup i need to know what it is so i can fix it |
| 15:41.20 | seanbright | how many calls are you handling per day per box? |
| 15:41.41 | seanbright | if it's less than 10 you are doing something wrong |
| 15:41.48 | Kobaz | about 1000 |
| 15:41.53 | seanbright | per day? |
| 15:41.55 | Kobaz | yeah |
| 15:42.00 | seanbright | that's... nothing |
| 15:42.03 | Kobaz | i know |
| 15:42.04 | Kobaz | haha |
| 15:42.12 | seanbright | are you doing something exotic? |
| 15:42.20 | Kobaz | i actually had a lockup wednesday |
| 15:42.24 | seanbright | or is this a typical office pbx? |
| 15:42.26 | Kobaz | it processed about 10 calls that day |
| 15:42.37 | Kobaz | nothing terribly exotic other than lots of local channels |
| 15:42.46 | Kobaz | and 200 phones |
| 15:42.47 | seanbright | and to clarify - these are 1000 concurrent calls? or 1000 individual calls over a 24 hour period? |
| 15:42.57 | Kobaz | individual calls over a 24 hour period |
| 15:43.00 | seanbright | wow |
| 15:43.01 | Kobaz | max concurrent is like 15-25 |
| 15:43.05 | Kobaz | it's like, nothing |
| 15:43.15 | seanbright | what are you using the local channels for? |
| 15:43.24 | Kobaz | agi driven call queue |
| 15:43.48 | Kobaz | our own custom stuffs we've been developing the last 6 years |
| 15:43.52 | seanbright | process based AGIs or fastagi? |
| 15:43.55 | Kobaz | and this just started happening this month |
| 15:43.59 | Kobaz | fastagi |
| 15:44.02 | seanbright | k |
| 15:44.12 | Kobaz | like. randomly started locking up with ast_db_put |
| 15:44.21 | seanbright | go back in source control and see what has changed since nov 1 |
| 15:44.31 | Kobaz | 0, nodda |
| 15:44.39 | seanbright | then your server is dying |
| 15:44.42 | seanbright | raid? |
| 15:44.47 | Kobaz | i have the same asterisk build since 2013 |
| 15:45.01 | Kobaz | that was the last build i did for that box |
| 15:45.02 | Kobaz | raid-1 |
| 15:45.09 | Kobaz | astdb is now on a tmpfs ramdisk |
| 15:45.16 | Kobaz | and still astdb is having problems |
| 15:45.20 | seanbright | weird |
| 15:45.32 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-nkxkvclvoyrfikqt) |
| 15:45.37 | seanbright | well, we're running around 1M calls per day per box |
| 15:45.41 | seanbright | on 11 |
| 15:45.45 | seanbright | all sip |
| 15:45.45 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-mvfhxqemovtyhjry) |
| 15:45.51 | Kobaz | http://pastebin.com/g1eGcwb3 |
| 15:45.52 | seanbright | agi based queueing (plus a lot more) |
| 15:45.55 | seanbright | and we don't crash at all |
| 15:46.04 | Kobaz | are you running into registration drops? |
| 15:46.12 | Kobaz | all of a suddon phones go unreachable for no reason |
| 15:46.13 | seanbright | not that we have seen |
| 15:46.20 | Kobaz | i'm gonna try the patch for that today |
| 15:46.41 | seanbright | chan_sip is such dogshit |
| 15:46.45 | Kobaz | :/ |
| 15:46.58 | seanbright | i mean... before chan_pjsip it was great |
| 15:47.00 | seanbright | now it's dogshit |
| 15:47.02 | Kobaz | hehe |
| 15:47.07 | seanbright | (we still run chan_sip) |
| 15:47.23 | Kobaz | yeapo |
| 15:47.36 | Kobaz | what do you think of that show locks? |
| 15:47.56 | Kobaz | that lock is being held for 2+ minutes |
| 15:48.12 | Kobaz | that same exact core show locks, byte for byte, ran though a differ |
| 15:48.29 | Kobaz | our watcher makes localhost sip calls to test asterisk health |
| 15:48.48 | Kobaz | three calls in a row fail and it restarts asterisk after it dumps a bunch of debug data (like core show locks) |
| 15:50.18 | seanbright | you're running latest 1.8 or some older version? |
| 15:50.51 | Kobaz | latest |
| 15:50.53 | Kobaz | i was on 1.8.12 |
| 15:51.00 | Kobaz | switched to the latest and had the same problem |
| 15:52.36 | seanbright | then something else is wrong |
| 15:52.50 | seanbright | because your lines numbers in that trace do not match chan_sip.c from 1.8 latest |
| 15:53.26 | Kobaz | that's 11 |
| 15:53.45 | seanbright | what is 11? the thread dump? |
| 15:54.26 | seanbright | it's not latest 11, line numbers still not right |
| 15:54.32 | seanbright | unless you're running patched 1.8 or 11 |
| 15:54.39 | seanbright | in which case, you have voided your warranty |
| 15:56.43 | Kobaz | patched |
| 15:56.52 | Kobaz | i have some fixes for ast_db_put locking up |
| 15:57.07 | Kobaz | it solves a lot of them |
| 15:59.03 | Kobaz | i need to put in this reg drop patch |
| 15:59.28 | Kobaz | https://issues.asterisk.org/jira/browse/ASTERISK-25476 |
| 16:02.27 | Kobaz | if the box was dieing, other things would be breaking |
| 16:02.30 | Kobaz | only asterisk is breaking |
| 16:06.46 | *** join/#asterisk-dev skirmisha (~skirmisha@213.16.62.70) |
| 16:06.54 | skirmisha | hi guys |
| 16:07.11 | skirmisha | can someone tell me where the call legs cdr defined? |
| 16:07.16 | skirmisha | in which source file? |
| 16:07.31 | Kobaz | call legs? |
| 16:07.34 | skirmisha | i have an issue with asterisk ver 13 where voip answer alive isnt send |
| 16:07.37 | Kobaz | can you be more specific |
| 16:07.46 | Kobaz | voip answer alive? |
| 16:08.39 | skirmisha | we send accounting over radius, but we see difference between very old 1.8 ver and latest 13 |
| 16:09.23 | skirmisha | we miss one of the call leg reported in the radius accounting, usually we have start answer, start originate and so on |
| 16:10.03 | skirmisha | then we have interim updates , but we never got interim for answer call leg, only for originate |
| 16:10.47 | skirmisha | i want to find out where the code is located for those cdrs being generated |
| 16:12.04 | Kobaz | i would need some more information, like what channel driver you're working with, what event you think you're missing |
| 16:12.52 | file | an entire specification for CDRs is at https://wiki.asterisk.org/wiki/display/AST/Asterisk+12+CDR+Specification which details how they work in 13 |
| 16:13.28 | skirmisha | it is sip channel |
| 16:15.10 | Kobaz | have you verified what you're seeing on the raw asterisk cdr? |
| 16:15.15 | Kobaz | and not what's on radius? |
| 16:16.11 | skirmisha | i am debuging radius packets as my colleague is maintaining the radius appl |
| 16:16.32 | skirmisha | i am missing VoIP answer interim updates |
| 16:16.40 | skirmisha | is this the function for interim updates |
| 16:16.43 | skirmisha | int ast_cdr_init(struct ast_cdr *cdr, struct ast_channel *c) |
| 16:16.50 | Kobaz | but have you checked the cdr output itself? |
| 16:16.56 | Kobaz | it may be an issue with your transfer to radius |
| 16:17.34 | skirmisha | i will talk to my colleague, but what he said is that this call leg never received |
| 16:17.50 | skirmisha | because we use same appl on ver 1.8 and there it is working |
| 16:19.32 | skirmisha | i will come back later |
| 16:19.59 | skirmisha | is there a way to debug on asterisk to see cdr's updates? |
| 16:20.13 | skirmisha | then i will be sure it is asterisk issue or in our appl? |
| 16:20.46 | Kobaz | you can use core set debug to see a *lot* of channel detail, but i think the best bet is set up cdr/csv logging and look at that |
| 16:20.49 | Kobaz | compare to what you have in radius |
| 17:39.44 | *** join/#asterisk-dev pchero (~pchero@109.70.54.56) |
| 17:42.48 | Kobaz | something is completely wiggity wack |
| 17:43.19 | Kobaz | # asterisk -cvvv -T -g |
| 17:43.19 | Kobaz | Illegal instruction (core dumped) |
| 17:43.36 | Kobaz | ==14426== valgrind: Unrecognised instruction at address 0x586647. |
| 17:43.36 | Kobaz | ==14426== at 0x586647: timesub (localtime.c:2007) |
| 17:46.18 | opticron | Kobaz, did you compile on a VM with an old version of gcc? |
| 17:47.18 | Kobaz | yes |
| 17:47.28 | Kobaz | it wasn't breaking before though |
| 17:48.12 | Kobaz | well it's not that old, it's gcc 4.7 |
| 17:51.09 | Kobaz | that was weird |
| 17:51.13 | Kobaz | i re-ran ./configure |
| 17:51.15 | Kobaz | and now it's fine |
| 17:51.30 | Kobaz | (and re-built clean of course) |
| 18:59.44 | opticron | yeah, 4.7 isn't the right kind of old to cause the issue I was thinking of |
| 20:37.09 | Kobaz | ugh |
| 20:37.12 | Kobaz | more locks |
| 20:37.39 | Kobaz | http://pastebin.com/6FBCEGfg |
| 21:29.29 | *** join/#asterisk-dev DivideBy0 (~DivideBy0@unaffiliated/divideby0x0) |
| 21:30.14 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-179-40-172.cust.bezeqint.net) |
| 21:30.14 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 21:54.37 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 21:57.36 | *** join/#asterisk-dev chadxz (Adium@nat/digium/x-sypnqchqbmkytnsq) |
| 22:16.53 | Kobaz | opticron: built it locally on the target machine and no more illegal instruction |
| 22:19.32 | Kobaz | https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56068 |
| 22:36.22 | *** part/#asterisk-dev superscrat (asanders@nat/digium/x-hfvulemflvuethwt) |
| 23:22.02 | *** join/#asterisk-dev snuff-work (~blatus@203.87.61.22) |
| 23:22.02 | *** mode/#asterisk-dev [+o snuff-work] by ChanServ |