| 00:11.28 | *** join/#asterisk-dev tuxd00d (~tuxd00d@ip68-106-11-138.ph.ph.cox.net) |
| 00:23.58 | *** join/#asterisk-dev zopsi_ (~zopsi@dir.ac) |
| 00:32.10 | *** join/#asterisk-dev jkroon (~jkroon@vc-nat-gp-s-41-13-8-98.umts.vodacom.co.za) |
| 00:53.01 | *** join/#asterisk-dev jkroon (~jkroon@41.13.8.98) |
| 03:34.46 | *** join/#asterisk-dev dakudos (~dakudos@c-73-153-12-13.hsd1.co.comcast.net) |
| 03:51.28 | *** join/#asterisk-dev jkroon (~jkroon@vc-nat-gp-s-41-13-28-208.umts.vodacom.co.za) |
| 04:22.25 | *** join/#asterisk-dev leedm777 (~textual@173-16-52-28.client.mchsi.com) |
| 04:22.25 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 06:39.45 | *** join/#asterisk-dev tsearle (~tsearle@37.19.10.33) |
| 06:39.46 | *** mode/#asterisk-dev [+o tsearle] by ChanServ |
| 06:49.50 | *** join/#asterisk-dev CELYA_ (~Thunderbi@LStLambert-657-1-90-73.w80-13.abo.wanadoo.fr) |
| 07:11.55 | *** join/#asterisk-dev tuxian (~tuxian@igilmour.plus.com) |
| 07:22.45 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 07:22.46 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 08:17.08 | *** join/#asterisk-dev thierr1 (~thierr1@124.8.154.77.rev.sfr.net) |
| 09:25.48 | *** join/#asterisk-dev tuxian (~tuxian@194.12.3.67) |
| 10:36.43 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 10:52.05 | *** join/#asterisk-dev wdoekes (~walter@wjd.osso.nl) |
| 10:52.05 | *** mode/#asterisk-dev [+o wdoekes] by ChanServ |
| 11:37.00 | *** join/#asterisk-dev wdoekes (~walter@wjd.osso.nl) |
| 11:37.00 | *** mode/#asterisk-dev [+o wdoekes] by ChanServ |
| 11:43.03 | *** join/#asterisk-dev jkroon (~jkroon@uls-154-73-35-201.wall.uls.co.za) |
| 12:33.04 | *** join/#asterisk-dev leedm777 (~textual@2604:2d80:8404:8011:6419:7d01:f305:fa97) |
| 12:33.04 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 12:34.57 | seanbright | openid server broked? |
| 12:35.45 | gtjoseph_ | wasn't a bit ago |
| 12:36.05 | seanbright | javax.servlet.ServletException: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.GenericJDBCException: Could not open connection |
| 12:36.37 | file | le sigh |
| 12:36.44 | gtjoseph_ | hmmm. just tried it again. no issues. is this from a gerrit login? |
| 12:36.49 | seanbright | yes |
| 12:36.54 | seanbright | maybe i have a bad cookie or something |
| 12:37.09 | file | fails for me too |
| 12:37.12 | gtjoseph_ | nope. i'm geting that now as well |
| 12:37.14 | seanbright | YESH |
| 12:37.19 | seanbright | oh thank goodness |
| 12:40.09 | gtjoseph_ | i've reported it |
| 12:40.38 | gtjoseph_ | again |
| 12:41.29 | file | and again... and again... |
| 13:34.09 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 13:34.09 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 13:50.44 | tsearle | hey cresl1n ! |
| 13:50.50 | tsearle | has the storm reached you yet? |
| 13:51.15 | cresl1n | hey tsearle if it has, it's died down quite a bit |
| 13:51.41 | opticron | it passed through last night |
| 13:51.48 | opticron | just lots of rain at this point |
| 13:51.53 | cresl1n | opticron!!!! |
| 13:51.57 | opticron | me! |
| 13:51.59 | cresl1n | :-) |
| 13:52.13 | file | who? |
| 13:52.31 | tsearle | I remember meeting opticron one astricon.... Atlanta if I remember correctly |
| 13:52.40 | opticron | that'd be the one |
| 14:15.37 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-etnxorjqbyuieufi) |
| 14:15.37 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 14:16.10 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 14:22.07 | *** join/#asterisk-dev bford (d8cff501@gateway/web/freenode/ip.216.207.245.1) |
| 14:22.07 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 14:32.16 | *** join/#asterisk-dev n_byrnes (~nate@173.225.53.16) |
| 14:41.32 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 14:48.59 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-kxcfxqviramzpjys) |
| 14:49.00 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 14:57.40 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-llygtpvjeqzdanlv) |
| 14:57.40 | *** mode/#asterisk-dev [+o rmudgett] by ChanServ |
| 16:03.12 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-pkqvsqwwtlszvduo) |
| 16:03.12 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 16:36.26 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d00e581cb93e839b530.ipv6.abo.wanadoo.fr) |
| 17:12.08 | *** join/#asterisk-dev jkroon (~jkroon@165.16.204.165) |
| 17:42.48 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d00e581cb93e839b530.ipv6.abo.wanadoo.fr) |
| 17:49.12 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-oeymndblpykqsfxa) |
| 17:49.12 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 17:51.15 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-tqkowpyujhbahkco) |
| 17:51.15 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 17:56.46 | *** join/#asterisk-dev joshelson (~joshelson@198.232.219.254) |
| 18:23.46 | joshelson | Hoping someone might have some quick thoughts on how to improve my troubleshooting here... but on fully patched Asterisk 14 with about 200 realtime endpoints, seeing multiple PJSIP partial lockups a day. All invites and registrations go unresponded to until Asterisk is restarted. But OPTIONS pass successfully and existing calls continue just fine. Non-obvious circumstances to repro. Any thoughts on how to troubleshoot this? |
| 18:24.32 | gtjoseph_ | 1. Are ytou seeing messages in the log about taskprocessors |
| 18:24.55 | gtjoseph_ | 2. How's your DNS infrastructure? |
| 18:25.41 | joshelson | I'd find it hard to believe we're having a DNS problem, but not impossible. |
| 18:25.54 | joshelson | I see this taskprocessor related error from around the time of the issue: taskprocessor.c: The 'subm:rtp_topic-00000051' task processor queue reached 500 scheduled tasks. |
| 18:26.25 | gtjoseph_ | whenever you see one of those messages, asterisk will stop taking new requests |
| 18:26.51 | gtjoseph_ | you should see another message when it goes below the low water mark |
| 18:26.56 | joshelson | Limited to the RTP topic in that case, or more broadly? |
| 18:27.02 | gtjoseph_ | also do a "core show taskprocessors" |
| 18:27.08 | gtjoseph_ | bropadly |
| 18:27.12 | gtjoseph_ | broadly |
| 18:27.31 | gtjoseph_ | any taskprocessor going over the high water mark will cause asterisk to stop taking new requests |
| 18:28.09 | joshelson | Ah.. ok... good to know. |
| 18:28.40 | gtjoseph_ | @rmudgett: can you comment on the RTP topic? |
| 18:37.42 | rmudgett | The rtp_topic taskprocessor is handling sent and received RTCP messages on the stasis message bus. |
| 18:37.53 | joshelson | Also can somewhat correlate this to this taskprocessor.c: The 'subm:cdr_engine-00000003' task processor queue reached 5000 scheduled tasks. message |
| 18:38.23 | joshelson | Both of those seem roughly correlated to most of the issues I've seen |
| 18:38.23 | gtjoseph_ | @rmudgett: loves cdrs :) |
| 18:38.42 | gtjoseph_ | if you don't need cdrs, turn them off |
| 18:38.50 | joshelson | Haha. Oh, I wish. |
| 18:39.05 | rmudgett | See http://blogs.asterisk.org/2016/07/13/asterisk-task-processor-queue-size-warnings/ |
| 18:39.09 | gtjoseph_ | if you do, you might want to look at the backend |
| 18:40.07 | joshelson | How do you mean? |
| 18:40.54 | gtjoseph_ | well whenever ANY taskprocessor goes over the limit, asterisk stops taking new requests. If cdr processing is backing up, that's cause it. do you write them to a database? |
| 18:41.04 | joshelson | We do. |
| 18:41.08 | gtjoseph_ | local or remote? |
| 18:41.18 | joshelson | Combination of both... but sometimes remote |
| 18:41.22 | gtjoseph_ | :) |
| 18:41.31 | joshelson | Haha. Got it :) |
| 18:42.05 | joshelson | For something like the rtp topic... how would we go about troubleshooting that? |
| 18:42.29 | gtjoseph_ | 5000 cdrs pending is a lot of cdrs. something's backing up whether it's network I/O, filesystem I/O or cpu cycles. |
| 18:45.32 | joshelson | Yeah.. hmm. We can run that down. Is there some way to peek inside the queue? |
| 18:45.55 | gtjoseph_ | noipe |
| 18:46.21 | rmudgett | noipe is a noisy nope :) |
| 18:46.31 | gtjoseph_ | :) |
| 18:46.32 | joshelson | But in general, a subm:cdr_engine queue of 5000 means that 5000 CDRs have not been written to the DB engien? |
| 18:46.41 | joshelson | engien is a very quiet engine |
| 18:46.57 | gtjoseph_ | essentially yes |
| 18:48.17 | rmudgett | The subm:cdr_engine taskprocessor is a queue of events the CDR module uses to generate CDR's. The queue is not CDR's themselves. |
| 18:50.48 | joshelson | Hmm... so just had it happen again with this message: taskprocessor.c: The 'pjsip/distributor-00000026' task processor queue reached 500 scheduled tasks. |
| 18:51.52 | gtjoseph_ | so that one means asterisk couldn't actually process requests fast enough, never mind cdrs |
| 18:52.31 | rmudgett | Well processing CDR's is CPU intensive. |
| 18:52.49 | joshelson | Hmm.... this is relatively beefy underloaded box. Not seeing anything that would historically indicate a load issue. |
| 18:52.53 | rmudgett | Are the HEP modules loaded? |
| 18:54.16 | joshelson | They are |
| 18:54.18 | joshelson | Killing now |
| 18:54.24 | joshelson | We do like/want to use however |
| 18:54.29 | joshelson | But we have other ways of getting this |
| 19:00.38 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-lgdrowghoemhbbza) |
| 19:00.38 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 19:05.04 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-179-40-170.cust.bezeqint.net) |
| 19:05.04 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 19:28.35 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 19:28.35 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 19:35.20 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 19:35.20 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 19:45.07 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 19:45.07 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 20:11.17 | *** join/#asterisk-dev joshelson (~joshelson@198.232.219.254) |
| 20:13.05 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-quwpiuxorynmnmnj) |
| 20:13.05 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 20:18.10 | *** join/#asterisk-dev tuxian (~tuxian@igilmour.plus.com) |
| 21:06.31 | *** join/#asterisk-dev tuxian (~tuxian@igilmour.plus.com) |
| 21:12.40 | *** join/#asterisk-dev joshelson (~joshelson@198.232.219.254) |
| 21:13.59 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 21:13.59 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:20.21 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 21:20.21 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:26.03 | *** join/#asterisk-dev philipp64 (~chatzilla@66-232-81-197.ctcweb.net) |
| 23:24.57 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-etnxorjqbyuieufi) |