| 00:20.01 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 00:20.01 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 07:09.58 | *** join/#asterisk-dev jkroon (~jkroon@165.16.204.172) |
| 07:34.11 | *** join/#asterisk-dev tsearle (~tsearle@37.19.10.33) |
| 07:34.12 | *** mode/#asterisk-dev [+o tsearle] by ChanServ |
| 07:48.04 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01:e0a:d:1111:790b:91d2:fe1e:d283) |
| 11:10.46 | *** join/#asterisk-dev jkroon (~jkroon@165.16.204.172) |
| 12:06.54 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 13:25.34 | *** join/#asterisk-dev leedm777 (~textual@2604:2d80:8404:ffae:6df2:b431:f4ae:f0d8) |
| 13:25.34 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 14:10.17 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-sfhbttetrbermzdw) |
| 14:10.17 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 14:47.25 | *** join/#asterisk-dev bford (uid283514@gateway/web/irccloud.com/x-lmhvtloebsfvjdoj) |
| 14:47.25 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 14:51.39 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-gzbvmjiztuiuvhas) |
| 14:51.39 | *** mode/#asterisk-dev [+o rmudgett] by ChanServ |
| 15:33.19 | gtjoseph | coreyfarrell: there might be legit failures on https://gerrit.asterisk.org/#/c/8390/ |
| 15:34.14 | stefan27 | We have about 200 asterisk-13-instances all have similar configuration and setup and they use the old chan_sip. We have found that typically the CPU usage of the asterisk-instance stands in close correlation with the channel count "core show channels". One instance however stands out by using 3 times as much CPU as the second highest CPU-consuming-instance, despite having the exact same |
| 15:34.14 | stefan27 | channel count per day as the second one. Naturally this could have a million different reasons, and I'm not gonna expect anyone to guide a troubleshooting process in a module you think is obsolete. |
| 15:34.44 | gtjoseph | i sense a "but..." coming :) |
| 15:35.06 | stefan27 | yes, but I'm gonna share an observation of mine and see if anyone wants to comment: |
| 15:36.15 | *** join/#asterisk-dev Kobaz (~kobaz@its.kobaz.net) |
| 15:38.19 | stefan27 | At a high level, the usage of high-consuming-cpu-instance seems very similar to the second one, (most objects you can measure are within -/+ 30% of the other machine's count... But the high consuming machine does have many unique sip devices sending subscribe requests, causing the asterisk -rx "sip show objects" | grep 'type: dialog' to output a 4 times larger number than the second |
| 15:38.20 | stefan27 | machine |
| 15:39.20 | stefan27 | Is that known to cause high chan_sip workloads? if you have say 3000 sip object dialogs |
| 15:39.29 | stefan27 | high CPU workload* |
| 15:40.08 | stefan27 | Alternatively, is there any good tool I can use to measure where an asterisk process spends its cpu time? |
| 15:40.14 | gtjoseph | if the expirations are short enough then yes they'll be a lot of time spent servicing those subscribes. |
| 15:40.57 | stefan27 | If it was a development environment I could run the thing in callgrind I guess, but the measure tool mustnt impact the behiavour of the system itself a lot |
| 15:42.00 | gtjoseph | you could try running top or htop with user threads turned on and see if you can correlate the activity |
| 15:42.42 | gtjoseph | i haven't tried that in a very long time so i'm not sure how much help it will be |
| 15:44.25 | stefan27 | Right... I may try htop or strace for extra info... asterisk -rx "core show threads" is the most info asterisk can give me (when no compile flags was present) |
| 15:44.34 | stefan27 | +? |
| 15:46.17 | gtjoseph | core show taskprocessors might also help. |
| 15:46.44 | gtjoseph | but maybe not with chan_sip |
| 15:48.57 | stefan27 | Ah, thanks a lot |
| 15:55.25 | *** join/#asterisk-dev DanQuinney_ (sid18169@gateway/web/irccloud.com/x-zfbsgvtybplkxszt) |
| 16:03.54 | *** join/#asterisk-dev skrusty (~skrusty@88.150.145.104) |
| 16:03.54 | *** mode/#asterisk-dev [+o skrusty] by ChanServ |
| 16:11.12 | *** join/#asterisk-dev Worldexe (~Worldexe@95-107-33-134.dsl.orel.ru) |
| 16:11.37 | *** join/#asterisk-dev file (sid178970@asterisk/developer-and-muffin-lover/file) |
| 16:11.37 | *** mode/#asterisk-dev [+o file] by ChanServ |
| 16:12.29 | *** join/#asterisk-dev fblackburn (~fblackbur@modemcable094.94-70-69.static.videotron.ca) |
| 16:40.15 | seanbright | we should consider using the pthread np stuff on linux so that we can name threads |
| 17:05.06 | rmudgett | We'll come up with some unique names like: Thread 1, Thread 2, Thread 3... |
| 17:20.28 | jpsharp | name them all George. |
| 17:43.00 | jsmith | jpsharp: That was my thought... "I'll hug him, and pet him, and name him George" |
| 17:53.22 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 17:53.22 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 17:55.53 | gtjoseph | you can name them all after me, that's fine. |
| 17:56.47 | jpsharp | Except for that one thread. We'll name it Bob. |
| 18:04.17 | stefan27 | gtjoseph whatever that commad you gave me measures, it correlates perfectly with actual cpu usage, in the following sense: the machine that had 3 times higher cpu usage on average also had 3 times higher value of asterisk -rx "core show taskprocessors" | awk '{ test=test+$2} END { print test}' |
| 18:04.49 | stefan27 | the sum of that column |
| 18:05.14 | stefan27 | the column called "Processed". might be chance of course |
| 18:05.35 | gtjoseph | well that shows how mant tasks were processed |
| 18:05.52 | gtjoseph | how about the "Max Depth" column? |
| 18:06.40 | stefan27 | I will investigate that more on monday, I'm not in office anymore. |
| 18:10.10 | stefan27 | thanks for your tips again |
| 18:10.42 | gtjoseph | no problem |
| 18:13.57 | jsmith | seanbright: Well, we could name all the threads "Matt".... |
| 18:14.21 | gtjoseph | that'd be good as we have multiple |
| 18:18.10 | jsmith | Exactly! |
| 18:25.05 | seanbright | jesus |
| 18:25.14 | seanbright | that's (not really) the last time i suggest something |
| 18:27.29 | jsmith | seanbright: :-) |
| 18:33.11 | coreyfarrell | I want one of the threads to be named nemo |
| 18:37.18 | *** join/#asterisk-dev ntwrk (~nic@69.167.51.204) |
| 18:37.35 | coreyfarrell | gtjoseph: so what's going on with the testing? half a dozen of my patches that aren't even covered by any test failed. |
| 18:37.59 | gtjoseph | there seems to be a network issue or something. i'm looking at it |
| 18:38.22 | gtjoseph | one of the tests i think was legit... the ari get modules test |
| 18:38.44 | coreyfarrell | yeah I'll look at that in a few |
| 18:43.41 | *** join/#asterisk-dev scgm11_ (~scgm11@r186-52-130-36.dialup.adsl.anteldata.net.uy) |
| 18:46.26 | coreyfarrell | gtjoseph: I think the get modules test must be failing because extra modules (the built-in ones) are loaded? the yaml is using match on response_body with an element per module, would this fail if extra array elements are found? |
| 18:46.54 | gtjoseph | i believe so |
| 18:48.29 | coreyfarrell | I'm not sure the best way to deal with this - if the loader should skip built-in modules from ast_update_module_list_data / other similar functions or if we need an ast_module_is_builtin() for ARI to call and skip, or if the test needs to be branched for master? |
| 18:49.19 | gtjoseph | I'd say the last option is the best |
| 18:49.42 | gtjoseph | since we now have testsuite branching |
| 18:49.44 | gtjoseph | :) |
| 18:50.34 | coreyfarrell | oh so I should just add the built-in modules to the testsuite master branch and that won't mess with 13/15? |
| 18:51.21 | gtjoseph | sure! |
| 18:52.06 | *** join/#asterisk-dev scgm11_ (~scgm11@r186-52-130-36.dialup.adsl.anteldata.net.uy) |
| 18:52.11 | coreyfarrell | cool. so I'm guessing you'll just manually merge the testsuite (even though it'll be temporarily broken) then regate the asterisk patch? |
| 19:19.55 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d00d569e0c6555ee4c5.ipv6.abo.wanadoo.fr) |
| 19:29.11 | coreyfarrell | well that was a real pain, when we do an ARI deep copy the logs give the absolute minimum help needed to compare expected vs received |
| 20:11.11 | seanbright | sounds like a patch opportunity |
| 20:49.51 | coreyfarrell | sounds like we have a volunteer :P |
| 22:15.35 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 22:15.35 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 23:21.38 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-sfhbttetrbermzdw) |