| 00:45.27 | *** join/#asterisk-dev pchero (~pchero@211.178.226.108) |
| 00:45.28 | *** mode/#asterisk-dev [+o pchero] by ChanServ |
| 04:35.38 | *** join/#asterisk-dev tzafrir (~tzafrir@lemon.tzafrir.org.il) |
| 04:35.38 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 05:49.25 | *** join/#asterisk-dev Dovid (~dovid@96.56.22.146) |
| 09:20.51 | *** join/#asterisk-dev mvanbaak (~mvanbaak@asterisk/contributor-and-bug-marshal/mvanbaak) |
| 09:20.51 | *** mode/#asterisk-dev [+o mvanbaak] by ChanServ |
| 12:14.34 | *** join/#asterisk-dev sduthil (~sduthil@198-48-220-170.cpe.pppoe.ca) |
| 12:28.12 | music | A couple weeks ago, I asked: âHi! Iâm the maintainer of the libss7 and libpri packages in Fedora Linux. Iâve been looking at some warnings that appear when libss7 is compiled with -Wstringop-truncation (Fedora usually adds -Werror=stringop-truncation). There are several cases where strncpy() is used to potentially truncate a string, and I suspect the result is expected to be null-terminatedâbut as the w |
| 12:28.12 | music | arnings point out, strncpy() doe s not null-terminate when the input is too long. I was hoping for an opinion from an upstream developer. Full details are at: https://bugzilla.redhat.com/show_bug.cgi?id=1932066. Thanks!â |
| 12:28.43 | music | At the time, it was a weekend, and gtjoseph said âwe can take a look on Mondayâ. Itâs far from urgent, but has anyone had a chance to take a look yet? |
| 12:29.58 | gtjoseph | Sorry, we've been swamped the past few weeks. I'm going to open an issue for it now. One minute and I'll give uou a link tot he issue. |
| 12:38.45 | gtjoseph | music: Here's the issue... https://issues.asterisk.org/jira/browse/SS7-64 |
| 13:05.47 | *** join/#asterisk-dev ZX81 (~zx81@186.188.186.181) |
| 13:08.23 | music | Thanks! Iâll keep an eye on that, and add a link in the downstream issue. |
| 13:09.46 | ZX81 | hiya, when using EAGI fd3 doesn't receive audio while playing audio - I've had a look inside res_agi.c but couldn't see why that would be the case. Maybe it doesn't receive a voice frame? |
| 13:13.01 | ZX81 | any ideas about ballpark where to look? |
| 14:00.33 | *** join/#asterisk-dev bford (uid283514@gateway/web/irccloud.com/x-qbuhbrjprhfabzxf) |
| 14:00.33 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 14:17.15 | *** join/#asterisk-dev kharwell (uid358942@gateway/web/irccloud.com/x-oiwbhzhxfmaclzfo) |
| 14:17.15 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 14:19.53 | seanbright | i feel like the RED crash is underflow on the properties array (somehow) |
| 14:20.01 | seanbright | but i don't see how that is possible |
| 14:20.15 | seanbright | i mean, it's definitely memory corruption, yeah? |
| 14:20.39 | seanbright | talking about ASTERISK-29376 |
| 14:40.43 | kharwell | yeah some kind of corruption. Rtp object could be going away or something. |
| 14:40.49 | kharwell | whatcha mean by underflow on the properties array? |
| 14:56.10 | kharwell | hrm looking in the core file itself shows instance->engine is NULL even. So the instance itself is corrupted it seems, or been freed already |
| 16:25.52 | seanbright | ->properties[-1] = 1 |
| 16:26.30 | seanbright | the member right after the instance pointer is a an int array called properties |
| 16:26.44 | seanbright | if you negative index that you could overwrite the instnace pointer |
| 16:27.00 | seanbright | but if you're seeing other stuff (i don't have the core file) then ignore all that |
| 16:35.49 | kharwell | here's the first part of the instance: https://pastebin.com/pYBe7Lwe |
| 16:36.12 | kharwell | What's really odd too is the stack starts in umount2 |
| 16:37.50 | seanbright | um... huh |
| 16:42.03 | seanbright | i would like him to reproduce on a supported version of asterisk (without modifications) |
| 16:42.29 | seanbright | if he's actually running 17.8.1 he's modified res_rtp_asterisk.c (the line numbers from the core file do not match up with the source code) |
| 16:43.39 | seanbright | or maybe he's running a redhat package that is patched by them? |
| 16:44.34 | kharwell | very good points. Yeah I'm fine with that too. If it is modified in some way we can't support it anyhow officially |
| 16:44.50 | kharwell | it's possible my backtrace is not the most accurate either. Or what I have listed |
| 16:45.21 | kharwell | I tried loading the core file he sent inside a centos8 docker container. Seems to mostly load fine but there were a few warnings |
| 16:45.26 | seanbright | no, file sent me a snippit the other day that was off by about 14 lines |
| 16:45.43 | seanbright | shenanigans |
| 16:45.51 | kharwell | aaah yeah then for sure |
| 16:47.12 | kharwell | chan_sip is also in the stack so maybe we should ask him to upgrade to pjsip too to see if it still crashes :-D |
| 16:47.45 | seanbright | oh snap, close as WONTFIX |
| 16:47.46 | seanbright | heh |
| 16:49.13 | file | chan_pjsip doesn't support it, which is why from a company perspective I don't put it under our supported list |
| 16:50.06 | seanbright | chan_pjsip doesn't support what? |
| 16:50.10 | seanbright | RED? |
| 16:50.20 | file | correct |
| 16:50.22 | seanbright | gotcha |
| 16:50.28 | file | T.140 RTT, specifically |
| 16:50.38 | seanbright | right |
| 16:50.59 | kharwell | well then upgrading to pjsip will certainly stop it from crashing :-P |
| 16:51.20 | seanbright | +1 |
| 16:51.49 | seanbright | you're a company man, now, kevin. |
| 16:55.06 | file | drink the kool aid! er I mean coffee |
| 17:08.53 | kharwell | speaking of ...time for another cup |
| 18:30.14 | *** join/#asterisk-dev pchero (~pchero@211.178.226.108) |
| 18:30.15 | *** mode/#asterisk-dev [+o pchero] by ChanServ |
| 18:30.41 | seanbright | kharwell: i like that OP can just reproduce this at will |
| 18:35.15 | kharwell | yes and surprisingly fast turn around times |
| 18:36.11 | seanbright | indeed |
| 18:50.33 | kharwell | so in the rtp output "red" is NULL, so I wonder if a NULL check on it should just be done at the offending crash location "if (rtp->red->t140.datalen > 0) {". Maybe too lock instance before getting "rtp" although not sure if that'd matter. |
| 18:51.13 | kharwell | oh well I suppose there is enough from a triage perspective unless folks can think of something else to get |
| 18:53.26 | seanbright | i suggested that to OP but it feels like a bandaid. i'm not sure how red becomes NULL there. |
| 18:54.33 | seanbright | i would say there must be a subtle timing issue involving the scheduler but the fact that he is able to reproduce so quickly makes me think there is a logical bug |
| 18:54.36 | seanbright | i just don't see it |
| 18:57.53 | kharwell | if a scheduled task is deleted from the scheduler while it's currently executing does the task finish? I'd assume so? |
| 18:59.00 | file | it does. |
| 19:00.00 | seanbright | so based on my reading of this: |
| 19:00.04 | seanbright | rtp->red->schedid = ast_sched_add(rtp->sched, generations, red_write, instance); |
| 19:00.21 | seanbright | generations as passed in from chan_sip -> rtp_engine -> res_rtp_asterisk is 2 |
| 19:00.26 | seanbright | which is 2ms in the future? |
| 19:02.09 | kharwell | so if it executes and then the rtp session is destroyed, or stops, at the same time then it seems like it would be possible for the rtp session to delete it from the scheduler, and set ->red = NULL, finish doing whatever with the rtp and instance, and then the scheduled, but now deleted task (red_write) that was waiting on "instance" continues and crashes? |
| 19:02.53 | file | if the task is currently executing then the scheduler deletion waits until it finishes, or is supposed to |
| 19:06.21 | kharwell | seanbright: I think so |
| 19:07.36 | seanbright | k |
| 19:08.51 | kharwell | file: right, so red_write tasks starts executing, rtp_stop deletes from schedule, but re-obtains "instance" lock before the red_write does. rtp_stop then sets red = NULL. Eventually "instance" gets released, and then red_write (which was waiting on lock) continues, but now "red = NULL". Seems possible to me. Unless I'm missing something? |
| 19:09.24 | file | kharwell: if red_write is invoked, then AST_SCHED_DEL blocks - or should |
| 19:10.12 | file | so until red_write returns, the thread executing rtp_stop would wait |
| 19:10.14 | kharwell | aah okay so that will block and make rtp_stop wait until red_write finishes |
| 19:10.16 | kharwell | yeah |
| 19:10.54 | seanbright | so the way the code _looks_ you should not be able to get into red_write with a NULL rtp->red, correct? |
| 19:11.05 | file | seems like it, correct |
| 19:11.19 | seanbright | well then we have a real pickle on our hands |
| 19:11.43 | seanbright | AST_SCHED_DEL() does give up after 10us |
| 19:13.21 | kharwell | it is a fried pickle? cause file really loves those |
| 19:13.47 | seanbright | who doesn't? |
| 19:13.58 | seanbright | ranch or spicy ranch? |
| 19:14.02 | file | neither. |
| 19:14.17 | seanbright | ew |
| 19:14.20 | seanbright | don't @ me |
| 19:14.33 | file | I don't ranch. At all. |
| 19:14.57 | seanbright | and you call yourself an americ... oh wait |
| 19:15.02 | kharwell | None, or spicy for me |
| 19:15.43 | seanbright | ok, so AST_SCHED_DEL fails to actually delete the entry, we NULL everything out, and the runs again |
| 19:15.46 | seanbright | this kills the asterisk |
| 19:15.48 | seanbright | that is my theory |
| 19:16.54 | seanbright | is there anyway to look at the scheduler heap in the core dump you have? |
| 19:19.04 | kharwell | hrm dunno. |
| 19:19.15 | kharwell | I'll have to look to see if there is a way |
| 19:20.05 | seanbright | now that i look at the code that might be extremely unlikely |
| 19:20.24 | seanbright | man i don't know |
| 19:20.30 | seanbright | programming is hard |
| 19:22.02 | file | bake cookies instead, that's easier |
| 19:22.41 | seanbright | don't be cookiest |
| 19:22.49 | seanbright | cookist? |
| 19:28.58 | seanbright | anddddd i'm done with that issue |
| 19:28.59 | seanbright | heh |
| 19:53.44 | kharwell | man I step away for a bit and that issue goes from solved to what the heck |
| 20:24.00 | seanbright | yup. |
| 20:43.30 | *** join/#asterisk-dev ZX81 (~zx81@186.188.186.181) |
| 22:57.39 | kharwell | seanbright: I've setup a sipp scenario and have had no luck in duplicating the "red" issue. Dunno if you are still going to try but if so hopefully you have more luck than me |