| 00:32.40 | *** join/#asterisk-dev elcontrastador (~textual@70-90-215-98-BusName-ca.sacra.hfc.comcastbusiness.net) |
| 01:19.04 | *** join/#asterisk-dev infobot (ibot@208.53.50.136) |
| 01:19.04 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 06:00.38 | *** join/#asterisk-dev GameGamer43 (sid5533@gateway/web/irccloud.com/x-ebaimuyzitwrcdzo) |
| 06:00.38 | *** mode/#asterisk-dev [+o GameGamer43] by ChanServ |
| 06:29.39 | *** join/#asterisk-dev tsearle (~tsearle@188.44.81.78) |
| 06:29.40 | *** mode/#asterisk-dev [+o tsearle] by ChanServ |
| 07:18.49 | *** join/#asterisk-dev CELYA_ (~Thunderbi@lstlambert-657-1-90-73.w80-13.abo.wanadoo.fr) |
| 07:23.44 | *** join/#asterisk-dev jkroon (~jkroon@165.16.203.61) |
| 07:26.33 | *** join/#asterisk-dev pchero_work (~pchero@dhcp-077-249-058-090.chello.nl) |
| 07:30.15 | *** join/#asterisk-dev pchero_work (~pchero@dhcp-077-249-058-090.chello.nl) |
| 09:47.43 | *** join/#asterisk-dev AsteriskRoss (259d3426@gateway/web/freenode/ip.37.157.52.38) |
| 11:30.16 | coreyfarrell | AsteriskRoss: fix posted to https://gerrit.asterisk.org/10646 (cherry-picks to be posted soon). |
| 11:41.20 | coreyfarrell | looks like https://jenkins2.asterisk.org/job/asterisk-unittests/job/16/408/ froze during startup. |
| 11:42.38 | coreyfarrell | it did startup (we have a few entries in full log), but didn't reach fully booted after 15 seconds. |
| 11:47.33 | file | erm |
| 12:34.52 | *** join/#asterisk-dev pc-m (~pcm@modemcable094.94-70-69.static.videotron.ca) |
| 13:46.43 | *** join/#asterisk-dev bford (uid283514@gateway/web/irccloud.com/x-axmmdilxefsicgmz) |
| 13:46.43 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 14:03.20 | coreyfarrell | csavinovich: that's odd, adding --no-tcp fixed chanspy_w_mixmonitor for me locally (tested with master). |
| 14:03.48 | coreyfarrell | if you prefer I can abandon those reviews? |
| 14:26.58 | csavinovich | hi corey, I am getting a 488 error. |
| 14:26.58 | csavinovich | https://pastebin.com/g8qYYyxM |
| 14:27.11 | csavinovich | Corey |
| 14:28.33 | csavinovich | yeah, I would prefer to finish it since I was working on it and I think I pretty much have the fix out shortly |
| 14:31.41 | csavinovich | Thanks |
| 14:37.45 | *** join/#asterisk-dev dandann00dle (~dandann00@2601:840:8401:d7ee:cd87:f9f3:b237:db82) |
| 15:06.37 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-bgwlyehknrofoqtz) |
| 15:06.37 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 15:08.56 | csavinovich | Hey Corey, I am thinking, maybe your test is successful cause you are running the external version of pjsua compiled with codecs. The bundled version of pjsua is compile with all codecs disabled. This causes pjsua to send a blank codec in the initial SDP which asterisk doesn't like and responds with a 488. |
| 15:14.56 | coreyfarrell | csavinovich: nope, my testing was done with the bundled pjsua in a container that doesn't have a system pjproject build. |
| 15:18.30 | kharwell | I noticed the same thing when initially testing/updating the barge test. When using bundled pjsua had no codecs available. I wonder why that seems to be the case for some vs others. Maybe something like the system? I was using Ubuntu 16.04 at the time. |
| 15:18.48 | coreyfarrell | csavinovich: do you have TEST_FRAMEWORK enabled? not enabling it causes resample and g711-codec to be disabled. |
| 15:19.53 | coreyfarrell | see third-party/pjproject/Makefile.rules. maybe that should be toggled based on --enable-dev-mode instead? |
| 15:21.04 | file | oh that could explain it... |
| 15:21.18 | gtjoseph | yeah i forgot about that |
| 15:21.46 | file | I'd be down with making it on enable-dev-mode |
| 15:21.48 | coreyfarrell | I checked the config.log for bundled pjproject, the configure command-line did not include '--disable-resample --disable-g711-codec' |
| 15:25.00 | kharwell | looks like I had it enabled and am getting 0 codecs. |
| 15:27.21 | file | did you enable it after pjproject was configured? |
| 15:27.41 | coreyfarrell | kharwell: do you have anything in third-party/pjproject/source/third_party/lib/ ? |
| 15:28.03 | kharwell | file: oh yeah I did. I ran menuselect after configure, so that could still be it |
| 15:28.43 | file | the pjproject configure should still get re-run, but I wonder if that's wonky? |
| 15:28.43 | coreyfarrell | impossible to run menuselect before running asterisk configure.. pjproject-bundled is supposed to rebuild after you enable TEST_FRAMEWORK. |
| 15:29.20 | file | coreyfarrell: I've got a libresample, that's it |
| 15:29.50 | file | build build build |
| 15:29.51 | kharwell | coreyfarrell: after building it appears third-party/pjproject/source/third_party/lib/ is empty for me |
| 15:31.31 | kharwell | after running make install I do have pjsua in /var/lib/asterisk/third-party/pjproject/pjsua |
| 15:31.46 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-vjirxrlkjpavedao) |
| 15:31.47 | *** mode/#asterisk-dev [+o rmudgett] by ChanServ |
| 15:32.04 | coreyfarrell | but you should have a libresample .a in the pjproject third_party/lib |
| 15:33.41 | file | when make was run, did you see pjproject configure get re-run? |
| 15:35.50 | coreyfarrell | kharwell: also what other 'Compiler Flags' options are you touching in menuselect? I'm using tests/CI/buildAsterisk.sh so enabling 'DONT_OPTIMIZE BETTER_BACKTRACES MALLOC_DEBUG DO_CRASH TEST_FRAMEWORK' then disabling 'COMPILE_DOUBLE BUILD_NATIVE'. |
| 15:35.50 | kharwell | This is all I see for pjproject when running make: https://pastebin.com/ATMbd8wq |
| 15:36.29 | file | it does not appear it noticed you enabled TEST_FRAMEWORK |
| 15:37.06 | kharwell | coreyfarrell: I have all those enabled as well. I also disable COMPILE_DOUBLE, but did not disable BUILD_NATIVE |
| 15:37.32 | kharwell | oh OPTIONAL_API appears to be selected as well |
| 15:37.59 | coreyfarrell | OPTIONAL_API is enabled by default |
| 15:38.19 | kharwell | think it might be BUILD_NATIVE. I just disabled it and I saw configure get called now for pjproject |
| 15:39.28 | file | might not have been directly, it could have caused the timestamp on the file or something to get changed resulting in it picking up the change and invoking configure |
| 15:42.39 | coreyfarrell | odd though any change to Compiler Flags in menuselect is supposed to remove .lastclean which should force a rebuild. |
| 15:42.45 | gtjoseph | pjproject _should_ get reconfigured/built if menuselect.makeopts is touched. |
| 15:43.05 | gtjoseph | make cmenuselect |
| 15:43.09 | gtjoseph | oops |
| 15:43.38 | file | if this was your Terminal, things would be very very different... |
| 15:44.17 | gtjoseph | :) |
| 15:44.17 | file | ooo a Linux distribution which is purely console only that you interact with in the form of a text adventure game |
| 15:44.26 | gtjoseph | Ha! |
| 15:44.49 | file | "You see a file named .bash_history in front of you" |
| 15:56.03 | coreyfarrell | maybe it's not any specific Compiler Flag option but instead something to do with multiple runs of menuselect? tests/CI/buildAsterisk.sh runs 'make menuselect.makeopts' followed by multiple calls to menuselect/menuselect. |
| 17:29.23 | coreyfarrell | not sure if anyone has looked into these yet but I think tests/channels/pjsip/connected_line/connected_line_no_send and tests/channels/pjsip/nat/rewrite_contact/route_set_reinvite are Asterisk failures. |
| 17:31.22 | coreyfarrell | for connected_line_no_send it appears Asterisk is having some kind of memory corruption, it's sending unicode 'ÃÃÃÃÃÃÃ' in place of the header name 'Remote-Party-ID'. |
| 17:31.45 | gtjoseph | hmmm |
| 17:35.35 | coreyfarrell | https://jenkins2.asterisk.org/job/asterisk-gatetests/job/master/332/artifact/tests/CI/output/pjsip1/logs/channels/pjsip/connected_line/connected_line_no_send/run_1/ - messages.txt shows the corrupted header name, this error happens to me locally as well. |
| 18:28.33 | *** join/#asterisk-dev CELYA_ (~Thunderbi@lfbn-1-5361-63.w90-105.abo.wanadoo.fr) |
| 19:10.35 | seanbright | yes. that does appear wrong. |
| 19:26.03 | seanbright | it's not unicode |
| 19:26.14 | seanbright | it's 0xdeaddead |
| 19:28.19 | seanbright | well... it 0xdead 6 times in a row |
| 19:28.22 | seanbright | but the bytes are swapped |
| 19:36.37 | coreyfarrell | seanbright: unless I'm misunderstanding it appears that pj_str_t->slen was still 15 when it did the '200 OK' packet. so the storage for pj_str_t was not freed, but the memory pj_str_t->ptr points to was freed. |
| 19:37.18 | seanbright | the only place that Remote-Party-ID (the string) is referenced is in a static allocation |
| 19:37.48 | seanbright | res_pjsip_caller_id.c:add_rpid_header() |
| 19:40.19 | coreyfarrell | then it's passed to create_new_id_hdr which uses pj_strdup to create a dynamic allocation |
| 19:42.53 | coreyfarrell | so the header name is a string allocated by tdata->pool. maybe it needs to use a session pool instead? I'm not very good with pjproject but from some debugging it seems like the value created for "180 Ringing" is being reused by the "200 OK". |
| 19:43.38 | coreyfarrell | I added some ast_log's to add_rpid_header and the messages were logged when creating the "180 Ringing" packet but not when the "200 OK" was created. |
| 19:46.11 | seanbright | well, the tdata pool should be valid at least until the message makes it on to the wire |
| 19:46.17 | seanbright | (i think) |
| 19:47.29 | file | aye |
| 19:47.31 | gtjoseph | something to think about... pjproject will reuse memory so a pj_str may have a pointer to a valid string but it's slen might be 0. |
| 19:47.52 | gtjoseph | it's the slen that counts |
| 19:49.42 | seanbright | are the data structures involved with provisional responses re-used for final responses? |
| 19:50.41 | file | the INVITE session code will reuse it, if I recall correctly |
| 19:53.57 | seanbright | also, can i lobby for #define AST_PJ_STR(s) { s, sizeof(s) - 1 } |
| 19:55.23 | coreyfarrell | gtjoseph: in this case it appears slen is still 15 (the correct value). it's pointing to 0xdead memory. weird thing to me is that the header value still points to valid memory. |
| 19:56.21 | seanbright | the 0xdead stuff is astmm right? |
| 19:56.38 | file | yes |
| 19:58.08 | seanbright | that's where i get lost |
| 19:58.21 | seanbright | the header is duped from a pj pool |
| 19:58.29 | seanbright | which are not freed with asterisk's region stuff |
| 19:58.51 | gtjoseph | if you have MALLOC_DEBUG on then yes it is |
| 19:59.10 | gtjoseph | rmudgett: extended the astmm stuff down to pjproject a while back |
| 20:01.00 | seanbright | ahhh, ok |
| 20:01.05 | seanbright | his fault then |
| 20:01.07 | seanbright | closing as wontfix |
| 20:01.18 | rmudgett | Yes. The pjproject pools are covered by MALLOC_DEBUG |
| 20:02.42 | coreyfarrell | how would headers from the "180 Ringing" response get saved for re-use by the "200 OK" response? is that asterisk or pjproject doing that? |
| 20:03.27 | file | it's PJSIP |
| 20:03.48 | file | specifically pjsip/src/pjsip-ua/sip_inv.c |
| 20:05.11 | *** join/#asterisk-dev jkroon (~jkroon@165.16.204.34) |
| 20:18.56 | seanbright | i like specificity |
| 20:19.28 | coreyfarrell | looks like pjproject expects the name field of pjsip_fromto_hdr to be static pj_str_t. |
| 20:20.34 | seanbright | what makes you say that? |
| 20:21.16 | seanbright | god their code formatting is awful |
| 20:21.25 | coreyfarrell | it looks like pjsip_inv_answer() calls pjsip_tx_data_clone() which copies headers. I traced to pjsip_fromto_hdr_clone() in pjsip/src/pjsip/sip_msg.c |
| 20:21.29 | seanbright | mixed tabs & spaces? sure! |
| 20:21.43 | coreyfarrell | yeah I hate pjsip formatting |
| 20:22.15 | seanbright | ah, it doesn't dup the name/sname |
| 20:22.41 | coreyfarrell | yes. so that's why name->slen == 15 but points to dead memory. |
| 20:22.55 | coreyfarrell | leaves me wondering how this ever worked |
| 20:23.42 | seanbright | spoiler alert: probably didn't |
| 20:23.44 | seanbright | heh |
| 20:23.54 | coreyfarrell | in the testsuite anyways.. obviously in production people don't have MALLOC_DEBUG so the use-after-free wouldn't always fail |
| 20:24.52 | seanbright | well. it doesn't have to be static, it just has to outlive the dialog. |
| 20:25.05 | seanbright | or live as long as, i should say |
| 20:29.21 | coreyfarrell | is it even safe to unload res_pjsip_caller_id while the system is live? maybe it would be better to just stop duplicating the header name and set the module to block manual unload. |
| 20:29.37 | file | while sessions are active definitely not |
| 20:41.01 | seanbright | coreyfarrell: session->inv_session->pool |
| 20:41.02 | seanbright | ? |
| 20:41.40 | seanbright | just kinda grasping at straws honestly :P |
| 20:50.05 | coreyfarrell | seanbright: nah. I just posted a review to stop duplicating the header name / block unload. |
| 20:50.43 | coreyfarrell | verified against master that it fixes the test |
| 20:51.33 | seanbright | ð |
| 20:56.04 | *** join/#asterisk-dev tuxd00d (~tuxd00d@unaffiliated/tuxd00d) |
| 20:59.14 | seanbright | good find |
| 21:27.43 | csavinovich | Okay, thanks to the conversation this morning I was clear about the fix for the testsuite failing on the chanspy test. George/Josh/cfarrell hope to hear your thoughts on this. We need to build PJSUA with 711 and resample enabled always (even when TEST_FRAMEWORK is enabled)... BUILD_NATIVE currently needs to be disabled for PJSUA to build (document it?). Question is: is it okay to eliminate the following "if" |
| 21:27.43 | csavinovich | statement from third-party/pjproject/Makefile.rules? |
| 21:27.43 | csavinovich | ifeq ($(findstring TEST_FRAMEWORK,$(MENUSELECT_CFLAGS)),) |
| 21:27.43 | csavinovich | PJPROJECT_CONFIG_OPTS += --disable-resample --disable-g711-codec |
| 21:27.43 | csavinovich | endif |
| 21:27.44 | csavinovich | ...which will have the effect of enabling resample and 711 always. |
| 21:27.44 | csavinovich | Thanks |
| 21:41.33 | coreyfarrell | I think that if statement should check for AST_DEVMODE == no, make sure the main configure script passes AST_DEVMODE to the third-party build. that way it's enabled based on arguments given to ./configure, instead of being based on stuff done in menuselect. |
| 21:43.50 | coreyfarrell | I don't see how BUILD_NATIVE could specifically matter. menuselect should be forcing rebuild of bundled pjproject on change to anything in "Compiler Flags", the fact that it isn't reliably doing that is the problem. |
| 22:03.01 | csavinovich | I am going to test it |
| 22:06.37 | csavinovich | I don't think the BUILD_NATIVE part is within the scope of testsuite-chanspy issue, although I am all for solving all issues as soon as they are discovered. The codec part is more related to the testsuite failure. |
| 22:09.55 | coreyfarrell | even if you get around the codec error that test will still fail because pjsua is using TCP over UDP. |
| 22:10.40 | coreyfarrell | also FWIW I've never seen that codec error when using tests/CI/buildAsterisk.sh from a clean working copy. |
| 22:11.48 | csavinovich | oh yes, I am also adding the --no-tcp part in the run_test script file of the test. |
| 22:18.41 | *** join/#asterisk-dev dandann00dle (~dandann00@2601:840:8401:d7ee:cd87:f9f3:b237:db82) |
| 23:05.51 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-bgwlyehknrofoqtz) |
| 23:54.13 | *** join/#asterisk-dev dandann00dle (~dandann00@2601:840:8401:d7ee:cd87:f9f3:b237:db82) |