| 00:08.35 | *** join/#asterisk-dev KaliLinuxGR (~KaliLinux@unaffiliated/kalilinuxgr) |
| 00:31.32 | snuff-work | emel_punk|2: so you have the 2 includes? <ffmpeg/avcodec.h> and <ffmpeg/swscale.h> |
| 00:31.42 | snuff-work | in your /usr/include |
| 00:33.00 | snuff-work | also.. maybe.. the defines might require updating.. |
| 00:33.11 | snuff-work | aka AV_PIX_FMT |
| 00:33.18 | snuff-work | instead of just PX_FMT |
| 00:33.19 | emel_punk|2 | yes |
| 00:33.23 | tompaw | It looks like ast_queue_hangup_with_cause doesn't really do anything, __ast_queue_frame simply returns 0 for AST_CONTROL_HANGUP. |
| 00:34.06 | tompaw | So, are you saying that you don't actually hangup channels, but simply decrease their reference counter until it hits 0? |
| 00:35.02 | snuff-work | tompaw: my guess is after ref hits 0 it will fall to being hung up.. but pure speculation on my part because i haven't read ANY of the code your talking about =) |
| 00:35.47 | file | the act of queueing creates a frame on a channel which is read/handled by another thread, that thread will then hangup the channel |
| 00:35.49 | snuff-work | emel_punk|2: as a 'test'.. try this against the 2 files.. sed -i 's/PIX_FMT/AV_PIX_FMT/g' channels/console_video.c |
| 00:36.09 | emel_punk|2 | snuff-work: OK |
| 00:36.20 | tompaw | snuff-work: it's all there > https://issues.asterisk.org/jira/browse/ASTERISK-26716 - I am about to try getting a REF_DEBUG dump as per https://wiki.asterisk.org/wiki/display/AST/Reference+Count+Debugging - maybe b( ) adds a rogue reference which under certain circumstances prevents the channels from being hung up |
| 00:36.43 | tompaw | (I am not pushing people to debug my shit on sunday, trying to do it myself) |
| 00:36.47 | file | as Local channels have pairs, one side will queue a hangup on the other so it appears as though it says "yo dawg I'm hung up" |
| 00:36.57 | file | it is unlikely that b() itself adds a rogue reference |
| 00:37.04 | snuff-work | file has come to rescue me from answer stuff about queues =P |
| 00:37.05 | file | it is more likely that something is causing the queued hangup to get lost/ignored |
| 00:37.20 | file | ie: it could be getting consumed somewhere in b() |
| 00:37.21 | tompaw | file: but it's not even being queued. |
| 00:37.41 | tompaw | file: inside __ast_queue_frame, it simply returns 0 for AST_CONTROL_HANGUP frames |
| 00:38.00 | tompaw | And it's the same for both affected (zombie) and unaffected hangups. |
| 00:38.16 | file | that means it has been queued. |
| 00:38.41 | emel_punk|2 | snuff-work: Nope, still not working... although the error is on vgrabbers.c and I did the change on the file |
| 00:38.43 | tompaw | does it? |
| 00:38.46 | file | yes. |
| 00:38.58 | tompaw | file: the code says: case AST_CONTROL_HANGUP: /* Don't queue anything. */ return 0; |
| 00:39.17 | file | that code is only executed if a hangup is already queued |
| 00:39.24 | file | as queueing it again would be redundant |
| 00:39.53 | snuff-work | mm new error output ? pastebin? |
| 00:40.14 | tompaw | file: that's not what the code suggests, unless I'm reading it wrong: if (cur && cur->frametype == AST_FRAME_CONTROL && !head && (!after || after == cur)) { |
| 00:40.21 | tompaw | which condition states that hangup is already queued? |
| 00:40.23 | file | tompaw: you're reading it wrong :) |
| 00:40.34 | file | tompaw: that code loops through the frames queued on the channel |
| 00:40.47 | file | er actually, it gets the LAST frame in the queue |
| 00:40.54 | file | <PROTECTED> |
| 00:41.05 | tompaw | yes, it doesn't seem to check if "hangup is already queued" |
| 00:41.20 | file | ast_channel_readq is the queued frames to be read |
| 00:41.26 | file | AST_LIST_LAST gets the last frame in the queue |
| 00:41.38 | tompaw | very nice, but where does it check if "hangup is already queued" |
| 00:41.46 | file | the code you previously posted |
| 00:41.48 | file | that is the check |
| 00:42.01 | file | get the last frame in the queue, if the last frame is a hangup frame then don't do anything |
| 00:42.04 | file | otherwise queue it up |
| 00:42.06 | tompaw | ahhhh, it's the case of the last frame! DOH! |
| 00:42.19 | tompaw | *NOT* the new one |
| 00:42.22 | file | correct |
| 00:42.23 | tompaw | sorry, you're right. |
| 00:42.31 | tompaw | that's even weirder then |
| 00:43.05 | tompaw | so in both cases the ;2 are queued to be hung up. what could be preventing one of them from actually BEING hung up? |
| 00:43.07 | tompaw | reference count? |
| 00:43.35 | file | no |
| 00:43.43 | file | something reading the hangup frame and not acting on it |
| 00:43.55 | tompaw | I wonder if it's a coincidence that in 100% cases where the hangup is ignored I'm getting this RTP error. |
| 00:44.08 | tompaw | I know that RTP should have nada to do with hangups, but it's still weird... |
| 00:45.10 | tompaw | file: if you haven't seen my issue url, but you're watching this channel, I can show you the error message here :> |
| 00:45.24 | file | I get an email for everything that happens on JIRA |
| 00:45.33 | file | has seen |
| 00:46.02 | tompaw | As you can see I've already spent tens of hours trying to fix it, I just don't know where to look next. |
| 00:47.40 | file | you could look at the code that reads the frame, add a condition based on hangup and see if something is reading the hangup as expected |
| 00:48.00 | file | if it is - then it is what is handling that |
| 00:48.33 | tompaw | "you could look at the code that reads the frame" << hence my question from an hour ago about what you call it so I can locate it |
| 00:49.54 | file | ast_Read? |
| 00:50.06 | file | er ast_read, dog nudged me |
| 00:52.33 | tompaw | ast_check_hangup(chan) sounds promising |
| 00:55.02 | tompaw | this function makes me wanna cry |
| 01:10.31 | tompaw | https://gist.github.com/tmpkn/0bbf51a00deceafa22477ea6c31ea05c < for the lack of better ideas |
| 01:10.54 | tompaw | (i already know both hit i_am_zombie, just added the other debugs to see if there are any differences) |
| 01:15.14 | *** join/#asterisk-dev emel_punk|2 (~emel_punk@190.67.252.41) |
| 01:15.24 | tompaw | of course they both hit ast_queue_control(chan, AST_CONTROL_END_OF_Q) |
| 01:15.56 | emel_punk|2 | Still doesn't work the workarround |
| 01:16.03 | emel_punk|2 | any other tip? |
| 01:17.21 | tompaw | gotta walk my gf, will continue the struggle in an hour. |
| 01:22.29 | emel_punk|2 | I am amaze that there is no info about console_video even when was introduced almost five years ago. |
| 01:26.35 | snuff-work | emel_punk|2: output from failed compile? |
| 01:26.56 | emel_punk|2 | the same one. |
| 01:27.18 | emel_punk|2 | vgrabbers.c:122:64: error: âAV_PIX_FMT_RGB555â undeclared (first use in this function) |
| 01:27.18 | emel_punk|2 | <PROTECTED> |
| 02:04.00 | *** join/#asterisk-dev Gugge (gugge@92.246.2.105) |
| 03:43.14 | snuff-work | mm.. i may try myself to build it soon see if I can work around it |
| 07:00.53 | *** join/#asterisk-dev bof22 (~Thunderbi@185.13.183.107) |
| 07:11.08 | *** join/#asterisk-dev oej (~oej@h87-96-134-129.cust.se.alltele.net) |
| 07:11.08 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 07:21.55 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d00b199e2db88abcd79.ipv6.abo.wanadoo.fr) |
| 07:29.04 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 07:29.05 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 07:33.07 | *** join/#asterisk-dev jkroon (~jkroon@uls-154-73-35-201.wall.uls.co.za) |
| 07:51.41 | *** join/#asterisk-dev pchero_work (~pchero@2a00:c80:1072::c18) |
| 08:27.28 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 08:41.25 | *** join/#asterisk-dev Tim_Toady (~fuzzy@snf-33276.vm.okeanos.grnet.gr) |
| 08:54.48 | *** join/#asterisk-dev CELYA_ (~Thunderbi@LStLambert-657-1-90-73.w80-13.abo.wanadoo.fr) |
| 08:58.09 | *** join/#asterisk-dev jkroon (~jkroon@uls-154-73-35-201.wall.uls.co.za) |
| 11:12.57 | *** join/#asterisk-dev jkroon (~jkroon@uls-154-73-35-201.wall.uls.co.za) |
| 12:37.19 | *** join/#asterisk-dev newtonr (~newtonr@173-17-133-211.client.mchsi.com) |
| 12:37.19 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 12:41.34 | *** join/#asterisk-dev fblackburn (~fblackbur@qubcpq0957w-lp130-02-70-53-160-248.dsl.bell.ca) |
| 13:08.14 | *** join/#asterisk-dev libardi (~libardi@187.64.229.96) |
| 14:07.06 | *** join/#asterisk-dev leedm777 (~textual@2604:2d80:8404:801b:d86f:1e13:a7ff:a44c) |
| 14:07.06 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 14:30.08 | *** join/#asterisk-dev pchero_work (~pchero@2a00:c80:1072::62d) |
| 14:30.43 | *** join/#asterisk-dev DivideBy0x0 (~DivideBy0@unaffiliated/divideby0x0) |
| 14:30.43 | *** mode/#asterisk-dev [+o DivideBy0x0] by ChanServ |
| 14:33.16 | *** join/#asterisk-dev pchero_work (~pchero@2a00:c80:1072::62d) |
| 15:14.07 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-xtwdlcgxvojnfejv) |
| 15:14.07 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 15:28.08 | *** join/#asterisk-dev putnopvut (putnopvut@asterisk/master-of-queues/mmichelson) |
| 15:28.08 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 15:34.36 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-favfctnhdcoqujrt) |
| 15:34.36 | *** mode/#asterisk-dev [+o rmudgett] by ChanServ |
| 15:41.18 | *** join/#asterisk-dev cresl1n (Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 15:41.18 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 15:59.45 | *** join/#asterisk-dev oej (~oej@194.15.212.5) |
| 15:59.45 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 16:12.53 | tompaw | Morning |
| 16:21.56 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-wofqnmjgqszfnidh) |
| 16:21.56 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 18:39.49 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-tjgigsyvaasnwaga) |
| 18:39.50 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 19:00.56 | *** join/#asterisk-dev leedm777 (~textual@73.106.78.192) |
| 19:00.56 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 19:13.06 | *** join/#asterisk-dev leedm777 (~textual@73.106.78.192) |
| 19:13.07 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 19:23.39 | tompaw | Is there a guide and/or manual to __ast_read? |
| 19:24.09 | tompaw | or only what exists in the comments? |
| 19:25.33 | file | what you see is what exists |
| 19:32.50 | tompaw | https://wiki.asterisk.org/wiki/display/AST/Frames < that doesn't help much either :< |
| 19:50.41 | tompaw | file: any tips as to which parts of this function I should be testing in particular? |
| 19:50.52 | file | um |
| 19:50.57 | file | not really |
| 19:52.16 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d00b4e2e7b0a535523f.ipv6.abo.wanadoo.fr) |
| 19:59.10 | *** join/#asterisk-dev leedm777 (~textual@63.92.231.41) |
| 19:59.10 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 20:05.39 | tompaw | ok, so the faulty ;2 doesn't hit this: https://github.com/asterisk/asterisk/blob/13.13/main/channel.c#L3939 |
| 20:05.40 | tompaw | while the proper ;2 does |
| 20:08.47 | tompaw | the OK one hits both AST_CONTROL_HANGUP and AST_CONTROL_END_OF_Q cases (I'm guessing 2 separate frames?) |
| 20:10.10 | tompaw | clicked "BLAME" on channel.c @github and it crashed my safari despite 32 gigs of ram |
| 20:12.30 | file | nudges rmudgett |
| 20:14.05 | rmudgett | falls over |
| 20:15.59 | file | rmudgett: ASTERISK-26716 |
| 20:18.09 | rmudgett | AST_CONTROL_HANGUP is going to hit both those switch cases because the AST_CONTROL_HANGUP case falls through to the other. |
| 20:18.35 | tompaw | ahhh right |
| 20:18.46 | tompaw | but still, the ;2 affected by this bug hits neither |
| 20:23.16 | tompaw | rmudgett: any idea how I can speed up the debug? as you can see I'm shooting in the dark by just adding ast_debug lines everywhere "hangup" is mentioned. |
| 20:23.31 | tompaw | and then comparing the log of OK vs FCKUP calls |
| 20:42.39 | tompaw | is gonna try REF_DEBUG as nothing else makes sense |
| 20:43.04 | *** join/#asterisk-dev pchero (~pchero@109.70.54.56) |
| 20:56.38 | tompaw | fuk this, I'm gonna try two Local channels with Dial(,,I) instead. |
| 20:58.09 | rmudgett | The logs you posted on the issue are too short. I would need to see the whole call because I want to see which threads are doing what and how far the call has gotten. |
| 20:58.28 | rmudgett | Also are you hanging up the originated channel immediately after originating it? |
| 21:00.45 | *** join/#asterisk-dev KaliLinuxGR (~KaliLinux@unaffiliated/kalilinuxgr) |
| 21:01.39 | tompaw | rmudgett: a few seconds after originating it |
| 21:02.21 | tompaw | rmudgett: but my ARI properly picks it up and manually hangs the other Local channel. the bug is that the Local channel will sometimes leave a rogue ;2 leg (PJSIP) running |
| 21:02.48 | tompaw | rmudgett: I have the logs here, let me attach one to the issue. |
| 21:03.25 | rmudgett | Please attach as a .txt file and reference it in the comment. |
| 21:03.42 | rmudgett | Much shorter scroll to the end of the issue that way. :) |
| 21:03.56 | tompaw | on it, let me just sanitize it because h4xx0rz |
| 21:04.27 | tompaw | rmudgett: it's gonna have some extra logs, but they are all marked with *TMP* |
| 21:08.54 | tompaw | rmudgett: done. I'm not sure why it took my file comment and added it as a separate issue comment |
| 21:09.06 | tompaw | rmudgett: https://issues.asterisk.org/jira/secure/attachment/54859/ari_h_error |
| 21:09.58 | rmudgett | Thats the way attach works. When you attach a file and supply a comment, the comment is appended to the issue as a regular comment. |
| 21:10.31 | tompaw | rmudgett: so in this log, Local/13051002000-18001002000@cont-fas-00000003;2 is the culprit |
| 21:12.00 | rmudgett | Without a .txt exten the file has to be downloaded to view it. Stupid JIRA. |
| 21:12.13 | tompaw | as you can see it receives hangup requests at 14:55:51 (right after ;1 goes down), but chooses to ignore it (doesn't hit that bit in __ast_read), and instead is happily making progress at 14:55:52. |
| 21:12.21 | tompaw | DAMN, let me reupload |
| 21:12.55 | tompaw | https://issues.asterisk.org/jira/secure/attachment/54860/ari_h_error.txt |
| 21:21.42 | tompaw | fck me sideways, 2 Local channel with Dial(,,I) has been the answer the whole time |
| 21:21.47 | tompaw | no b( ) involved |
| 21:22.31 | tompaw | file: your tip about I was correct, but you applied it to wrong Dial |
| 21:23.47 | file | oic |
| 21:25.09 | tompaw | file: I need to "I" the 1st channel (Agent), to stop its cli from leaking all over my bridge. And since Originate cannot do "I", I have to replace it with another Local channel that does Dial(,,I) explicitly. |
| 21:25.19 | tompaw | wonders if dialplan can decode base64 |
| 21:25.46 | tompaw | yes it can, awesome |
| 21:57.27 | *** join/#asterisk-dev leedm777 (textual@nat/digium/x-rbwvskmcerebahdr) |
| 21:57.27 | *** mode/#asterisk-dev [+o leedm777] by ChanServ |
| 22:17.00 | *** join/#asterisk-dev snuff-work (~snuff-wor@203.87.61.22) |
| 22:17.00 | *** mode/#asterisk-dev [+o snuff-work] by ChanServ |
| 22:20.27 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-82-81-175-197.red.bezeqint.net) |
| 22:20.27 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 22:46.38 | tompaw | god damn, it doesn't work either in 2xLocal config. cli rate up to 30% but still not 100 |
| 22:46.43 | tompaw | anyone here doing bounties? |
| 22:56.09 | rmudgett | tompaw: Could you attach a patch file showing where you put all your *TMP* debug msgs? |
| 22:56.27 | tompaw | yep, sure |
| 22:56.31 | rmudgett | thanks |
| 22:57.24 | tompaw | rmudgett: would it make you a difference if I attached the amended files instead? |
| 22:58.25 | rmudgett | That can work. |
| 23:01.27 | tompaw | rmudgett: https://issues.asterisk.org/jira/secure/attachment/54861/tmp_core.tar |
| 23:01.53 | tompaw | rmudgett: thanks for picking it up btw, I've been messing with this for days now with no luck. |