| 02:28.43 | *** join/#asterisk-dev babak (uid19622@gateway/web/irccloud.com/x-tcmtisjtuvkmxder) |
| 03:17.35 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 03:17.35 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 04:08.58 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 04:08.58 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 04:18.52 | *** join/#asterisk-dev luckman212 (~luckman21@unaffiliated/luckman212) |
| 04:33.59 | *** join/#asterisk-dev klow (~textual@c-67-161-106-89.hsd1.wa.comcast.net) |
| 04:59.33 | *** join/#asterisk-dev bof22 (~Thunderbi@185.13.183.107) |
| 05:24.46 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@24-177-86-98.dhcp.nwtn.ct.charter.com) |
| 07:11.15 | *** join/#asterisk-dev pchero_work (~pchero@109.70.54.56) |
| 07:39.32 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 07:39.32 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 07:52.51 | *** join/#asterisk-dev klow (~textual@c-98-247-49-57.hsd1.wa.comcast.net) |
| 09:27.38 | *** join/#asterisk-dev bof22 (~Thunderbi@185.13.183.107) |
| 12:46.26 | *** join/#asterisk-dev newtonr (RustyNewto@nat/digium/x-veulcliwouewgceb) |
| 12:46.26 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 12:55.31 | *** join/#asterisk-dev putnopvut (~putnopvut@asterisk/master-of-queues/mmichelson) |
| 12:55.31 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 13:52.04 | *** join/#asterisk-dev luckman212 (~luckman21@unaffiliated/luckman212) |
| 13:52.47 | *** join/#asterisk-dev babak (uid19622@gateway/web/irccloud.com/x-wdpzhhpespszhobw) |
| 14:12.30 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-yzwgijccevzybepp) |
| 14:21.36 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 14:21.36 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 14:31.25 | *** join/#asterisk-dev roramirez (~rodrigo@190.209.106.228) |
| 14:51.32 | cresl1n | waves |
| 14:58.56 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-kqettstqqholgxel) |
| 15:10.35 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 15:10.35 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 15:12.03 | file | Moo |
| 15:17.49 | coreyfarrell | woof |
| 15:34.46 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 15:34.46 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 15:34.53 | cresl1n | sends file a bucket of sand |
| 15:38.24 | file | I already have lots |
| 15:38.32 | file | How are you? |
| 15:44.24 | cresl1n | Surrounded by tall buildings |
| 15:58.50 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 15:58.50 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 16:15.53 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 16:15.53 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 16:35.09 | *** join/#asterisk-dev klow (~textual@c-98-247-49-57.hsd1.wa.comcast.net) |
| 16:50.58 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 16:50.58 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 17:22.53 | *** join/#asterisk-dev jameswf (uid27319@gateway/web/irccloud.com/x-jqrshyzvgzoazpxo) |
| 17:53.29 | *** join/#asterisk-dev piling (~piling@unaffiliated/piling) |
| 18:28.32 | gtjoseph | rmudgett: where do you think a deadlock is still possible? |
| 18:30.14 | *** join/#asterisk-dev pchero (~pchero@109.70.54.56) |
| 18:30.32 | rmudgett | in refer_incoming_invite_request() when you call ast_move_channel() |
| 18:30.52 | rmudgett | s/move_channel/channel_move/ |
| 18:31.36 | rmudgett | The function is called by a serializer despite what the comment above the ast_channel_move says. |
| 18:33.01 | rmudgett | What happens if you have two of these deadlocking scenarios at the same time but with two different serializers involved? They could mutually deadlock each other. |
| 18:34.09 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 18:34.09 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 18:35.08 | gtjoseph | is it also called by the distributor serialize peocess? |
| 18:35.16 | rmudgett | I initially thought that two masquerades could happen at the same time until I found that ast_channel_move() will only allow one masquerade at a time. |
| 18:35.33 | rmudgett | Yes this would be a pjsip serializer and likely a pjsip/serializer |
| 18:36.17 | gtjoseph | but is it the distributor itself? I.E. using the hash logic? |
| 18:36.25 | rmudgett | s/pjsip\/serializer/pjsip\/distributor/ |
| 18:36.34 | rmudgett | yes |
| 18:37.52 | gtjoseph | ah, the channel_move_lock is a global and it's held while do_masquerade is called. |
| 18:38.42 | rmudgett | Since only one masquerade could happen at a time, I though the scenario would be avoided. However, with two serializers involved both mutually trying to do a masquerade they would block each other. |
| 18:39.34 | rmudgett | One would be doing the masquerade and trying to suspend the serializers. The other would be blocked trying to start a masquerade. |
| 18:40.19 | gtjoseph | you're thinking 2 completely unrelated masquerades who's hashes place them on separate serializers |
| 18:42.18 | rmudgett | I'm thinking of two masquerades. Serializer1 masquerading two channels that are on serializer 1 and 2 while Serializer 2 is trying to masquerade two other channels that are on serializer 2 and 1. |
| 18:42.59 | rmudgett | Since there is no ordering you have to do some sort of deadlock avoidance. |
| 18:43.07 | gtjoseph | ew |
| 18:44.11 | gtjoseph | this is an issue then regardless of alexei's patch |
| 18:44.37 | rmudgett | I was researching who was calling ast_channel_move(). Only refer_incoming_invite_request() gets called by a serializer. The others get called by independent threads AFAIKT. |
| 18:46.06 | rmudgett | The only good thing is that these scenarios would be quite rare because you would have to have two people doing something that requires masquerades at the same time. |
| 18:46.29 | rmudgett | And those masquerades would have to affect the same serializers. |
| 18:47.41 | gtjoseph | yeah i suppose 2 unrelated masquerade attempts using channels that both happen to be on the same serializers ad another masquerade attempt |
| 18:51.34 | rmudgett | call_pickup_incoming_request() is another vector to get these masquerades since that is also called by pjsip/distributor serializer. |
| 18:55.28 | rmudgett | A key to eliminating these deadlocks is to get ast_channel_move() executed by a thread/serializer not associated with calls. Likely we could use the pjsip/default serializers for this. |
| 18:57.53 | rmudgett | Then we cause call_pickup_incoming_request() and refer_incoming_invite_request() to push the masquerade job to the pjsip/default serializers. |
| 18:58.01 | file | fun |
| 18:58.57 | gtjoseph | BUT... is any of this related to the specific problem in ASTERISK-26145 and Alexei's 3424 patch? |
| 18:59.00 | rmudgett | file: Yes avoiding deadlocks is always "fun" |
| 18:59.56 | rmudgett | gtjoseph: I think Alexei's patch is a start. |
| 19:00.48 | rmudgett | It does address deadlocking when the masquerade happens to two channels that are in the same serializer. |
| 19:00.55 | gtjoseph | kinda |
| 19:01.27 | gtjoseph | the first masquerade blocks the serializer, then the second just returns but then the first unblocks the serializer... |
| 19:01.40 | gtjoseph | what happens to the seocnd nopw that the serializer is unblocked |
| 19:02.34 | rmudgett | I was just wondering what would happen if the scenario happened twice at the same time and came up with those permutations. |
| 19:02.52 | gtjoseph | i think a block counter is needed |
| 19:03.13 | gtjoseph | so it's the LAST masquerader that kills the block task |
| 19:03.22 | rmudgett | The second does not attempt to block the serializer so when it tries to unsuspend it has nothing to do. |
| 19:03.50 | gtjoseph | not only that but the taskprocessor is unblocked while the seocnd masquerador is running |
| 19:04.11 | rmudgett | The masquerade code suspends the serializers on both channels and when it completes it unsuspends the serializers on both channels. |
| 19:05.27 | rmudgett | When it unsuspends the channels, the masquerade is complete so a second could start. However since only one masquerade can happen at any one time we do not need that count you were thinking of. |
| 19:06.07 | gtjoseph | you sure? |
| 19:06.21 | gtjoseph | hmmmm. |
| 19:06.23 | rmudgett | ast_channel_move() has that global lock remember? |
| 19:06.46 | rmudgett | That lock is in effect for the entire masquerade. |
| 19:07.06 | gtjoseph | yeah but doesn't that still mean that tasks could still creep in on the session serializer? |
| 19:07.12 | gtjoseph | between the 2 |
| 19:07.44 | gtjoseph | thinks it through |
| 19:08.57 | rmudgett | That is why I think those two functions I mentioned above need to push the masquerade job off onto the pjsip/default serializer or a new pjsip/masquerade serializer to allow the serializers involved with calls to process tasks. |
| 19:09.26 | rmudgett | lunch time |
| 19:18.42 | *** join/#asterisk-dev Marquis42 (~bwatkins@198.91.76.200) |
| 19:21.04 | gtjoseph | rmudgett: when you're finished with lunch, ping me back. :) |
| 19:28.54 | rmudgett | gtjoseph: ping |
| 19:29.22 | gtjoseph | that was fast. :) |
| 19:29.34 | gtjoseph | thinking about alexei's patch... |
| 19:30.30 | gtjoseph | if only do_masquerade sets MASQUERADE_NOTIFY and do_masquerade is onlt called by channel_move with the global lock held... |
| 19:31.12 | gtjoseph | and only chan_pjsip_indicate calls ast_sip_session_suspend... |
| 19:31.51 | rmudgett | You can unsuspend in the opposite order? |
| 19:32.30 | gtjoseph | well, is the call to ast_sip_session_suspend on the same thread as the original channel_move? or is there a taskprocessor somewhere in between? |
| 19:33.30 | rmudgett | oh. Are you trying to be really nasty?? :P |
| 19:34.00 | gtjoseph | no i'm trying to figure out why alexei's patch works |
| 19:35.07 | rmudgett | no. The MASQUERADE_NOTIFY is not pushed onto a pjsip serializer. It is a direct call. |
| 19:35.12 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 19:35.12 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 19:35.50 | gtjoseph | then that would mean that ast_sip_session_suspend could never be called in parrallel by anyone |
| 19:36.00 | rmudgett | yes |
| 19:36.09 | rmudgett | It is the only caller. |
| 19:36.24 | gtjoseph | so what's the point of ast_sip_session_suspend checking to see if the serializer is suspended? |
| 19:36.40 | gtjoseph | it can't be |
| 19:37.09 | rmudgett | If both channels are processed by the same serializer you would call the suspend twice on the same serializer. |
| 19:37.24 | rmudgett | Once for the original channel and the other for the clonechan. |
| 19:39.28 | gtjoseph | ok, so in this case, the lock is acquired, then suspend is once for each channel, correct? |
| 19:39.39 | rmudgett | yes |
| 19:40.15 | gtjoseph | if both channels happen to be handled by the same serializer, then we get both suspend taska on the same serializer and we're dead. |
| 19:40.25 | rmudgett | Before Alexei's patch we would be waiting forever on the second channel to suspend the taskprocessor which is already suspended. |
| 19:40.25 | gtjoseph | yeah? |
| 19:40.41 | rmudgett | yes |
| 19:40.43 | gtjoseph | too many object in place. :) |
| 19:40.47 | gtjoseph | play |
| 19:40.58 | rmudgett | place works too :) |
| 19:41.03 | gtjoseph | now i understand why alexei's patch works |
| 19:41.08 | gtjoseph | for this case anyway |
| 19:41.43 | rmudgett | The other scenarios that I was talking about would still happen with or without Alexei's patch. |
| 19:42.18 | gtjoseph | yep, understood |
| 19:43.53 | rmudgett | And since the suspend/unsuspend cannot happen outside of the global masquerade lock, some race conditions I was thinking about couldn't happen either. |
| 20:05.28 | *** join/#asterisk-dev klow (~textual@73.109.63.138) |
| 20:11.13 | *** join/#asterisk-dev Marquis42 (~bwatkins@198.91.76.202) |
| 20:21.21 | rmudgett | I suppose I should write up an issue about the other deadlock potentials. |
| 20:39.21 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 20:39.21 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 20:45.10 | *** join/#asterisk-dev klow (~textual@96.81.150.137) |
| 20:48.35 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 20:48.35 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:11.27 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 21:11.27 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:22.11 | rmudgett | file: putnopvut How is an INVITE w/ refer used? |
| 21:22.37 | rmudgett | Is it an alternate to using the REFER message? |
| 21:23.50 | putnopvut | Do you mean "replaces" instead of "refer" ? |
| 21:23.58 | rmudgett | yes |
| 21:24.22 | putnopvut | An INVITE with replaces isn't an alternative to a REFER. It's part of the process when performing an attended transfer when one of the dialogs involved is not on the local box. |
| 21:24.30 | putnopvut | A "remote attended transfer" if you will. |
| 21:24.49 | rmudgett | thanks |
| 21:24.50 | putnopvut | Say Alice calls Bob through Asterisk 1. She then puts Bob on hold and places a call to Carol through Asterisk 2. |
| 21:24.54 | putnopvut | She then completes the transfer. |
| 21:25.23 | putnopvut | She'll send a REFER to Asterisk 1. The Refer-To header will reference the dialog Alice has with Carol on Asterisk 2. |
| 21:25.59 | putnopvut | So what Asterisk 1 does is to send an INVITE with Replaces header saying that Asterisk 2 needs to replace the dialog that it had with Alice with this new one. |
| 21:26.23 | putnopvut | That was probably not the most thorough explanation, but hopefully you get the gist. |
| 21:27.16 | rmudgett | Yes. The deadlock I'm seeing in the code would happen if we received that INVITE w/ replaces. |
| 21:29.27 | rmudgett | ala: refer_incoming_invite_request() and the ast_channel_move() interacting with another refer_incoming_invite_request() at the same time |
| 21:30.36 | rmudgett | I'm trying to document the deadlock scenario |
| 21:31.32 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 21:31.32 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:37.46 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 21:37.46 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 21:54.44 | file | rmudgett, sorry - I was figuring out why there's water in my basement again |
| 21:54.54 | file | answer: water is coming from my HVAC system |
| 21:56.11 | file | there was half an inch of water sitting in the part below my air handler |
| 21:56.17 | rmudgett | Good thing you don't need AC very often. You'd be floating down the Mississippi |
| 21:58.31 | file | THE YELLOW PAGES! |
| 22:06.35 | *** join/#asterisk-dev roramirez (~rodrigo@190.209.106.228) |
| 22:06.57 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 22:06.57 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 22:12.11 | cresl1n | water needs to drain from the air handler |
| 22:12.13 | cresl1n | file |
| 22:12.25 | file | cresl1n, it has a drain! and I even cleaned it |
| 22:12.29 | cresl1n | ah, ok |
| 22:12.38 | cresl1n | you know this already then :-) |
| 22:12.56 | file | unless the pipe is clogged some, causing not all of the water to get pumped out and ultimately causing it to overflow |
| 22:13.04 | file | and then it collects at the bottom |
| 22:13.06 | cresl1n | I had that problem on mine last year |
| 22:13.07 | file | which is possible |
| 22:16.19 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@wsip-98-179-138-226.ri.ri.cox.net) |
| 22:56.13 | *** join/#asterisk-dev piling_ (~piling@unaffiliated/piling) |
| 23:02.54 | *** join/#asterisk-dev piling (~piling@unaffiliated/piling) |
| 23:04.35 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-yzwgijccevzybepp) |
| 23:16.16 | *** join/#asterisk-dev cresl1n (~Adium@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 23:16.16 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 23:24.01 | *** join/#asterisk-dev snuff-work (~snuff-wor@203.87.61.22) |
| 23:24.01 | *** mode/#asterisk-dev [+o snuff-work] by ChanServ |
| 23:37.24 | *** join/#asterisk-dev luckman212 (~luckman21@unaffiliated/luckman212) |