| 00:01.00 | jpeeler | Corydon76-lap: you know, i could have just used the return code instead of setting a variable |
| 00:05.05 | *** join/#asterisk-dev ZaVoid (n=zavoid@c-67-165-25-195.hsd1.pa.comcast.net) |
| 00:05.19 | jpeeler | but the main thing i wanted approval of was to hangup a channel in ast_masq_park_call, since i couldn't figure out any other way to do it |
| 00:08.13 | Corydon76-lap | jpeeler: does it work? |
| 00:08.40 | jpeeler | yes |
| 00:08.50 | Corydon76-lap | Sounds good to me, then |
| 00:09.35 | Corydon76-lap | The only other person to consult with would be file, since he's a bit more familiar with that section of code |
| 00:09.53 | Corydon76-lap | Or Russell |
| 00:10.24 | jpeeler | ok, well i'll check with them tomorrow then, thanks for looking |
| 00:11.08 | *** join/#asterisk-dev davevg-btwtech (n=davevg-b@nj-67-76-177-147.sta.embarqhsd.net) |
| 00:11.45 | Corydon76-lap | I'm going home |
| 00:14.13 | *** join/#asterisk-dev bkruse (n=bkruse@phoenixii.total-vision.net) |
| 00:14.13 | *** mode/#asterisk-dev [+o bkruse] by ChanServ |
| 00:15.20 | *** join/#asterisk-dev ZaVoid (n=zavoid@c-67-165-25-195.hsd1.pa.comcast.net) |
| 00:17.17 | seanbright | wow |
| 00:17.23 | seanbright | i've been signed on since may 9th |
| 00:17.25 | seanbright | heh |
| 00:31.42 | *** join/#asterisk-dev ZaVoid (n=zavoid@c-67-165-25-195.hsd1.pa.comcast.net) |
| 00:39.29 | *** join/#asterisk-dev bkruse (n=bkruse@phoenixii.total-vision.net) |
| 00:39.29 | *** mode/#asterisk-dev [+o bkruse] by ChanServ |
| 00:39.30 | Juggie | your a maverick seanbright! |
| 00:56.00 | *** join/#asterisk-dev obnauticus (n=obnautic@c-71-193-156-193.hsd1.wa.comcast.net) |
| 01:00.58 | *** join/#asterisk-dev Iamnacho (i=Iamnacho@ip68-103-153-140.ks.ok.cox.net) |
| 01:18.31 | lmadsen | you're!!!!!!!! |
| 01:18.47 | lmadsen | if you won't use the right word, at least switch to the slang word 'yer' |
| 01:18.56 | tzanger | ? |
| 01:19.07 | lmadsen | <Juggie> your a maverick seanbright! |
| 01:42.48 | Juggie | i will not conform! |
| 01:44.12 | *** join/#asterisk-dev ZaVoid (n=zavoid@c-67-165-25-195.hsd1.pa.comcast.net) |
| 01:46.38 | *** join/#asterisk-dev IgorG (n=IgorG@195.162.49.23) |
| 02:07.07 | *** part/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 02:12.52 | *** join/#asterisk-dev ZaVoid (n=zavoid@c-67-165-25-195.hsd1.pa.comcast.net) |
| 02:13.59 | *** join/#asterisk-dev brimstone (n=brimston@pdpc/sponsor/digium/brimstone) |
| 02:14.08 | brimstone | anyone up? |
| 02:14.17 | ctooley | yeah |
| 02:14.22 | brimstone | i found a funny little issue building zaptel 1.4 on a debian box |
| 02:14.43 | ctooley | oh, I don't do zaptel much |
| 02:16.36 | brimstone | i get the same error as this guy: http://forums.digium.com/viewtopic.php?p=68605 and fixed it doing this: http://pastebin.ca/1044472 |
| 02:16.37 | Juggie | whats zaptel? :) |
| 02:16.42 | Juggie | isnt it dahdi? |
| 02:17.16 | brimstone | it's still zaptel to me |
| 02:18.57 | lmadsen | brimstone: looks like a bug based on your patch -- can you submit that to the bug tracker pls? |
| 02:19.12 | brimstone | is there a bug already/ |
| 02:19.14 | brimstone | ? |
| 02:19.20 | brimstone | or should i create one? |
| 02:19.21 | lmadsen | I'm not too sure |
| 02:19.28 | brimstone | oooh maybe i'll get some karma! |
| 02:19.38 | lmadsen | you can search the zaptel section, and if you don't find the bug, go ahead and file a new one |
| 02:19.38 | *** join/#asterisk-dev russellb (n=russellb@asterisk/developer-and-stable-maintainer/drumkilla) |
| 02:19.38 | *** mode/#asterisk-dev [+o russellb] by ChanServ |
| 02:19.47 | lmadsen | brimstone: karma may be possible :) |
| 02:19.56 | brimstone | blames russellb |
| 02:20.00 | lmadsen | omghirussellb! |
| 02:20.01 | brimstone | oh, hi russell |
| 02:20.09 | russellb | O.O |
| 02:20.13 | russellb | what'd i do |
| 02:20.21 | lmadsen | nothing... except break things |
| 02:20.25 | lmadsen | tsk tsk tsk |
| 02:20.25 | Juggie | you broke it all! |
| 02:20.32 | Juggie | zzzz, so sleepy |
| 02:20.33 | russellb | lies |
| 02:20.34 | russellb | all lies |
| 02:20.37 | lmadsen | you're right |
| 02:20.38 | Juggie | your right |
| 02:20.41 | Juggie | file breaks it all |
| 02:20.42 | lmadsen | you called my bluff |
| 02:20.44 | Juggie | russellb fixes it |
| 02:20.50 | *** kick/#asterisk-dev [Juggie!n=Leif@asterisk/documenteur-extraordinaire/blitzrage] by lmadsen (learn how to use the right words pls!) |
| 02:21.02 | *** join/#asterisk-dev Juggie (i=Juggie@CPE001601df17fb-CM001a6682d7b6.cpe.net.cable.rogers.com) |
| 02:21.06 | Juggie | i hate you lmadsen |
| 02:21.12 | lmadsen | get in line! |
| 02:21.22 | Juggie | your going to make me install an irc grammar checker! |
| 02:21.31 | Juggie | you and your nazi grammar |
| 02:21.33 | *** kick/#asterisk-dev [Juggie!n=Leif@asterisk/documenteur-extraordinaire/blitzrage] by lmadsen (try again!) |
| 02:21.40 | *** join/#asterisk-dev Juggie (i=Juggie@CPE001601df17fb-CM001a6682d7b6.cpe.net.cable.rogers.com) |
| 02:21.44 | Juggie | sobs |
| 02:21.51 | lmadsen | your != you're |
| 02:22.00 | lmadsen | in fact... just say 'you are' |
| 02:22.22 | Juggie | i'm just not going to talk any more lmadsen! :( |
| 02:22.49 | lmadsen | you can talk... just learn the difference between "your" and "you're" |
| 02:23.00 | lmadsen | I don't even say the words the same... the inflection should be different |
| 02:23.22 | Juggie | you know its not a big deal to everyone but you :) |
| 02:23.38 | lmadsen | and seanbright :) |
| 02:23.50 | lmadsen | and I think you meant to say 'anyone' :) |
| 02:24.18 | Laureano | Bye guys. See you tomorrow. |
| 02:24.22 | lmadsen | night |
| 02:24.34 | lmadsen | I'm going to bed now... feel free to use as much incorrect grammar as you wish Juggie ;) |
| 02:24.58 | Juggie | i will.... i'm going to use the word 'your' as much as possible to get it out of my system :) |
| 02:25.22 | brimstone | your silly Juggie |
| 02:26.08 | brimstone | someone else are having this same problem with there system |
| 02:26.18 | brimstone | http://bugs.digium.com/view.php?id=12750 |
| 02:26.29 | brimstone | or is there some fancy bug bot in here? |
| 02:28.56 | russellb | M12750 |
| 02:28.57 | MuffinMan | [new] [Zaptel] General 0012750: Can't install zaptel trough sudo reported by atis (Karma: +13.25) http://bugs.digium.com/view.php?id=12750 |
| 02:29.05 | brimstone | ah |
| 03:01.41 | *** join/#asterisk-dev bkruse (n=bkruse@phoenixii.total-vision.net) |
| 03:01.41 | *** mode/#asterisk-dev [+o bkruse] by ChanServ |
| 03:24.41 | *** join/#asterisk-dev CunningPike (n=arodgers@vpn.dnv.org) |
| 03:28.09 | *** part/#asterisk-dev brimstone (n=brimston@pdpc/sponsor/digium/brimstone) |
| 04:47.26 | *** join/#asterisk-dev IgorG (n=IgorG@195.162.49.23) |
| 05:16.10 | *** join/#asterisk-dev pputman (n=centrex@216.207.245.1) |
| 05:24.09 | *** join/#asterisk-dev IgorG (n=IgorG@195.162.49.23) |
| 05:52.44 | *** join/#asterisk-dev denon (n=denon@tooth.decay.org) |
| 06:12.42 | *** join/#asterisk-dev denon (n=denon@tooth.decay.org) |
| 06:29.55 | *** join/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 06:48.24 | *** join/#asterisk-dev tzafrir_laptop (n=tzafrir@local.xorcom.com) |
| 06:48.24 | *** mode/#asterisk-dev [+o tzafrir_laptop] by ChanServ |
| 06:53.34 | *** join/#asterisk-dev oej (n=olle@bkkb-gw.voop.net) |
| 06:53.34 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 06:54.25 | oej | Morning, friends and foes |
| 07:03.55 | jmls | foes ? The swedish have no foes |
| 07:04.08 | jmls | not for a 1000 years, at least ;) |
| 07:04.47 | jmls | on the other hand, the germans, french, english have *plenty* :p |
| 07:25.28 | *** join/#asterisk-dev tengulre (n=tengulre@125.71.208.16) |
| 07:34.49 | tzafrir_laptop | jmls, 1000? make that 300 years or so |
| 08:43.30 | jmls | I was wrong. They are a belligerent race ;) |
| 08:43.31 | jmls | http://en.wikipedia.org/wiki/List_of_Swedish_battles |
| 08:47.18 | *** join/#asterisk-dev gmaruzz (n=gmaruzz@213-140-6-122.ip.fastwebnet.it) |
| 09:02.06 | *** join/#asterisk-dev pputman (n=centrex@216.207.245.1) |
| 09:27.56 | *** join/#asterisk-dev _ys (i=yuri@91.151.196.254) |
| 09:38.43 | *** join/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 09:39.04 | *** part/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 09:42.59 | *** join/#asterisk-dev sergee (n=serg@voip1.west-call.com) |
| 09:45.02 | tzafrir_laptop | jmls, look who's talking. who sided them last time they were on a fight? |
| 10:21.13 | *** join/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 10:25.31 | *** join/#asterisk-dev kamanashisroy (n=kamanash@202.56.7.138) |
| 10:27.38 | *** join/#asterisk-dev qdk (n=qdk@85.235.253.139) |
| 11:22.24 | *** part/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 11:26.46 | *** join/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 11:38.24 | *** join/#asterisk-dev RoyK (n=roy@ip-192-27-149-91.dialup.ice.no) |
| 11:59.21 | *** join/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 12:06.02 | *** part/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 12:14.51 | *** join/#asterisk-dev caio1982 (i=caio1982@CAcert-br/caio1982) |
| 12:24.26 | *** join/#asterisk-dev agx (n=AGX@host63-216-static.34-88-b.business.telecomitalia.it) |
| 12:25.25 | *** join/#asterisk-dev lmadsen (n=Leif@asterisk/documenteur-extraordinaire/blitzrage) |
| 12:25.25 | *** mode/#asterisk-dev [+o lmadsen] by ChanServ |
| 12:29.13 | *** join/#asterisk-dev shido6 (n=shido6@74-130-224-188.dhcp.insightbb.com) |
| 12:33.18 | *** join/#asterisk-dev RoyK (n=roy@ip-80-5-149-91.dialup.ice.no) |
| 12:34.23 | agx | "jointcapability" is the field inside a channel structure that hold the current codec for the call? |
| 12:40.28 | ctooley | oej, sent you the last diff for 300 Multiple choices "Broken style" |
| 12:40.59 | oej | Ok, did you switch from the To header to the request uri to avoid extra parsing? |
| 12:42.14 | ctooley | Note the "broken style" |
| 12:42.35 | ctooley | I'm still trying to figure out why it's not matching the ACK to the message when it comes in. |
| 12:43.43 | ctooley | But, now that I've gotten you _that_ I'll go modify it to use the URI and setting the "data" as the domain in contact headers |
| 12:47.53 | ctooley | oej, You know of some other function that uses the request URI? I don't even know where it's stored in the pvt struct |
| 12:56.25 | *** join/#asterisk-dev anonymouz666 (n=anonymou@201.19.140.193) |
| 12:56.49 | oej | The request uri is parsed and stored in the initreq |
| 12:57.06 | oej | It's used everywhere, check the function that initiates a response and you'll find it. |
| 12:57.09 | *** join/#asterisk-dev DagMoller (n=aguirre@unaffiliated/dagmoller) |
| 12:58.39 | ctooley | oej, thanks |
| 12:59.07 | oej | np |
| 12:59.42 | ctooley | doh! AST_STRING_FIELD(uri); /*!< Original requested URI */ |
| 13:01.22 | oej | :-) |
| 13:06.06 | ctooley | oej, if you look at sip_redirect_multiplechoices, that entire if block doesn't actually even do anything that I can see. (It's been a long time since we wrote this code) |
| 13:24.54 | ctooley | oej, nm, figured it out myself, going to need the if(!host) block to build Contact headers anyway though. |
| 13:28.00 | *** part/#asterisk-dev agx (n=AGX@host63-216-static.34-88-b.business.telecomitalia.it) |
| 13:31.51 | oej | Right, but you don't need to parse the To: any more |
| 13:32.02 | ctooley | oej, never really did. :) |
| 13:32.35 | ctooley | Actually, I'll still need to pull the extension out of the To: in case it's different than what's in the req uri, don't I? |
| 13:34.18 | oej | No. That's a big NO. |
| 13:34.25 | oej | You should never ever route on To: |
| 13:34.29 | oej | (Unless you have to) |
| 13:34.41 | oej | The request URI is what you want. Nothing else. |
| 13:34.52 | oej | As I said yesterday, the To: header is mostly cosmetics |
| 13:36.28 | oej | It's like e-mail. The To: inside the mail is not used for mail routing at all. |
| 13:36.44 | ctooley | ok |
| 13:55.32 | putnopvut | ctooley: by the way the uri field of the sip_pvt struct is actually the uri from the Contact: header, not the request URI specified on the first line of the SIP request. |
| 13:58.18 | ctooley | uhhh ok |
| 14:00.00 | ctooley | cusses the SSL cert some more |
| 14:00.15 | ctooley | sees that cussing is having very little effect so goes in search of a hammer. |
| 14:00.37 | *** join/#asterisk-dev hubguruJR (n=hubguruJ@mail.ntegratedsolutions.com) |
| 14:01.02 | *** part/#asterisk-dev hubguruJR (n=hubguruJ@mail.ntegratedsolutions.com) |
| 14:01.24 | *** join/#asterisk-dev bweschke (n=BJ_Wesch@c-76-124-82-215.hsd1.nj.comcast.net) |
| 14:01.25 | *** mode/#asterisk-dev [+o bweschke] by ChanServ |
| 14:02.00 | file | bweschke: ltns! |
| 14:02.18 | bweschke | file: Hi! Long time no chat! |
| 14:02.45 | *** join/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 14:03.25 | file | bweschke: how goes? |
| 14:03.35 | bweschke | not so bad .. not so bad .. |
| 14:03.42 | *** join/#asterisk-dev shido6 (n=shido6@74-130-224-188.dhcp.insightbb.com) |
| 14:03.45 | bweschke | how goes with you and the rest of the good Digium folk? |
| 14:04.04 | jsmith | Well if it itsn't bweschke |
| 14:04.10 | *** join/#asterisk-dev codefreeze-lap (n=murf@216.166.159.235) |
| 14:04.10 | *** mode/#asterisk-dev [+o codefreeze-lap] by ChanServ |
| 14:04.11 | bweschke | jsmith: hi! |
| 14:04.54 | file | is peachy |
| 14:05.15 | Corydon76-dig | is nectariney |
| 14:06.36 | *** join/#asterisk-dev _Krieger_ (n=krieger@193.39.118.158) |
| 14:08.07 | bweschke | mmmmm.. |
| 14:08.57 | bweschke | ummm.... I've seen a bunch of posts from russellb about the state of chan_iax2 after a security fix .. is that kinda all worked out now, or is chan_iax2 in 1.4 branch still not so happy? |
| 14:09.58 | file | should be all worked out |
| 14:11.09 | Corydon76-dig | bweschke: technically, the security fix didn't cause the issue... it just exacerbated an existing issue |
| 14:11.55 | bweschke | Corydon76-dig: got ya .. I've got a deadlock situation in chan_iax2, if not all of Asterisk core on 1.4 branch from a couple days ago .. that's reproduced itself probably 2-3 times now .. |
| 14:12.15 | bweschke | full DEBUG log just scrolls with: chan_iax2.c: Avoiding IAX destroy deadlock |
| 14:12.53 | Corydon76-dig | bweschke: compile with DEBUG_THREADS and get a 'core show locks' when it occurs |
| 14:13.16 | bweschke | Corydon76-dig: I think I did that already just as a factor of always putting that on |
| 14:13.20 | bweschke | I've got it in that state now |
| 14:13.25 | bweschke | lemme dbl check and see what I've got |
| 14:13.27 | bweschke | one sec |
| 14:13.45 | bweschke | r is 121442 |
| 14:14.08 | bweschke | shoot |
| 14:14.09 | bweschke | no |
| 14:14.11 | bweschke | DEBUG_THREADS |
| 14:14.15 | bweschke | lemme recompile |
| 14:14.17 | bweschke | and kick it |
| 14:15.49 | putnopvut | How have you been bweschke? |
| 14:17.46 | exvito | hi all. i'm trying to diagnose an apparent asterisk 1.4.20.1 hang - now running with don't optimize + debug locks + debug threads. asterisk hung again about 5 hours ago and, again, while writing this! when i tried to run 'gdb /usr/sbin/asterisk $(pidof asterisk)' it resulted in a gdb hang ! any hints on how to proceed diagnostic ? (checkout http://pastebin.ca/1044941 for a log of CLI errors + increasing locks + gdb hung output) |
| 14:17.49 | bweschke | putnopvut: I've been pretty good... busy.. crazy busy with one particular client .. unfortunately not Asterisk related ... but hopefully going to get back in to the mix again shortly |
| 14:18.16 | ctooley | From #rhel: <ctooley> Can someone tell me what to change on the server side to resolve: https://bugzilla.redhat.com/show_bug.cgi?id=446392 |
| 14:18.23 | ctooley | <merlinthp> ctooley: It looks like the cert needs regenerating with some feature turned on, but I'm not sure exactly what. |
| 14:18.30 | ctooley | That doesn't look super good. |
| 14:18.40 | ctooley | hopefully merlinthp is wrong, that would suck |
| 14:18.49 | putnopvut | exvito: I'm looking at your pastebin right now... |
| 14:18.53 | jsmith | ctooley: Yeah, we basically know that already... but it's not the personal cert I don't think |
| 14:18.56 | exvito | putnopvut: ...thanks. |
| 14:19.18 | ctooley | jsmith, if you regen the server cert I'd think you'd have to regen the client cert too |
| 14:19.46 | jsmith | ctooley: Yeah... I don't know if it really requires a complete regen, or if you can just tweak the bits on the server cert |
| 14:22.07 | putnopvut | exvito: it's a shame that gdb hung there. a backtrace from the thread that is constantly holding that channel lock would be helpful. I'll see if I can figure out what exactly is happening there. |
| 14:22.43 | putnopvut | exvito: by the way, is this Asterisk 1.4? |
| 14:22.49 | putnopvut | Ah, yeah it is. |
| 14:22.53 | exvito | putnopvut: I'm increasingly suspecting the TC400B we installed a week ago. The system started hanging systematically since then... |
| 14:23.23 | exvito | putnopvut: ...since asterisk/gdb do not respond to signals could one assume a bug in TC400B driver / bad HW ? |
| 14:23.41 | putnopvut | exvito: it's certainly possible. |
| 14:24.23 | *** join/#asterisk-dev sruffell (n=sruffell@216.207.245.1) |
| 14:24.23 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 14:24.46 | exvito | putnopvut: bad hw would be very unfortunate as this is the 3rd TC400B we are trying... (the other two didn't even get recognized by the driver!) |
| 14:25.06 | putnopvut | sruffell: you may be one to ask on this. Have you heard of any deadlock issues happening when using TC400B cards? |
| 14:28.32 | exvito | putnopvut: btw, i forgot to mention that after attaching and hanging, ps shows gdb as <defunct>... funny ? |
| 14:30.01 | putnopvut | That is odd. I'm looking at the code and what seems weird to me is that there doesn't appear to be any code that could block indefinitely or loop infinitely in the path that is currently holding the channel lock. |
| 14:31.28 | sruffell | putnopvut: yes I've seen it on SMP systems. Deadlock with TC400B cards that is. |
| 14:31.58 | putnopvut | sruffell: any leads on what's causing them? |
| 14:32.50 | sruffell | putnopvut: yes....and I believe I have them corrected in the development branch. |
| 14:33.30 | putnopvut | sruffell: ah, cool. By any chance does it have to do with the module use count updater? That's the only thing I'm seeing from Asterisk that even remotely looks like it could be hanging? |
| 14:33.32 | sruffell | exvito: Could you try my branch? I still have a few more things to do before merging it into the trunk, but if you're willing. |
| 14:34.11 | exvito | sruffell: ...how "distant" would you say your branch is from 1.4.20.1 ? (we're trying to run a production system here...) :-) |
| 14:34.21 | sruffell | no...I don't think it had to do with module use count. |
| 14:34.51 | sruffell | exvito: I have a few more stress tests to run, but by the end of this week. I'm working on back porting it to older kernels now. |
| 14:35.29 | sruffell | it would be a change to both asterisk and zaptel. |
| 14:35.45 | exvito | sruffell: this system is running latest centos 5.1 kernel 2.6.18-53.1.21.el5... |
| 14:36.01 | sruffell | http://svn.digium.com/svn/zaptel/team/sruffell/zaptel-1.4-transcoder and http://svn.digium.com/svn/asterisk/team/sruffell/asterisk-1.4-transcoder |
| 14:36.31 | sruffell | That is the same specs as one of my production systems in the lab here. |
| 14:36.40 | exvito | sruffell: not versed in svn... can you give a hint as on how to co it ? |
| 14:37.27 | sruffell | 'svn co http://svn.digium.com/svn/asterisk/team/sruffell/asterisk-1.4-transcoder' |
| 14:37.46 | sruffell | and 'svn co http://svn.digium.com/svn/zaptel/team/sruffell/zaptel-1.4-transcoder' |
| 14:38.02 | exvito | ..then build + install as usual, i guess ? |
| 14:38.23 | sruffell | yes...but make sure you install the zaptel branch before building the asterisk branch, so it picks up the new zaptel header. |
| 14:38.37 | exvito | sruffell: ok... |
| 14:40.15 | sruffell | also, with the current state of the branches in the repository, you might get a sip channel lock error if transcoding from sip to sip. I haven't yet committed my work to make the allocation of the transcoding channel non-blocking. |
| 14:41.09 | exvito | sruffell: putnopvut: thanks for your support... we'll give these branches a try... |
| 14:41.24 | exvito | FYI... IT JUST HUNG AGAIN ! :(( |
| 14:41.37 | exvito | (same behaviour, gdb hangs, gdb defunct...) |
| 14:45.11 | exvito | putnopvut: ...since I can't read 'core show locks' output, could you please check that http://pastebin.ca/1044993 is the same behaviour as earlier ? |
| 14:46.08 | putnopvut | Yes it is. There are more threads waiting on locks in this second set of output, but it's still the same call path that is blocking the other threads. |
| 14:47.06 | exvito | putnopvut: ...ok, thanks. this time i managed to get the channel list... the iax channels are g.729 so yes, i'm increasingly inclined to the tc400b / driver... |
| 14:48.36 | file | yup... |
| 14:49.48 | file | it is probably hanging somewhere in zap_translate of codec_zap |
| 14:51.15 | *** join/#asterisk-dev jpeeler (n=jpeeler@216.207.245.1) |
| 14:51.15 | *** mode/#asterisk-dev [+o jpeeler] by ChanServ |
| 15:00.56 | exvito | sruffell: i was taking a peek at your branch's zaptel/wctc4xxp and found codec_test_threaded.c which i assume is a util to stress test the driver/hw |
| 15:01.35 | exvito | sruffell: ...if what we're experiencing is something related to a deadlock/hang in kernelspace/hw, would running this tool be enough to bring out the hang ? |
| 15:02.07 | exvito | sruffell: ...could the tool be run on top of our current zaptel (1.4.11) ? |
| 15:03.38 | *** join/#asterisk-dev smth (n=chatzill@CPE000d48380389-CM00195eda2522.cpe.net.cable.rogers.com) |
| 15:04.03 | sruffell | exvito: no...it can't, because of the interface change. |
| 15:04.20 | sruffell | exvito: I have that tool in there to help me test the driver in isolation of anything going on with asterisk. |
| 15:05.37 | sruffell | exvito: but based on what you're seeing...I'd be surprised if you're not having problems with the wctc4xxp driver in 1.4.11. |
| 15:05.54 | *** join/#asterisk-dev Cresl1n (n=matt@216.207.245.1) |
| 15:05.54 | *** mode/#asterisk-dev [+o Cresl1n] by ChanServ |
| 15:06.34 | file | Cresl1n: !!! |
| 15:06.59 | Cresl1n | hey file! |
| 15:07.30 | exvito | sruffell: ok... i'm trying to evaluate current options regarding stability vs. identifying+fixing this: a) I can move from g.729 to gsm and disable wctc4xxp driver loading and, hopefully, obtain stability... and later (when?) test your branch/newer asterisk/zaptel releases... or b) I can move onwards immediately with your branch -- IIRC you mentioned that by the end of the week your branch would be more stress-tested and, thus, better... advice/feeling |
| 15:07.52 | exvito | sruffell: (we don't do sip-sip transcoding) |
| 15:09.43 | sruffell | exvito: for maximum stability, and if gsm is an option, I would move to it until the branches are merged into the trunk. If you really want to use g729 and are accepting of some hiccups, there are other customers who have the branches in beta currently. but if gsm is a viable option for you, I would just use that until you have a good maintenance window. |
| 15:10.54 | exvito | sruffell: ...what would you consider a "good maintenance window" ? for what purpose ? |
| 15:11.19 | sruffell | exvito: Time to install everything and test your configuration without upsetting your users. |
| 15:12.28 | exvito | sruffell: ...the solution is quite simple. The main issue with _testing_ is triggering vs. not triggering this hang - this is something we haven't figured out exactly how to. |
| 15:15.33 | sruffell | exvito: I don't know if I have a good answer for you on how to quickly generate the hang under your conditions. |
| 15:16.05 | *** join/#asterisk-dev kamanashisroy (n=kamanash@202.56.7.134) |
| 15:18.45 | *** join/#asterisk-dev russellb (n=russell@asterisk/developer-and-stable-maintainer/drumkilla) |
| 15:18.45 | *** mode/#asterisk-dev [+o russellb] by ChanServ |
| 15:19.00 | exvito | sruffell: ...sure, thanks. i was just thinking out loud... so i guess that, for now, we'll go back to gsm as i won't be able to assist the system in a 100% timely manner neither tomorrow nor friday... i expect things to go smoothly this way. monday evening we'll switch to your branch and see how we go from there... |
| 15:19.16 | exvito | ...again, putnopvut + sruffell, thanks a lot for your feedback. |
| 15:20.13 | ctooley | oej, BTW I'm beating up Asterisk using that routing code I have been sending you patches for. 250 calls per second are being parsed, data sent to the routing server, and asterisk is responding. |
| 15:20.36 | oej | Cool |
| 15:20.37 | ctooley | And that's with Asterisk not even correlating ACKs correctly, so there's a timeout on every 300 response. |
| 15:20.42 | exvito | sruffell: so you would consider that current tc400b driver in zaptel 1.4.11 is not good enough for smp systems, is that it ? |
| 15:20.43 | oej | Ouch |
| 15:23.20 | sruffell | exvito: I can't say that, all I know is that I've personally had issues with it, and rolling in stability changes as part of the change to the interface that I was tasked to make anyway seemed more expedient than working down both paths. |
| 15:24.30 | exvito | sruffell: ...sure. again, thanks. i'll give your branch a try next tuesday. for now i'll try to disable hw transcoding asap... |
| 15:24.37 | ctooley | grrr can't remember how to attach to running asterisk to get a backtrace while it's deadlocked. |
| 15:24.57 | ctooley | putnopvut, I'm trying to get you the full trace of the locked sip_read, need to know what you want. |
| 15:25.31 | putnopvut | ctooley: to attach to asterisk, do gdb /usr/sbin/asterisk `pidof asterisk` |
| 15:25.48 | putnopvut | For locks, the thread apply all bt from gdb and core show locks from the CLI are useful. |
| 15:25.57 | ctooley | putnopvut, did that, when i do 'thread apply all bt' I get no output |
| 15:26.13 | putnopvut | um, that's weird. |
| 15:26.13 | ctooley | just returns to the gdb prompt |
| 15:26.32 | ctooley | OOOPS |
| 15:26.36 | ctooley | wrong asterisk binary |
| 15:26.41 | putnopvut | ah |
| 15:26.46 | ctooley | hehe |
| 15:27.20 | *** join/#asterisk-dev puzzled (n=patrick@puzzled.xs4all.nl) |
| 15:30.38 | *** join/#asterisk-dev CunningPike (n=arodgers@204.239.8.149) |
| 15:31.22 | *** join/#asterisk-dev Deeewayne (n=Deeewayn@216.207.245.1) |
| 15:31.22 | *** mode/#asterisk-dev [+o Deeewayne] by ChanServ |
| 15:33.15 | *** join/#asterisk-dev igascream (n=igascrea@80.179.192.178.cable.012.net.il) |
| 15:34.06 | ctooley | putnopvut, http://www.pastebin.ca/1045069 |
| 15:34.25 | ctooley | which thread do you want a full trace of and how do I do a full back trace of a single thread? |
| 15:34.37 | igascream | HI all need help with app_rxfax ????? |
| 15:35.02 | russellb | igascream: #asterisk |
| 15:36.00 | putnopvut | ctooley: I'm still checking it out. In order to get a trace from a specific thread, you first select it in gdb with "thread <num>" |
| 15:36.03 | igascream | I think you should understand it better : It's about the code |
| 15:36.06 | putnopvut | Then you can get a bt full of it. |
| 15:37.03 | putnopvut | ctooley: here's something interesting. There are clearly threads waiting on locks but they're not showing up in the core show locks output. |
| 15:37.21 | ctooley | putnopvut, dunno man. |
| 15:37.50 | putnopvut | ctooley: actually, it's just one thread waiting on a sip_pvt lock. |
| 15:37.55 | ctooley | has an idea to simplify things... I'll lock it up again and get another round of traces |
| 15:38.00 | *** join/#asterisk-dev anonymouz666 (n=anonymou@201.19.140.193) |
| 15:38.09 | ctooley | can lock it up pretty reliably. |
| 15:39.07 | putnopvut | ctooley: thanks, it's not clear at all where/why it's locked up. |
| 15:40.43 | ctooley | pulled chan_iax2 out altogether as I don't use it and half the stuff in the backtrace was related to that. |
| 15:41.07 | ctooley | I can actually remove a lot of stuff. |
| 15:41.23 | ctooley | of course that will probably result in it taking longer to lock up for some bizarre reason. :) |
| 15:41.27 | igascream | russellb: simple question if you know what may cause the Channel T30 DONE < 0 reply? it enter the loop for a long time )) i changed res=1 and break on it just then I can receive the fax |
| 15:42.07 | *** join/#asterisk-dev outtolunc (n=me@adsl-66-218-53-172.dslextreme.com) |
| 15:43.58 | igascream | russellb: or still #asterisk ))))))) |
| 15:50.00 | tzanger | interesting |
| 15:50.14 | tzanger | I get a kernel trace when I head /proc/zaptel/# for a span in RED alarm |
| 15:51.05 | ctooley | grrrr putnopvut I removed chan_iax2 and it's definitely at least taking longer to crash |
| 15:52.01 | *** join/#asterisk-dev joetester2 (n=chatzill@216.191.34.13) |
| 15:52.11 | ctooley | Just ran 100,000 calls through at 200 calls per second and it's not locking. |
| 15:52.36 | putnopvut | ctooley: that's not entirely a bad thing. |
| 15:52.40 | ctooley | I need to figure out how to tell it not to worry about RTP if there's no SDP |
| 15:53.05 | ctooley | putnopvut, no, it's not |
| 15:55.22 | *** part/#asterisk-dev igascream (n=igascrea@80.179.192.178.cable.012.net.il) |
| 15:57.31 | tzafrir_laptop | tzanger, that's nice |
| 15:57.42 | tzafrir_laptop | could you be more specific? |
| 15:59.16 | tzanger | sure |
| 15:59.23 | tzanger | http://pastebin.ca/1045033 |
| 16:01.17 | tzafrir_laptop | tzanger, did it actually printk anything? |
| 16:01.25 | ctooley | putnopvut, over 200K calls now. |
| 16:01.45 | tzafrir_laptop | osunds like a curropted field somewhere at the zt_span or so |
| 16:01.50 | tzanger | tzafrir_laptop: no, just the trace |
| 16:02.02 | tzanger | mind you I am logged in via ssh |
| 16:02.13 | tzanger | the particular system may not printk anything (although that usually ends up in dmesg anyway) |
| 16:02.18 | tzafrir_laptop | So it's in the first line. e.g: in the span name or description |
| 16:02.55 | *** join/#asterisk-dev uTx (n=unix@modemcable232.79-58-74.mc.videotron.ca) |
| 16:03.03 | tzanger | tzafrir_laptop: the 4 tdmoe spans are pretty much identical, but I'll check again |
| 16:03.25 | tzanger | I mean cat /proc/zaptel/1 lists fine (link is fine), as is /2 |
| 16:03.37 | tzafrir_laptop | sorry, my mistake: of source it won't print anyththing partially |
| 16:03.40 | tzanger | <PROTECTED> |
| 16:03.43 | ctooley | starting a new 1M call job to see if I can crash it with a very minimal set of modules loaded, only the things I actually need. |
| 16:09.49 | tzafrir_laptop | other ways to look at similar information: zttool |
| 16:10.35 | jsmith | ctooley: So you're blaming the IAX2 channel driver, even though you're not sending any calls through IAX? |
| 16:10.50 | ctooley | jsmith, no, absolutely not |
| 16:11.09 | ctooley | very likely it's just that having other things loaded exacerbates the problem. |
| 16:11.44 | jsmith | I'd be very interested to know which modules exacerbate the issue |
| 16:12.43 | ctooley | jsmith, just removing chan_iax2 seemed to help |
| 16:12.54 | ctooley | I removed it first and did 100K calls at 200cps. |
| 16:13.03 | ctooley | Before that I couldn't get past 25K at that rate. |
| 16:13.15 | ctooley | usually died after 5-15K |
| 16:13.42 | jsmith | Very interesting |
| 16:13.56 | ctooley | jsmith, could have very well been just pure chance though. |
| 16:14.10 | ctooley | After this million calls is done I'll run another job with everything except chan_iax2 in. |
| 16:14.41 | Juggie | ctooley, do you run your boxes in a minimal config, eg, selectively load only the modules you need? |
| 16:14.55 | ctooley | Juggie, that box wasn't |
| 16:15.02 | ctooley | it is now |
| 16:20.55 | ctooley | putnopvut, if it runs to completion it'll be done in a little under an hour and a half. If it doesn't I'll have trace data for you guys. |
| 16:24.31 | putnopvut | ctooley: All righty. I'll actually be at lunch an hour and a half from now, but if you have trace data, I'll look at it after lunch. |
| 16:25.00 | Juggie | putnopvut, who said you could eat! |
| 16:25.12 | ctooley | putnopvut, I'll be at lunch by then too, but we'll see what happens. |
| 16:25.26 | putnopvut | Juggie: he did --> |
| 16:26.30 | ctooley | <PROTECTED> |
| 16:26.32 | ctooley | :) |
| 16:26.56 | ctooley | I say he gets to eat, I want him to look at my deadlock data. :) |
| 16:45.12 | *** join/#asterisk-dev CunningPike (n=arodgers@204.239.8.151) |
| 16:57.30 | Corydon76-dig | russellb: ping |
| 16:58.13 | file | I believe he is in meetings |
| 16:58.15 | jpeeler | he just left i believe |
| 16:58.20 | Corydon76-dig | Changing the length of a CDR field would break binary compatibility in 1.4, so... should we or not? |
| 16:58.46 | Corydon76-dig | Someone pointed out that uniqueid is not long enough when you add in systemname |
| 16:59.17 | codefreeze-lap | Probably not! Adding in sysname is something not to do in 1.4 also! |
| 17:00.08 | ctooley | Isn't uniqueid an auto_increment int? |
| 17:00.12 | codefreeze-lap | Not unless you want screaming hordes of CDR application designers screaming and waving pitchforks |
| 17:00.22 | *** join/#asterisk-dev jameswf-home (n=james@dsl093-157-131.phx1.dsl.speakeasy.net) |
| 17:00.29 | codefreeze-lap | ctooley: sort of |
| 17:00.38 | Corydon76-dig | codefreeze-lap: yeah, that was my thought, as well |
| 17:01.54 | codefreeze-lap | I didn't even mention the hot tar and pitchforks. I guess these days, they'd use superglue. :-D |
| 17:02.03 | codefreeze-lap | eh, I meant feathers |
| 17:02.19 | codefreeze-lap | but pitchforks might be used to tenderize |
| 17:11.10 | jsmith | jpeeler: As long as you're updating backtrace.txt, you may want to add a note about "set paginate off" |
| 17:12.11 | jpeeler | jsmith: yeah, i forgot about that |
| 17:14.54 | jsmith | jpeeler: I'd make the change myself, buy since I'm running Fedora 9, I can't :-( |
| 17:15.05 | Corydon76-dig | jpeeler: while you're at it, add "list" |
| 17:15.29 | jsmith | All in favor of better documentation, say "Aye!" |
| 17:18.07 | *** join/#asterisk-dev JT (n=j@unaffiliated/jt) |
| 17:18.36 | jpeeler | Corydon76-dig: list to display the source? |
| 17:18.48 | Corydon76-dig | Right |
| 17:19.04 | Corydon76-dig | Displays the 5 lines immediately before and after |
| 17:19.15 | Corydon76-dig | Helps when the source has changed since |
| 17:19.43 | jpeeler | ah good point |
| 17:19.58 | Corydon76-dig | If the crash is in a header file, I also sometimes ask for "frame 1" "list" |
| 17:20.41 | Corydon76-dig | (to see where in the C file it crashed) |
| 17:20.49 | jpeeler | right... |
| 17:21.06 | jpeeler | well i certainly want to maintain the simplicity of the doc for people who have never touched gdb before |
| 17:22.32 | *** part/#asterisk-dev exvito (n=exvito@195.23.54.153) |
| 17:22.34 | Corydon76-dig | jpeeler: teach people small tricks, and they learn... |
| 17:22.39 | *** join/#asterisk-dev outtolunc (n=me@adsl-66-218-53-172.dslextreme.com) |
| 17:23.26 | ctooley | oej, is there a reason that the RTP setup occurs if an INVITE does not contain SDP? |
| 17:25.36 | jsmith | ctooley: Well, I can't answer that, but I do know that for example if you have video support turned on, Asterisk will allocate RTP ports for video, even if the SDP doesn't do anything related to video |
| 17:28.47 | ctooley | jsmith, right, which seems not only pointless but counterproductive. |
| 17:29.35 | jsmith | ctooley: Yes, it seems that way to me too, but I don't know the code, and I don't know the reasons why those decisions were made |
| 17:29.52 | jsmith | ctooley: It's easy for me to point fingers and say "That sounds stupid", because I don't know enough :-) |
| 17:30.18 | mvanbaak | well, that sounds stupid ;) |
| 17:31.00 | ctooley | jsmith, I'm hoping to be educated as to the rationale. Because it seems like a big no-no |
| 17:32.18 | oej | ctooley: yes, it's delayed call setup. |
| 17:32.32 | oej | We answer with our SDP in the 200 OK, then they answer with their SDP in the ACK |
| 17:32.36 | oej | So we're required to do that |
| 17:33.02 | ctooley | oej, but then we should spin the RTP stuff up at the time we're going to send the 200 OK |
| 17:33.06 | *** join/#asterisk-dev nny_1 (n=Scott_My@64.203.239.83) |
| 17:33.27 | ctooley | If we're not going to answer, there's no need for RTP |
| 17:33.43 | ctooley | is sure he's sounding like a complete moron. |
| 17:34.07 | oej | ctooley: There are tons of optimizations we could do in the sip channel. For myself, I bundled them together and gave them a fruit name |
| 17:34.41 | oej | ctooley: This actually works, so I won't change it now just for optimization at this point... Too scared to destroy stuff. |
| 17:35.31 | ctooley | oej, so, the reason was "to make it work". That's a perfectly legitimate reason to do it. I'm just verifying that optimizing it was something _could_ be done. |
| 17:35.50 | mvanbaak | WHOA ! |
| 17:35.56 | mvanbaak | I had another thick ! |
| 17:36.12 | mvanbaak | freaking animals |
| 17:36.13 | mvanbaak | blegh |
| 17:39.42 | ctooley | Well.... good news and bad news putnopvut. The 1M call job running at 200cps didn't lock asterisk. But, I don't know why it's not locking now. So..... I'm running a 10M call job at the same rate to see what happens. |
| 17:39.46 | oej | ctooley: If so, we need to optimize the whole video/text/t.38 stuff at the same time |
| 17:39.58 | oej | That's a big mess |
| 17:40.08 | ctooley | oej, goodie, sounds like a couple hours of work, tops. |
| 17:40.09 | ctooley | :) |
| 17:42.33 | Corydon76-dig | ctooley: meanwhile some mid-level manager says "Let's see how it's doing..." and it immediately locks up |
| 17:42.52 | putnopvut | ctooley: yeah, that's a sure way to get it to lock up. |
| 17:43.42 | ctooley | has to wait another half hour for the mid-level managers to get back from lunch (which I haven't gotten to even get yet) |
| 17:51.10 | jsmith | ctooley: Don't forget about early media... we can start sending audio long before we send a 200 OK |
| 17:54.40 | ctooley | jsmith, have to have SDP for that |
| 17:55.02 | jsmith | ctooley: Right... just clarifying that... |
| 17:55.27 | jsmith | ctooley: I just wanted to make sure nobody thought that it always appropriate to wait for the 200 OK |
| 17:56.15 | ctooley | right, but it'd be nice to not start up RTP stuff until it's needed. |
| 17:59.26 | jsmith | ctooley: Absolutely! |
| 18:03.41 | *** join/#asterisk-dev esk (i=esk@esk.cs.usu.edu) |
| 18:22.45 | *** join/#asterisk-dev bkruse (n=bkruse@phoenixii.total-vision.net) |
| 18:22.45 | *** mode/#asterisk-dev [+o bkruse] by ChanServ |
| 18:26.21 | ctooley | putnopvut, it locked up |
| 18:26.31 | ctooley | http://www.pastebin.ca/1045235 |
| 18:28.31 | jsmith | ctooley: After how many calls? |
| 18:28.55 | Corydon76-dig | ctooley: that looks familiar |
| 18:28.56 | ctooley | 459K |
| 18:29.04 | ctooley | Corydon76-dig, yeah, I know |
| 18:29.29 | ctooley | Here's one with the a "bt full" of thread 3 |
| 18:29.31 | ctooley | http://www.pastebin.ca/1045240 |
| 18:29.32 | Corydon76-dig | bmd added something to do backtraces within core show locks, I thought |
| 18:29.59 | *** join/#asterisk-dev _ys (n=yuri@236-069.nat.mns.ru) |
| 18:30.05 | ctooley | Corydon76-dig, I tried upgrading to 1.4.21-rc2... it's worse. :( I'm going to set it up now and get debug information for it. |
| 18:31.22 | _ys | Hi |
| 18:32.08 | _ys | putnopvut You realy to can't reproduce #12835? |
| 18:33.04 | nny_1 | is there a variable that macthes hint states? |
| 18:33.07 | nny_1 | matches* |
| 18:34.08 | ctooley | putnopvut, and the sip.diff file: http://www.pastebin.ca/1045245 |
| 18:44.45 | Corydon76-dig | ctooley: I have a patch for you |
| 18:45.01 | ctooley | Corydon76-dig, ok |
| 18:45.15 | ctooley | against which version of chan_sip? |
| 18:45.37 | Corydon76-dig | It's against channel.c |
| 18:45.42 | ctooley | oh |
| 18:46.01 | Corydon76-dig | http://asterisk.drunkcoder.com/patches/20080611__for_ctooley.diff.txt |
| 18:46.37 | Corydon76-dig | That should stop the deadlock. There should also be an error message generated which might tell us what the exact problem was |
| 18:46.53 | ctooley | Corydon76-dig, is that trunk or 1.4? |
| 18:46.56 | Corydon76-dig | 1.4 |
| 18:47.04 | ctooley | ok, will test now |
| 18:47.38 | Corydon76-dig | Specifically, the backtrace indicates that it is blocking on a fd that should have been set nonblocking. |
| 18:47.53 | Corydon76-dig | Hence, I deduce that the setting of the nonblocking mode failed. |
| 18:48.00 | ctooley | hmm coo |
| 18:48.47 | ctooley | building now, will see how it does after recompile. |
| 18:48.51 | *** join/#asterisk-dev anonymouz666 (n=anonymou@201.19.140.193) |
| 18:50.57 | ctooley | Corydon76-dig, thanks, throwing a million calls at it now to test |
| 18:51.41 | Corydon76-dig | Cool |
| 18:52.24 | ctooley | So, I should see "Unable to set alertpipe nonblocking! .... " |
| 18:52.26 | ctooley | if it fails |
| 18:52.48 | anonymouz666 | what else is needed to put the manipulate audiohooks mode to work in 1.4.21? I mean the DTMF digits to control the volume. I already got FUNC_VOLUME working. |
| 18:55.43 | *** join/#asterisk-dev DagMoller (n=aguirre@unaffiliated/dagmoller) |
| 18:55.50 | *** join/#asterisk-dev qdk (n=qdk@0x573fe57e.bynqu2.broadband.tele.dk) |
| 19:03.17 | *** part/#asterisk-dev Rem| (n=remi@bg-fw2out.monmouth.com) |
| 19:20.20 | *** join/#asterisk-dev CunningPike (n=arodgers@204.239.8.157) |
| 19:27.55 | *** join/#asterisk-dev sruffell (n=sruffell@216.207.245.1) |
| 19:27.55 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 19:36.20 | putnopvut | ctooley: I'm back now. I'll take a look at them there traces. |
| 19:36.48 | putnopvut | ah, crud _ys left but yeah, I really can't reproduce it. |
| 19:38.45 | ctooley | [Jun 11 19:19:04] WARNING[22191]: channel.c:772 ast_channel_alloc: Unable to set alertpipe nonblocking! (9: Bad file descriptor) |
| 19:39.47 | jsmith | ctooley: You broke it again :-) w00t! |
| 19:39.59 | jsmith | Bad file descriptor, eh? |
| 19:40.04 | ctooley | jsmith, but, it moved on. |
| 19:40.06 | ctooley | MUCH better |
| 19:40.09 | putnopvut | ctooley: ah, interesting. it appears to be blocking while attempting to read. The fact that it wasn't set to non-blocking would make sense as to why it's blocking. |
| 19:40.28 | putnopvut | s/to read/to read the alertpipe/ |
| 19:43.19 | putnopvut | ctooley: did you add that warning message about not being able to set the alertpipe nonblocking? |
| 19:43.48 | jsmith | putnopvut: That came from a patch that Corydon76-dig made for him |
| 19:44.01 | ctooley | putnopvut, |
| 19:44.01 | ctooley | <Corydon76-dig> http://asterisk.drunkcoder.com/patches/20080611__for_ctooley.diff.txt |
| 19:44.48 | putnopvut | ctooley: ah, okay. |
| 19:45.58 | ctooley | putnopvut, but, it does beg the question of WTH? |
| 19:46.16 | putnopvut | Yeah, considering that the pipe() call was successful. |
| 19:46.30 | putnopvut | What's the ulimit for fd's on that system? |
| 19:47.23 | putnopvut | (that would be ulimit -n) |
| 19:47.50 | ctooley | 128000 |
| 19:47.57 | ctooley | I'm not running out of fd's |
| 19:48.03 | ctooley | hit that wall, blew it up |
| 19:48.16 | Corydon76-dig | putnopvut: if that was the problem, then pipe(2) would fail |
| 19:48.32 | putnopvut | Corydon76-dig: that's kind of what I thought too, but I wasn't certain. |
| 19:48.55 | ctooley | I am not really seeing more than 4-5000 open handles anymore |
| 19:49.41 | ctooley | I'm still unsure of why we're "losing" tcp sockets for app_externalivr.c either, but I figured I'd tackle the "asterisk just up and dies" problem first. |
| 19:54.14 | putnopvut | ctooley: that is just...bizarre. Why would fcntl fail on an fd that was just created? |
| 19:54.30 | ctooley | putnopvut, yeah, that's what I'd like to know. |
| 19:54.57 | ctooley | Why would a file descriptor for a socket that just said it had something to read suddenly not be there? |
| 19:56.14 | Corydon76-dig | If the file descriptor is closed, then a SIGPIPE will alert as if it had something to read |
| 19:56.49 | Corydon76-dig | IOW, poll(2) can be interrupted by a signal |
| 19:58.34 | putnopvut | Corydon76-dig: if it is interrupted, then ast_waitfor_nandfds will return NULL, indicating there is no data to read. |
| 20:01.05 | *** join/#asterisk-dev didz_ (n=hoje@201.19.140.193) |
| 20:03.52 | Corydon76-dig | Ah |
| 20:04.51 | ctooley | boom! there goes that server. |
| 20:05.26 | putnopvut | Perhaps what's happening is that some fd has data ready, but not the alertpipe. As a result, ast_read gets hung because it unconditionally reads from the alertpipe expecting it to be nonblocking. |
| 20:06.34 | Corydon76-dig | Wow, lots more analysis... |
| 20:06.39 | putnopvut | It still makes absolutely no sense to me how a brand new fd could be "bad." |
| 20:07.07 | *** join/#asterisk-dev russellb (n=russell@asterisk/developer-and-stable-maintainer/drumkilla) |
| 20:07.07 | *** join/#asterisk-dev jmls (n=jmls@host217-36-208-155.in-addr.btopenworld.com) |
| 20:07.07 | *** join/#asterisk-dev gmaruzz (n=gmaruzz@213-140-6-122.ip.fastwebnet.it) |
| 20:07.07 | *** join/#asterisk-dev Iamnacho (i=Iamnacho@ip68-103-153-140.ks.ok.cox.net) |
| 20:07.07 | *** join/#asterisk-dev seanbright (n=sean@c-69-251-175-43.hsd1.md.comcast.net) |
| 20:07.07 | *** join/#asterisk-dev file (n=file@asterisk/developer-and-muffin-lover/file) |
| 20:07.07 | *** mode/#asterisk-dev [+oovo russellb seanbright seanbright file] by irc.freenode.net |
| 20:07.07 | *** join/#asterisk-dev citats (n=james@mrplow.gnuinternet.com) |
| 20:07.07 | *** mode/#asterisk-dev [+o citats] by irc.freenode.net |
| 20:07.13 | putnopvut | Corydon76-dig: was that a jab at me? :) |
| 20:07.41 | Corydon76-dig | Not a jab... |
| 20:08.34 | *** join/#asterisk-dev Cresl1n (n=matt@216.207.245.1) |
| 20:08.34 | *** mode/#asterisk-dev [+o Cresl1n] by ChanServ |
| 20:12.48 | *** join/#asterisk-dev _ShrikE (n=_ShrikE@adsl-074-185-215-060.sip.msy.bellsouth.net) |
| 20:14.03 | ctooley | crap |
| 20:15.30 | ctooley | it locked up again. |
| 20:15.42 | ctooley | pulling new debug info. |
| 20:16.18 | ctooley | core show locks shows the lock in the same place |
| 20:19.07 | ctooley | http://www.pastebin.ca/1045359 |
| 20:23.13 | Corydon76-dig | wtf? |
| 20:23.57 | ctooley | Corydon76-dig, yeah, that's what I said too |
| 20:23.57 | Corydon76-dig | How the fuck is a file descriptor LOSING the nonblocking mode? |
| 20:24.20 | putnopvut | It's losing it? That's even weirder. |
| 20:24.32 | Corydon76-dig | putnopvut: gotta be |
| 20:25.56 | putnopvut | ctooley: are you using AGI on that box? |
| 20:27.01 | jsmith | putnopvut: I'll bet he's using ExternalIVR |
| 20:27.56 | ctooley | putnopvut, no AGI |
| 20:27.58 | ctooley | EIVR |
| 20:29.25 | putnopvut | ctooley: I'm not sure how reference counts work with fds internally, but perhaps the close(2) call that happens on fds 2-1024 in app_exec() in app_externalivr might be closing the alertpipe's fds? |
| 20:29.49 | putnopvut | Just a thought. |
| 20:29.49 | ctooley | How the hell would it be doing that? |
| 20:29.57 | ctooley | And why would it happen so infrequently. |
| 20:30.13 | ctooley | I mean it took 750K calls to blow up. |
| 20:30.21 | ctooley | I'm not saying you're wrong, just seems odd. |
| 20:30.53 | putnopvut | Yeah, like I said, it's just a thought. I'm probably wrong. |
| 20:32.24 | ctooley | putnopvut, hold on a sec. |
| 20:32.49 | putnopvut | Plus, I'm pretty certain that since that loop executes in a forked process, the closure of those fd's would probably just actually decrease an internal reference count, not close a valid fd in another thread. |
| 20:33.37 | ctooley | http://simplified.org/app_externalivr.c |
| 20:34.19 | ctooley | As the "maintainer" I"m running a slightly modified version (it's not even worth posting the diff, it's bigger than the file) |
| 20:35.57 | *** part/#asterisk-dev nny_1 (n=Scott_My@64.203.239.83) |
| 20:37.33 | Corydon76-dig | ctooley: http://asterisk.drunkcoder.com/patches/20080611__for_ctooley__2.diff.txt |
| 20:39.11 | ctooley | piss, I have to revert the first patch. |
| 20:39.24 | putnopvut | Corydon76-dig: there's a small problem with that patch. |
| 20:39.27 | ctooley | Did you change the first chunk at all? |
| 20:39.37 | putnopvut | if (fcntl(chan->alertpipe[0], F_SETFL, flags | O_NONBLOCK) == 0) { |
| 20:39.41 | putnopvut | That should be < 0 |
| 20:41.01 | Corydon76-dig | Oh, right |
| 20:41.30 | Corydon76-dig | http://asterisk.drunkcoder.com/patches/20080611__for_ctooley__3.diff.txt |
| 20:41.42 | Corydon76-dig | ctooley: I did not change the first chunk, no |
| 20:42.18 | Corydon76-dig | although it's probably not necessary, now that we know that it's actually losing the O_NONBLOCK mode |
| 20:42.49 | Corydon76-dig | but we should've been checking the return value all along |
| 20:43.01 | ctooley | is on the phone, will apply and retest soon. |
| 20:43.18 | Corydon76-dig | I don't know how many bugs could have been found, had we simply checked return values. |
| 20:43.27 | Corydon76-dig | s/found/avoided/ |
| 20:46.28 | jsmith | Corydon76-dig: Isn't that typical with C programming though... it's easier to just assume things always work? |
| 20:46.46 | jsmith | Corydon76-dig: Isn't that the whole reason Java went with exceptions? |
| 20:47.13 | putnopvut | I hate checked exceptions in Java. |
| 20:47.41 | jsmith | putnopvut: Mee to |
| 20:47.43 | snuff-home | frowns on try {} catch{} |
| 20:47.47 | jsmith | fires his typist |
| 20:49.54 | ctooley | The nice thing about try and catch is that you can realistically handle things. It's just that NPE's don't provide enough info to track them down. |
| 20:51.02 | seanbright | most people don't use exceptions properly though... |
| 20:51.14 | seanbright | try { foo(); } catch { /* Ignore error */ } |
| 20:51.37 | jsmith | seanbright: Amen, brother! |
| 20:51.43 | putnopvut | The biggest thing that bothers me with Java is having to declare all your functions like int foo() throws blah |
| 20:52.03 | seanbright | if you aren't going to do something meaning with an exception, you shouldn't catch it |
| 20:52.03 | jsmith | string function() throws food |
| 20:52.23 | seanbright | that's why i like java, you either have to handle it in your method, or declare that the handling code must handle it |
| 20:52.34 | seanbright | s/handling/calling/ |
| 20:53.44 | Corydon76-dig | seanbright: the handling code handles the handling for the handled code? |
| 20:53.44 | seanbright | Corydon76-dig: handle this. |
| 20:53.44 | seanbright | points |
| 20:53.44 | seanbright | heh |
| 20:54.59 | seanbright | nothing worse than: public void Foo() throws XException, YException, ZException, ThisException, ThatException |
| 20:55.05 | seanbright | but it is what it is. |
| 20:55.21 | ctooley | Corydon76-dig, running new test job now. |
| 20:55.35 | Corydon76-dig | throws an exception at seanbright |
| 20:55.56 | seanbright | because inevitably the calling code does this: try { this.Foo(); } catch (Exception e) { /* Handle exceptions of all types here... poorly */ } |
| 20:56.15 | seanbright | rethrows the exception |
| 20:56.55 | Corydon76-dig | There's what we need at a convention... a bunch of tennis balls with "Exception" labelled on them... and a pitching machine to lob them into the crowd... |
| 20:57.05 | ctooley | So it's a matter of poorly handling the exception and having information about the exception in the output, or.... poorly checking for return codes and having little to no information. |
| 20:58.18 | Corydon76-dig | Same cause, same solution |
| 20:58.29 | seanbright | i think most exception handling code (throwing or catching) is poorly written |
| 20:58.37 | seanbright | people use exceptions as return codes, when they are not |
| 20:59.32 | ctooley | seanbright, right, and good Java developers handle things at least as well as good C developers, therefore it's not the language's problem, but the base of people using it. |
| 20:59.43 | seanbright | ctooley: absolutely |
| 21:00.40 | seanbright | that's why when we write asterisk 2.0 in c++, we're going to use exceptions sparingly... |
| 21:00.43 | seanbright | ducks |
| 21:01.14 | jsmith | I thought we were rewriting it in Erlang, to finally make it multi-threaded ;-) |
| 21:01.16 | jsmith | ducks |
| 21:01.22 | seanbright | hah |
| 21:01.33 | ctooley | recommends Prolog |
| 21:01.37 | Corydon76-dig | seanbright: you do realize that a rewrite in another language would a) exclude a large number of developers, and b) be unlikely to use C++ |
| 21:01.44 | jsmith | ctooley: Likes(jsmith,prolog); |
| 21:01.50 | seanbright | Corydon76-dig: i realize that yes. |
| 21:02.01 | seanbright | Corydon76-dig: you do realize that i was joking |
| 21:02.03 | jsmith | ctooley: You forget that I met my wife in a Prolog class |
| 21:02.14 | jsmith | ctooley: But multi-threaded Prolog is a pain |
| 21:02.22 | ctooley | jsmith, I didn't forget, I simply never knew |
| 21:02.30 | jsmith | ctooley: Now you know! |
| 21:02.42 | ctooley | --- the rest of the story. |
| 21:02.48 | Corydon76-dig | Clearly, Asterisk 2.0 will be coded in multithreaded COBOL |
| 21:02.58 | seanbright | Corydon76-dig: actually, why would it exclude a large number of developers? |
| 21:03.01 | ctooley | YAY! COBOL! |
| 21:03.07 | Corydon76-dig | seanbright: I don't know C++ |
| 21:03.22 | seanbright | Corydon76-dig: so? |
| 21:03.31 | Corydon76-dig | C is the lowest common denominator of the developers |
| 21:03.41 | seanbright | Corydon76-dig: ah |
| 21:03.44 | seanbright | Corydon76-dig: interesting. |
| 21:03.46 | outtolunc | i thought assembler was <G> |
| 21:03.58 | Corydon76-dig | outtolunc: Most developers don't know assembler, either |
| 21:04.12 | seanbright | only knows machine code |
| 21:04.21 | Corydon76-dig | Besides, assembler is not a single language |
| 21:04.22 | seanbright | i have a good machine code -> c convertor |
| 21:04.28 | bweschke | Without C, We would have only Pasal and Obol |
| 21:04.48 | Corydon76-dig | Obol? |
| 21:04.57 | seanbright | cobol... without the C |
| 21:05.01 | Corydon76-dig | Oh, right |
| 21:05.07 | bweschke | I have a geek t-shirt that says that |
| 21:05.24 | Corydon76-dig | My eyes automatically filled in the C in Pascal |
| 21:05.31 | seanbright | my favorite is 'there are 10 types of people... those that understand binary, and those that don't" |
| 21:06.14 | Corydon76-dig | My favorite is "Pigfucking moronic inbred losers for Bush" |
| 21:06.26 | seanbright | hippy |
| 21:06.51 | Corydon76-dig | and "If I have to find Jesus, does that mean he's hiding?" |
| 21:06.56 | ctooley | Drop the first and last word and the rest of those initials are MILF |
| 21:07.11 | Corydon76-dig | and "Everytime you see a rainbow, God is having gay sex" |
| 21:07.28 | seanbright | we're going to replace an administration that steals our money to fight a war we don't want, to an administration that steals our money to pay for social programs that don't work |
| 21:07.41 | seanbright | i'd much rather us be killing people with my money |
| 21:07.42 | seanbright | heh |
| 21:08.15 | ctooley | seanbright, unfortunately (as we found out with the Boomer generation) wars don't alleviate overpopulation |
| 21:08.29 | seanbright | ctooley: well it makes for better movies |
| 21:08.30 | Corydon76-dig | seanbright: I prefer a good economy with social programs that don't work to a poor economy where the government is engaged in a campaign to kill as many people as it can |
| 21:08.58 | seanbright | Corydon76-dig: ah yes, the magical 'good economy' |
| 21:09.27 | Corydon76-dig | and not just on the battlefield, either. Lack of adequate health care kills people just as effectively as a machine gun |
| 21:09.45 | ctooley | Corydon76-dig, Without giving up the White House every other 8 years Democrats wouldn't look so good when the cycle comes around to a good economy. |
| 21:09.46 | seanbright | i should probably adopt the no sex, no politics, no religion rule for IRC as well... |
| 21:10.07 | seanbright | but i didn't start it |
| 21:10.07 | ctooley | departs now to check on asterisk builds. |
| 21:10.08 | seanbright | heh |
| 21:10.27 | ctooley | [Jun 11 21:05:04] ERROR[15287]: channel.c:1996 __ast_read: Alertpipe on channel SIP/54839-ac62f3e0 lost O_NONBLOCK?!! |
| 21:10.35 | jsmith | "The difference between the Republicans and the Democrats is that one group is borrow-and-spend, and the other is tax-and-spend." |
| 21:10.53 | Corydon76-dig | ctooley: I really wouldn't mind if they were given a chance to screw up the economy... |
| 21:11.02 | Corydon76-dig | and take the fallout |
| 21:11.52 | ctooley | Corydon76-dig, The good news is that it didn't print the following log message, about being "Unable to set alertpipe nonblocking" |
| 21:13.13 | Corydon76-dig | jsmith: I used to assume that the Republicans were good for the economy and the Democrats were good for social justice. But apparently, the Republicans aren't so good for the economy. |
| 21:13.19 | Corydon76-dig | Where does that leave us? |
| 21:13.25 | seanbright | fucked either way |
| 21:13.35 | seanbright | right there ^ |
| 21:13.43 | Corydon76-dig | Need a third party that hasn't been created yet |
| 21:13.44 | jsmith | Corydon76-dig: What does it say when the Democrats are the fiscally conservative ones? |
| 21:13.52 | seanbright | who is the libertarian candidate? |
| 21:13.55 | jsmith | Corydon76-dig: The Radical Moderates! |
| 21:14.45 | jsmith | seanbright: Someone you've never heard of |
| 21:14.47 | Corydon76-dig | I'd vote for a libertarian, if one ever ran for local office. Unfortunately, they only send up longshot candidates for the White House |
| 21:14.59 | seanbright | jsmith: brown? |
| 21:15.16 | seanbright | i would have voted for huckabee... |
| 21:15.19 | seanbright | for *1* reason |
| 21:15.23 | seanbright | FairTax |
| 21:15.28 | Qwell | because he knew Chuck Norris? |
| 21:15.28 | Corydon76-dig | or worse, Republicans who couldn't get the Republican nomination, so they carpetbag the Libertarian ticket |
| 21:16.00 | seanbright | jsmith: bob barr? wtf is that? |
| 21:16.01 | seanbright | heh |
| 21:16.21 | jsmith | rests his case and goes back to work |
| 21:16.26 | seanbright | haha |
| 21:17.36 | seanbright | so what do you guys think about the christian right? |
| 21:17.38 | seanbright | ducks |
| 21:18.05 | ctooley | After 323K calls, I had 3 O_NONBLOCK messages, turned down the cps from 250 to 200 and haven't had another. |
| 21:18.07 | putnopvut | more like the christian "wrong!" amiright? *high fives* |
| 21:18.12 | seanbright | haha |
| 21:18.16 | ctooley | spoke too soon |
| 21:18.31 | ctooley | just had another one. |
| 21:18.34 | jsmith | ctooley: Obviously it's just harder to trigger the bug with 200 |
| 21:18.43 | seanbright | ctooley: try 199 |
| 21:19.10 | ctooley | seanbright, it's increments of 10 cps, but 190 doesn't make a difference. |
| 21:19.21 | seanbright | might have been joking |
| 21:19.21 | Corydon76-dig | ctooley: I'm concerned less about the messages than with the lockups |
| 21:19.28 | seanbright | will shut up and let the real work commence |
| 21:19.49 | Corydon76-dig | Okay, it may lose O_NONBLOCK, but if we can detect that and fix it, no harm, no foul |
| 21:20.31 | ctooley | Corydon76-dig, still very odd |
| 21:21.17 | Corydon76-dig | Agreed |
| 21:21.27 | Corydon76-dig | but I'll do what I have to to make it work |
| 21:22.26 | ctooley | yeah, thanks for that. |
| 21:26.55 | Corydon76-dig | http://www.neilinnes.org/audio/Neil%20Innes%20-%20One%20Of%20Those%20People%201.mp3 |
| 21:27.37 | Corydon76-dig | topical |
| 21:31.36 | ctooley | goes off to burn the song to a disk so it can be applied. |
| 21:35.22 | Corydon76-dig | ctooley: that site has a ton of songs from the same artist |
| 21:36.37 | ctooley | I kind of gathered that it might have more, based on the domain name |
| 21:37.07 | Corydon76-dig | http://www.neilinnes.org/songblog2/songs/ |
| 21:37.22 | Corydon76-dig | and http://www.neilinnes.org/audio/ |
| 21:47.29 | ctooley | Has anyone had a chance to look at 12772? codefreeze-lap? |
| 21:48.11 | seanbright | M12772 |
| 21:48.12 | MuffinMan | [new] [Asterisk] Channels/chan_sip/General 0012772: Trunk version of chan_sip significantly slower than 1.4.19.1 reported by ctooley (Karma: +2.00) http://bugs.digium.com/view.php?id=12772 |
| 21:48.28 | seanbright | M12772#last |
| 21:48.29 | MuffinMan | [12772#0087659] Commented by russell on 06-02-08 at 11:14: If the issue is going to be on the tracker, then the information should be posted here. |
| 21:48.38 | seanbright | not helpful. |
| 21:48.40 | seanbright | heh |
| 21:48.55 | ctooley | seanbright, as I told russellb the other day, didn't have the files anymore, they were emailed to codefreeze-lap. :) |
| 21:49.05 | seanbright | ah |
| 21:49.19 | codefreeze-lap | ctooley: I remember that one well. This bug will have to be solved before 1.6 is released! |
| 21:49.37 | ctooley | codefreeze-lap, yeah, I kind of think so too |
| 21:49.44 | codefreeze-lap | ctooley: Oh, yes, I suppose I could post those, but they were of no real use... |
| 21:49.45 | seanbright | codefreeze-lap: could you attach the files ctooley e-mailed you to the issue? |
| 21:50.06 | codefreeze-lap | seanbright: I *could*, but... |
| 21:50.39 | ctooley | codefreeze-lap, oh, something else I could provide that might be? |
| 21:51.42 | codefreeze-lap | ctooley: I've been working on a diagnostic tool in my spare (midnights, early-early mornings) that I hope will either point out the problem, or acquit one of my suspicions: lock contention |
| 21:52.19 | jsmith | Silly locks... why do they have to behave like codefreeze-lap's sons? Why do they have to be so contentious? |
| 21:52.55 | *** join/#asterisk-dev Cresl1n (n=matt@216.207.245.1) |
| 21:52.56 | *** mode/#asterisk-dev [+o Cresl1n] by ChanServ |
| 21:53.04 | ctooley | jsmith, they just want love and they're all contending for it. |
| 21:53.20 | codefreeze-lap | jsmith: they aren't so contentious, as fooling around. Result is the same: little to no work gets done.... sigh... |
| 21:54.00 | codefreeze-lap | but of the two, I'd rather hear a lot of laughter around the house, than squabbling, anyday. |
| 21:54.22 | jsmith | codefreeze-lap: Good point ;-) |
| 21:54.43 | *** join/#asterisk-dev qdk (n=qdk@195.242.194.41) |
| 21:55.18 | codefreeze-lap | Oh, and BTW, SCHOOL IS OUT FOR THE SUMMER |
| 21:55.38 | putnopvut | Rock on! |
| 21:59.54 | lmadsen | the school's been blown to peices |
| 22:14.42 | ctooley | well, it crossed the million call mark and is still trucking... the job's set up to run 10 million calls, I'll have the resutls of it tomorrow. |
| 22:15.15 | lmadsen | w00t! |
| 22:16.34 | putnopvut | ctooley: good news! Corydon76-dig, great job with the patch! |
| 22:17.26 | ctooley | putnopvut, I ran a million and a half through it earlier before it crashed, I'm not certain we're out of the woods yet, but it is certainly looking more promising. |
| 22:17.42 | putnopvut | ctooley: yeah, perhaps I spoke too soon. |
| 22:17.43 | *** join/#asterisk-dev denon (n=denon@tooth.decay.org) |
| 22:19.05 | ctooley | But, 366K calls since the last "O_NONBLOCK" error, so, we'll see how it goes at the 2MM call mark |
| 22:27.47 | ctooley | This is definitely a high cps problem. We have a job running on another server that just passed 2 million calls over a 24 hour period and it hasn't experienced the issue. |
| 22:29.39 | *** join/#asterisk-dev jets (n=brian@pdpc/supporter/active/jets) |
| 22:34.26 | ctooley | wonders if the fix that Corydon76-dig gave him will resolve the issues with chan_iax2 deadlocks |
| 22:36.53 | ctooley | goes off to make dinner. |
| 22:41.30 | *** join/#asterisk-dev jameswf-home (n=james@dsl093-157-131.phx1.dsl.speakeasy.net) |
| 23:03.53 | jpeeler | why would you ever want to allow parking to an extension that is already taken in the dialplan? |
| 23:28.37 | ctooley | Corydon76-dig, Well, it got to 1.7M calls before it blew up. But that wasn't Asterisk's fault, server ran out of disk space |
| 23:29.42 | ctooley | I'm starting a 15M call job. That will last until tomorrow some time, we'll see what it's doing in the morning.t |
| 23:34.57 | *** join/#asterisk-dev jameswf (n=james@dsl093-157-131.phx1.dsl.speakeasy.net) |