| 00:00.06 | *** join/#asterisk-dev nix8n82 (~AndChat27@67-130-74-235.dia.static.qwest.net) |
| 00:58.52 | *** join/#asterisk-dev michael_work_ (~michael_w@bzq-82-168-31-134.red.bezeqint.net) |
| 02:39.22 | *** join/#asterisk-dev burper (~burper@modemcable082.140-131-66.mc.videotron.ca) |
| 02:39.23 | *** part/#asterisk-dev burper (~burper@modemcable082.140-131-66.mc.videotron.ca) |
| 04:04.19 | *** join/#asterisk-dev spditner (~simon@76-10-137-81.dsl.teksavvy.com) |
| 04:43.42 | *** join/#asterisk-dev Corydon76 (~quassel@c-69-137-93-158.hsd1.tn.comcast.net) |
| 04:43.42 | *** mode/#asterisk-dev [+o Corydon76] by ChanServ |
| 04:59.50 | *** part/#asterisk-dev mjordan (mjordan@nat/digium/x-ddxioqsxykeinhty) |
| 05:23.40 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 05:23.40 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 06:44.47 | *** join/#asterisk-dev oej (~Adium@h87-96-134-129.dynamic.se.alltele.net) |
| 06:44.48 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 07:05.48 | *** join/#asterisk-dev bulkorok (~Benjamin@85.183.61.47) |
| 08:21.05 | *** join/#asterisk-dev tzafrir (~tzafrir@local.xorcom.com) |
| 08:21.05 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 09:58.09 | *** join/#asterisk-dev hehol (~hehol@2001:1438:1009:200:4cb0:63dd:fda3:f086) |
| 09:59.54 | *** join/#asterisk-dev michael_work_ (~michael_w@bzq-82-168-31-134.red.bezeqint.net) |
| 10:42.59 | *** join/#asterisk-dev hehol (~hehol@2001:1438:1009:200:d4b0:ea1c:5375:f316) |
| 11:31.28 | *** join/#asterisk-dev michael_work (~michael_w@bzq-82-168-31-134.red.bezeqint.net) |
| 11:32.20 | *** join/#asterisk-dev michael_work_ (~michael_w@192.117.186.204) |
| 11:37.40 | *** join/#asterisk-dev michael_work (~michael_w@192.117.186.204) |
| 11:51.58 | *** join/#asterisk-dev michael_work_ (~michael_w@192.117.186.204) |
| 11:54.44 | *** join/#asterisk-dev oej (~Adium@h87-96-134-129.dynamic.se.alltele.net) |
| 11:54.44 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 12:34.48 | *** join/#asterisk-dev michael_work_ (~michael_w@bzq-82-168-31-134.red.bezeqint.net) |
| 12:44.07 | *** join/#asterisk-dev Marquis42 (~mrbmw@d14-69-246-193.try.wideopenwest.com) |
| 13:10.57 | *** join/#asterisk-dev [TK]D-Fender (~chatzilla@216-191-106-163.dedicated.allstream.net) |
| 13:31.41 | *** join/#asterisk-dev oej (~Adium@212.181.100.7) |
| 13:31.41 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 14:13.53 | *** join/#asterisk-dev oej (~Adium@212.181.100.7) |
| 14:13.54 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 14:22.20 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-pmzntvytadgubltz) |
| 14:33.05 | *** join/#asterisk-dev newtonr (newtonr@nat/digium/x-slmwydxluelfftif) |
| 14:33.05 | *** mode/#asterisk-dev [+o newtonr] by ChanServ |
| 14:36.31 | *** join/#asterisk-dev sruffell (sruffell@asterisk/the-kernel-guy/sruffell) |
| 14:36.31 | *** mode/#asterisk-dev [+o sruffell] by ChanServ |
| 14:37.17 | *** join/#asterisk-dev putnopvut (~putnopvut@asterisk/master-of-queues/mmichelson) |
| 14:37.17 | *** mode/#asterisk-dev [+o putnopvut] by ChanServ |
| 14:43.15 | *** join/#asterisk-dev oej (~Adium@212.181.100.7) |
| 14:43.15 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 15:05.57 | *** join/#asterisk-dev goseeped (~goseeped@190.203.3.227) |
| 15:14.50 | goseeped | Hi guys , i want to know what changes has current code asterisk 13.1 "rtp_engine" , "format" ... that webrtc web browsers doesn't understand the VP8 "passtrough" format sended by asterisk. unlike asterisk 11 (patch version) AFAIK since 12 is official supported. |
| 15:20.05 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-rzuzfxjpewxjdkvc) |
| 15:20.05 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 15:22.18 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-kuxgaychdttpckzp) |
| 15:31.13 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-wqmplzqjbldxcovt) |
| 15:43.03 | *** part/#asterisk-dev bulkorok (~Benjamin@85.183.61.47) |
| 15:45.12 | mjordan | file: so here's a thought on ASTERISK-24692 |
| 15:45.31 | mjordan | we could only generate a 181 (since PJSIP is going to add an SDP to it) if the channel supports in band progress |
| 15:45.48 | mjordan | and, in that same callback, immediately queue an AST_CONTROL_PROGRESS frame onto the channel |
| 15:46.02 | mjordan | that will cause a 183 to immediately follow, which should trip the normal 'progress' logic in the core |
| 15:47.16 | skrusty | would that deal with the ari issue i had where we had no event on progress? |
| 15:47.33 | file | mjordan, ummmm |
| 15:49.55 | file | mjordan, I don't know if it'll do what you want... |
| 15:50.00 | file | skrusty, nope? |
| 15:50.58 | skrusty | sorry, just re-read and realised |
| 15:51.37 | file | mjordan, er if inband progress is turned on this should just work regardless |
| 15:57.05 | mjordan | hm. |
| 15:58.02 | mjordan | maybe we only indicate 181 if progress is enabled then. |
| 16:09.01 | *** join/#asterisk-dev gtjoseph (~gtj0@unaffiliated/gtj) |
| 16:20.27 | *** join/#asterisk-dev sgriepentrog (sgriepentr@nat/digium/x-ngbhikbpnoxgtmix) |
| 16:32.26 | *** join/#asterisk-dev Kobaz (~kobaz@its.kobaz.net) |
| 16:32.42 | Kobaz | sooo... storesipcause WARNING: enabling this option carries a significant performance burden. |
| 16:32.45 | Kobaz | how significant? |
| 16:33.46 | Kobaz | and what does it affect? |
| 16:33.52 | Kobaz | it shouldn't affect like audio i would imagine |
| 16:34.24 | mjordan | Kobaz: profile and find out. But yes, it's expensive. |
| 16:34.41 | Kobaz | what does it have to do? i kind of remember discussing this like a year ago |
| 16:34.46 | Kobaz | extra locks i know |
| 16:35.10 | opticron | remembers something about locking and walking the channel list |
| 16:36.24 | mjordan | Kobaz: I don't know. I'd have to go dig into chan_sip myself to find out, it's been a long time since I've looked at that code |
| 16:36.30 | mjordan | I do remember that it was rather hideous. |
| 16:37.03 | opticron | more recent versions are less hideous since it was rewritten for more generic usage |
| 16:38.05 | mjordan | right, but if you're looking at storesipcause, you probably aren't running 11+ |
| 16:38.25 | Kobaz | nope |
| 16:38.27 | Kobaz | 1.8 very much |
| 16:38.34 | Kobaz | rather stuck on it until this spring/summer |
| 16:38.54 | Kobaz | my last big project from 2014 will be past the babysitting phase and i can get back to dev work |
| 16:45.40 | *** join/#asterisk-dev w9sh (~sph@50-79-224-193-static.hfc.comcastbusiness.net) |
| 16:45.48 | Kobaz | so exciting |
| 16:45.53 | Kobaz | gonna port all my stuff to 13 |
| 16:46.44 | *** part/#asterisk-dev w9sh (~sph@50-79-224-193-static.hfc.comcastbusiness.net) |
| 16:52.24 | Kobaz | i'm really surprised, wow |
| 16:52.27 | Kobaz | i just checked this one system |
| 16:52.36 | Kobaz | up 7 weeks, using 112mB res memory |
| 16:52.47 | Kobaz | it really depends on the workload it's doing |
| 16:53.07 | Kobaz | i have other systems that need a nightly restart, since after 4 days it's using 4gB at idle with no channels |
| 17:54.19 | pabelanger | Kobaz, IIRC, it was pretty significant. Enough for a patch to disable it. |
| 18:05.06 | *** join/#asterisk-dev Jamuel (~Adium@67.180.156.186) |
| 18:08.14 | *** join/#asterisk-dev benasse (~benasse@nowhere.cicogna.fr) |
| 18:15.24 | *** join/#asterisk-dev oej (~Adium@h87-96-134-129.dynamic.se.alltele.net) |
| 18:15.24 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 18:33.19 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 18:33.19 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 19:11.08 | *** join/#asterisk-dev leedm777 (leedm777@nat/digium/x-etwewhxtikvsordw) |
| 19:22.05 | *** join/#asterisk-dev oej (~Adium@h87-96-134-129.dynamic.se.alltele.net) |
| 19:22.05 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 19:22.38 | *** join/#asterisk-dev tzafrir (~tzafrir@bzq-218-28-58.cablep.bezeqint.net) |
| 19:22.38 | *** mode/#asterisk-dev [+o tzafrir] by ChanServ |
| 19:32.04 | *** join/#asterisk-dev oej (~Adium@h87-96-134-129.dynamic.se.alltele.net) |
| 19:32.04 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 19:34.55 | putnopvut | has anyone in here succesfully set up Jitsi with Asterisk (specifically using res_pjsip) using TLS? I'm attempting to do this myself, and like every time in the past I've tried to set up TLS, it's not working, and just like in the past, I'm not getting any feedback from Asterisk or the client as to where the failure is. |
| 19:35.17 | putnopvut | So unfortunately, I can't give any further diagnosis than "it's not working" |
| 19:35.43 | putnopvut | But if someone has got it working in the past, I'd be interested to know what you did, specfically on the Jitsi side, to get things running smoothly. |
| 19:36.58 | sgriepentrog | I did it once, but uninstalled it since. The pjsip configs I used are in github.com/stgnet/asterisk_configs |
| 19:38.07 | rmudgett | We probably should change the testsuite cdr tests to not use the extended support cdr-csv backend but the cdr-custom backend. |
| 19:49.19 | putnopvut | sgriepentrog: thanks. Your TLS settings look identical to mine (of course with paths and file names altered). I think it's strictly the jitsi side where getting TLS to work properly is being dumb. |
| 19:50.03 | sgriepentrog | I used the windows version, selected the matching port # to TLS and set the transport for that too. Otherwise nothing special on jitsi end. |
| 19:50.23 | putnopvut | Hm... |
| 19:50.52 | sgriepentrog | I found it easier to get things working when each transport was on it's own separate port. |
| 19:53.07 | sgriepentrog | Also my server was on a public IP if that makes any difference. |
| 19:53.37 | putnopvut | Yeah, this may have something to do with my server running on a VM on the same box where JITSI is. |
| 19:54.33 | putnopvut | either way, I'm *this* close to giving up. |
| 19:57.21 | malcolmd | i've got tis on 5061 and i can't, either, get jtsi to work with tls. |
| 19:58.03 | sgriepentrog | I'll set it up again on my end and retest. I had it working with a yealink also. |
| 20:03.52 | *** join/#asterisk-dev oej (~Adium@87.96.134.129) |
| 20:03.54 | *** mode/#asterisk-dev [+o oej] by ChanServ |
| 20:10.35 | *** join/#asterisk-dev mjordan (~mjordan@75.76.55.191) |
| 20:10.35 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 20:17.33 | sgriepentrog | putnopvut: connected jitsi (windows) to asterisk (pub ip) |
| 20:17.47 | putnopvut | sgriepentrog: using TLS? |
| 20:17.52 | sgriepentrog | yes |
| 20:18.13 | putnopvut | nice. I'm starting to think it's 100% an issue with my internal network/running on a VM |
| 20:18.55 | sgriepentrog | I set the registrar to my server, port 5062, then turned off proxy auto and set the proxy to the same server, same port, and selected TLS transport. |
| 20:19.05 | sgriepentrog | Nothing else changed, works. |
| 20:19.15 | putnopvut | Sounds exactly like what I've been doing :-/ |
| 20:19.35 | sgriepentrog | Are you receiving the RESGISTER on asterisk? |
| 20:19.39 | putnopvut | I'm going to try again at the office tomorrow. There I have two separate physical boxes that I can try to get communicating with each other. |
| 20:19.44 | putnopvut | sgriepentrog: no I'm not |
| 20:20.23 | sgriepentrog | Yea, sounds like a networking issue, and if you don't have a second machine to try connecting from it'd be hard to prove. |
| 20:37.10 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-rnnpogdsocxxnfsd) |
| 20:37.10 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 20:54.16 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-fzzmpecehmxhmuob) |
| 20:54.16 | *** mode/#asterisk-dev [+o mjordan] by ChanServ |
| 20:54.50 | mjordan | okay... |
| 20:54.52 | mjordan | so, what'd we break |
| 20:54.53 | mjordan | https://bamboo.asterisk.org/bamboo/browse/AST-ATSEM-331 |
| 20:55.05 | mjordan | kharwell: could this be related to the res_pjsip shutdown patch? |
| 20:55.14 | file | that's what I'm thinking |
| 20:55.52 | mjordan | nice, 569 tests failed |
| 20:55.52 | mjordan | https://bamboo.asterisk.org/bamboo/browse/AST-ATSF1-C664TE-148 |
| 20:56.21 | kharwell | mjordan: perhaps. I'll look into it |
| 21:00.14 | file | the message router subscription usage is... interesting |
| 21:10.47 | mjordan | I'm not sure how the res_pjsip unload blew up everyone |
| 21:11.38 | mjordan | although, in these crashes, I'm also seeing lots of stuff dumping out elsewhere as well |
| 21:11.59 | mjordan | i.e., here is chan_multicast_rtp nuking itself - https://bamboo.asterisk.org/bamboo/artifact/AST-ATSF1/C664TE/build-148/Test-Suite-Logs/channels/iax2/basic-call/backtrace_8012.txt |
| 21:28.24 | putnopvut | mjordan: it's possible that tests that autoload modules load res_pjsip, thus making all of them blow up on shutdown. kharwell linked me to a test run where I was seeing media reindexing causing crashses like rmudgett had been seeing. |
| 21:28.33 | putnopvut | I'm trying to locally reproduce this to see what I can find out. |
| 21:29.39 | mjordan | kk |
| 21:32.46 | rmudgett | mjordan: The assert for memory corruption happens whenever it is detected. It likely isn't the fault of the thread that detected the corruption. |
| 21:33.41 | rmudgett | The corrupted memory block is pushed out of the freed pool by the thread that detected the corruption. |
| 21:38.06 | putnopvut | rmudgett: so you're saying that the media reindex doesn't necessarily have anything to do with the crash, even though the crash is happening in that thread? |
| 21:39.08 | mjordan | rmudgett: good point. |
| 21:39.28 | mjordan | putnopvut: I'd guess the indexer is hitting this a lot since it allocates a lot of memory on start up |
| 21:39.30 | rmudgett | putnopvut: yes. A freed block is put into a freed pool. When the pool is full, newly freed blocks push out an old freed block. That old freed block is then checked for corruption. |
| 21:39.43 | mjordan | sounds like valgrind time |
| 21:40.40 | rmudgett | Valgrind does something similar. The corrupted block has a note for who/where the block was initially allocated. That is reported in the mmlog message. |
| 21:41.24 | mjordan | hm. I didn't know we had who initially allocated it in the mmlog |
| 21:41.42 | rmudgett | file, func, line number is recorded |
| 21:41.44 | putnopvut | yes it's quite handy |
| 21:42.03 | mjordan | heh |
| 21:42.20 | mjordan | looks like DO_CRASH may happen fast enough to prevent the mmlog from getting anything. Either that, or we still aren't archiving it |
| 21:42.50 | rmudgett | The mmlog message does get into the run_1/messages.txt file |
| 21:44.23 | mjordan | ah. |
| 21:44.27 | rmudgett | All of the failed tests I was seeing yesterday were showing chan_sip's peers container as corrupted. |
| 21:44.35 | mjordan | yeah, these have some Stasis. |
| 21:52.54 | putnopvut | FYI, I can reproduce the memory corruption in the testsuite when running the realtime "single" test |
| 21:59.49 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 21:59.55 | *** part/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 21:59.58 | *** join/#asterisk-dev [TK]D-Fender (~joe@64.235.216.2) |
| 22:06.27 | putnopvut | I have a possible hypothesis on stasis corruption stuffz, but I would have suspected it would have been happening for a while though. Does stasis_unsubscribe_and_join() (and by extension stasis_message_router_unsubscribe_and_join()) make the assumption of one thread per subscriber? There was a change recently to be able to apply a threadpool to Stasis subscriptions. |
| 22:07.13 | putnopvut | So if those the *_and_join() variants wait for only one thread to be finished with their subscription, but other threads are still processing things, those other threads might corrupt data? |
| 22:07.21 | putnopvut | s/those// |
| 22:12.08 | X-Rob | 2015-01-16 08:11:06] WARNING[20614][C-00000010] channel.c: Unable to find a codec translation path from (slin) to (slin) |
| 22:12.35 | putnopvut | wtf |
| 22:12.47 | WIMPy | That makes sense somehow. :-) |
| 22:13.10 | file | different rates |
| 22:15.05 | mjordan | stupid stringification |
| 22:15.39 | mjordan | putnopvut: not sure if it makes that assumption or not, but if so, then we should have been crashing like crazy |
| 22:16.05 | putnopvut | yeah |
| 22:16.58 | putnopvut | I think I'm going to configure a failing test to issue the "memory backtrace on" command so I can get more details about the corruption. I think there's a cli.conf setting to run a CLI command on startup... |
| 22:17.21 | file | I'm confused over the lifetime of the subscription in the message router... documentation says when you unsubscribe that drops the ref, but then it's accessed in the destructor |
| 22:19.42 | putnopvut | I'm not seeing the access in the destructor. Is it in ROUTE_TABLE_ELEM_CLEANUP() or something? |
| 22:20.24 | putnopvut | Oh I looked at the wrong destructor OOPS |
| 22:20.38 | putnopvut | router_dtor checks if the subscription is done and such. |
| 22:21.19 | file | but how are you guaranteed for it to still be valid? and I think Kevin made it so router->subscription is actually NULLed |
| 22:21.29 | mjordan | thars dragons in there |
| 22:21.36 | mjordan | and I hate what we did there as well. |
| 22:21.54 | mjordan | There's some place where it checks to see if the reference is 2, then nukes itself, as for some crazy edge case it has to hold a reference to itself |
| 22:22.50 | putnopvut | btw, reverting the change to NULL the router's ref to the subscription on unsubscribe causes the failing tests to no longer report memory corruption on my system. |
| 22:23.29 | file | still feels like something is... wrong |
| 22:27.19 | *** join/#asterisk-dev leedm777 (~leedm777@50-81-28-152.client.mchsi.com) |
| 22:28.46 | *** join/#asterisk-dev spditner (~simon@206-248-160-115.dsl.teksavvy.com) |
| 22:32.12 | mjordan | eek |
| 22:32.25 | kharwell | well if I don't null it out but try to check the reference count then it crashes. prob safe to assume then no more refs around and it got freed anyhow |
| 22:32.40 | mjordan | uh |
| 22:32.47 | putnopvut | So here's something weird. |
| 22:32.50 | mjordan | is perturbed by this |
| 22:34.03 | putnopvut | kharwell told me that stasis_message_router.c was changed to what it is because otherwise, trying to unload res_pjsip.so would crash. And he's right. |
| 22:34.06 | putnopvut | BUT! |
| 22:34.25 | putnopvut | I can't find any references to stasis_message_routers in res_pjsip.c or in res_pjsip/*.c |
| 22:34.40 | putnopvut | soooo why does that have any affect at all? |
| 22:34.48 | putnopvut | or "effect" event. |
| 22:34.49 | putnopvut | even |
| 22:34.51 | file | it indirectly uses it? |
| 22:34.51 | putnopvut | I can't type |
| 22:35.00 | kharwell | putnopvut: it keeps a ref in persistent endpoints |
| 22:35.00 | putnopvut | I didn't realize anything did that. |
| 22:35.18 | kharwell | that's what is using it I mean |
| 22:35.47 | kharwell | k |
| 22:36.16 | kharwell | dumb emacs shortcuts and wrong window! |
| 22:38.29 | putnopvut | s/ s// |
| 22:38.41 | putnopvut | oh I can't do that to other people's sentences. |
| 22:38.42 | putnopvut | lame |
| 23:22.35 | *** join/#asterisk-dev infobot (ibot@rikers.org) |
| 23:22.35 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Asterisk 13.0.0 now released. Do the release dance! -=- Follow on Twitter at @AsteriskDev |
| 23:23.26 | putnopvut | I guess the race condition you're talking about could happen if a message router is unsubscribed and then immediately destroyed. The subscription hasn't had time to be NULLed out yet, so you attempt to lock what is at freed memory. Problem is, I'd expect that to have happened before the change to stasis_message_router.c |
| 23:27.53 | kharwell | putnopvut: I would think so too |
| 23:28.37 | *** join/#asterisk-dev moy (sid47040@gateway/web/irccloud.com/x-xozifzvhdabbmdxw) |
| 23:28.53 | *** join/#asterisk-dev newtonr (newtonr@nat/digium/x-fudyslffphuknzxr) |
| 23:28.53 | *** mode/#asterisk-dev [+o newtonr] by barjavel.freenode.net |
| 23:28.55 | kharwell | putnopvut: putting in some more logging shows just that scenario. a message router is being destroyed and unsubscribed at the same time from two different threads |
| 23:29.05 | *** join/#asterisk-dev GameGamer43 (sid5533@gateway/web/irccloud.com/x-czmjynahxgchktwt) |
| 23:29.34 | *** join/#asterisk-dev sgriepentrog (sgriepentr@nat/digium/x-uiczggpbhwwlypth) |
| 23:29.37 | putnopvut | ew |
| 23:30.20 | putnopvut | Any idea which router? |
| 23:30.41 | *** join/#asterisk-dev jrose_atDigium (jrose_atDi@nat/digium/x-slrbpufnfbsibhwh) |
| 23:30.57 | *** join/#asterisk-dev mjordan (mjordan@nat/digium/x-fquetveypjuxoedx) |
| 23:31.32 | kharwell | no not sure which one |
| 23:32.28 | *** join/#asterisk-dev snuff-work (~blatus@203.87.61.22) |
| 23:32.29 | *** mode/#asterisk-dev [+o snuff-work] by ChanServ |
| 23:37.27 | *** join/#asterisk-dev Jamuel (~Adium@50.141.65.14) |
| 23:42.01 | X-Rob | tap tap |
| 23:42.59 | WIMPy | [swish] |
| 23:47.21 | *** part/#asterisk-dev mjordan (mjordan@nat/digium/x-fquetveypjuxoedx) |
| 23:51.36 | putnopvut | I'm going hardcore with this. Turning on REF_DEBUG |