| 00:03.04 | *** part/#asterisk-dev kharwell (kharwell@nat/digium/x-qngzludauprexqgt) |
| 01:24.42 | *** join/#asterisk-dev infobot (ibot@208.53.50.136) |
| 01:24.42 | *** topic/#asterisk-dev is Asterisk Development Discussion -=- http://www.asterisk.org/developers -=- Tier 2 and 3.14159265 support is in #asterisk -=- Check out our blog! blogs.asterisk.org -=- Follow on Twitter at @AsteriskDev |
| 03:07.29 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@24-181-237-90.dhcp.oxfr.ma.charter.com) |
| 03:07.30 | *** mode/#asterisk-dev [+o coreyfarrell] by ChanServ |
| 06:36.39 | *** join/#asterisk-dev tsearle (~tsearle@188.44.81.78) |
| 06:36.40 | *** mode/#asterisk-dev [+o tsearle] by ChanServ |
| 07:22.33 | *** join/#asterisk-dev pchero_work (~pchero@87.213.240.121) |
| 07:31.17 | *** join/#asterisk-dev jkroon (~jkroon@165.16.203.61) |
| 07:33.09 | *** join/#asterisk-dev pchero_work (~pchero@87.213.240.121) |
| 08:14.57 | *** join/#asterisk-dev CELYA_ (~Thunderbi@46.218.88.74) |
| 08:24.00 | *** join/#asterisk-dev pchero_work (~pchero@87.213.240.121) |
| 08:28.29 | *** join/#asterisk-dev hehol (~hehol@gatekeeper.loca.net) |
| 09:27.26 | tsearle | Still waiting to schedule task |
| 09:27.26 | tsearle | All nodes of label âswdev-dockerâ are offline |
| 09:27.46 | tsearle | think you jenkins has issues |
| 12:35.14 | *** join/#asterisk-dev bford (uid283514@gateway/web/irccloud.com/x-szbywzhaqobwhxlu) |
| 12:35.14 | *** mode/#asterisk-dev [+o bford] by ChanServ |
| 12:35.26 | tsearle | waves at bford |
| 12:35.40 | bford | waves back |
| 12:39.33 | tsearle | everybody had a good centennial armistice celebration? |
| 12:40.08 | tsearle | given that docker is dead and it's been dead quiet here, I'm guessing yes :-D |
| 12:41.45 | coreyfarrell | tsearle: I did. BTW for docker I think gtjoseph is working on upgrading some hardware, I'm guessing he wasn't able to finish yesterday. |
| 12:41.55 | file | it had to rebalance |
| 12:42.06 | coreyfarrell | you mean RAID? |
| 12:42.13 | file | glusterfs |
| 12:42.29 | tsearle | ah, so not balancing as a sobriety test then ;-) |
| 12:42.37 | file | coreyfarrell: otherwise on segmenting - seem good? |
| 12:44.41 | coreyfarrell | file: before I say definitely yes let me give another look at stasis_channels.c. I really only reviewed the main snapshot allocator. I'm sure the sub-structure allocators are probably fine but I'll say for sure in a few minutes. |
| 12:44.54 | file | nods |
| 12:49.24 | *** join/#asterisk-dev pc-m (~pcm@modemcable094.94-70-69.static.videotron.ca) |
| 12:54.45 | coreyfarrell | file: the allocators seem fine but it looks like you're missing some invalidations in channel_internal_api.c. so I'm working on posting some additional findings. |
| 13:00.54 | file | oh fudge I didn't have tests enabled either |
| 13:01.12 | gtjoseph | for what? |
| 13:01.20 | file | gtjoseph: my snapshot changes |
| 13:01.28 | file | into the abyss I go |
| 13:16.36 | gtjoseph | coreyfarrell: was your off-nominal crash in jansson patch accepted upstream? |
| 13:16.51 | coreyfarrell | gtjoseph: yes |
| 13:16.59 | gtjoseph | cool. +2ing |
| 13:17.01 | coreyfarrell | do you see an issue with it? |
| 13:17.17 | gtjoseph | nope. |
| 13:17.24 | gtjoseph | just checking |
| 13:17.51 | coreyfarrell | the upstream patch includes expanded testing, I removed that from the bundled patch because it didn't merge plus we don't 'make check' the bundled jansson anyways. |
| 13:18.03 | gtjoseph | yep i saw that |
| 13:18.30 | gtjoseph | well, i saw your comment to that effect |
| 13:18.33 | gtjoseph | :) |
| 13:22.09 | gtjoseph | coreyfarrell: thanks for the el_end fix. that always bugged the hell out of me. |
| 13:24.30 | coreyfarrell | gtjoseph: same. before investigating for the reporter I never experienced the rasterisk piping issue but I've had the issue with 'asterisk -c' being stopped from rasterisk. |
| 13:24.52 | gtjoseph | yep, the latter for me too |
| 13:34.38 | coreyfarrell | gtjoseph: got a segfault at https://jenkins2.asterisk.org/job/asterisk-unittests/job/master/546/ - don't think it's related to my change. |
| 13:34.52 | gtjoseph | yeah i'm looking at that now |
| 13:35.42 | file | that's... different |
| 13:36.08 | gtjoseph | yeah sorcery thrash test |
| 13:44.54 | gtjoseph | coreyfarrell: i've run it a few times without issue so let's see what the gates say. |
| 13:45.49 | coreyfarrell | since it happened in the thrash test I'm guessing some really rare race happened. |
| 14:22.31 | coreyfarrell | gtjoseph: it was test_res_pjsip_scheduler.c scheduler_policy that caused the assertion due to write-after-free (found in mmlog). I'm testing a patch that will cause the test to fail instead of crashing. |
| 14:22.51 | gtjoseph | wow |
| 14:23.04 | gtjoseph | good finid |
| 14:23.09 | gtjoseph | find |
| 14:33.58 | gtjoseph | coreyfarrell: the exclude of lock.h with #define _ASTERISK_LOCK_H causes a big mess. gonna have ot leave it for now. |
| 14:37.13 | *** join/#asterisk-dev dandann00dle (~dandann00@2601:840:8401:d7ee:cd87:f9f3:b237:db82) |
| 14:37.16 | coreyfarrell | gtjoseph: you mean compile errors? FWIW I did test compile before I posted the suggestion |
| 14:37.23 | gtjoseph | yes |
| 14:37.42 | gtjoseph | I think it's from strings.h |
| 14:37.49 | gtjoseph | which i needed for S_OR |
| 14:38.00 | gtjoseph | but i'm testing with a local definition of S_OR |
| 14:41.12 | gtjoseph | nope. S_OR requires ast_strlen_zero which is in utils.h which includes things that need the ast_lock stuff |
| 14:42.10 | coreyfarrell | weird. I don't know how my compile succeeded without S_OR. |
| 14:45.42 | coreyfarrell | oh I tested that without BETTER_BACKTRACES enabled. could put '#define S_OR(a, b) (a && a[0] != '\0') ? a : b' into backtrace.c? |
| 14:46.16 | gtjoseph | will try |
| 14:46.49 | coreyfarrell | that would basically isolate backtrace.c from all the asterisk debug trickery. would need to directly include <pthread.h>. |
| 14:47.03 | gtjoseph | yep |
| 14:47.53 | gtjoseph | ASTERISK_FILE_VERSION is dead in 13 now correct? |
| 14:50.30 | coreyfarrell | for all practical purposes yes. 'core show file version' lists filenames but they're all listed as being the asterisk version. that CLI command is removed from 16. |
| 15:00.25 | gtjoseph | coreyfarrell: how's this... https://pastebin.com/6YwTb78a |
| 15:04.02 | coreyfarrell | looks good. using '[inlined]' is probably clearer than '[%p] i' from the previous revision. |
| 15:06.26 | gtjoseph | yep |
| 15:19.03 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-vuvjbuhclcpcoblu) |
| 15:19.03 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |
| 15:21.50 | *** join/#asterisk-dev CELYA_1 (~Thunderbi@46.218.88.74) |
| 15:24.41 | gtjoseph | is not pleased with jenkins |
| 15:25.40 | file | coreyfarrell: oy yeah |
| 15:32.32 | file | I'm going to see how widespread it really is... |
| 15:32.49 | gtjoseph | what really is? |
| 15:33.02 | file | gtjoseph: directly manipulating certain callerid/connected/dialed information |
| 15:33.13 | gtjoseph | oh |
| 15:34.00 | file | I think the answer is, "fairly" |
| 15:34.29 | gtjoseph | are you talking about in the codebase? |
| 15:35.33 | file | yes |
| 15:35.35 | *** join/#asterisk-dev rmudgett (rmudgett@nat/digium/x-oiwxfockhltregvp) |
| 15:35.35 | *** mode/#asterisk-dev [+o rmudgett] by ChanServ |
| 15:36.06 | file | that information is in segments in my patch, so every where that touches it has to invalidate... and barely anywhere is using the functions to do that, they directly get the structure and do their thing |
| 15:36.20 | file | which makes me sad |
| 15:36.23 | gtjoseph | ah, now i understand |
| 15:37.21 | file | I'm kind of considering making those segments always get regenerated for now, and creating a second issue to further const the channel API and force things through the proper API so more can be done (flags too for example) |
| 15:38.19 | file | my stasis mucking is already rather large, letting things settle for a bit would be nice |
| 15:39.13 | file | thoughts? |
| 15:42.53 | kharwell | sounds reasonable to me |
| 15:46.00 | coreyfarrell | file: +1 on always regenerate caller id segment, separate patch to make it const |
| 17:50.28 | *** join/#asterisk-dev pc-m (~pcm@cable-66-129-132-216.b2b2c.ca) |
| 17:55.52 | *** join/#asterisk-dev CELYA_ (~Thunderbi@2a01cb0583d31d003142f30dfb8f85a6.ipv6.abo.wanadoo.fr) |
| 17:57.52 | *** join/#asterisk-dev cresl1n (uid299068@asterisk/libpri-and-libss7-expert/Cresl1n) |
| 17:57.52 | *** mode/#asterisk-dev [+o cresl1n] by ChanServ |
| 18:55.04 | *** join/#asterisk-dev jkroon (~jkroon@165.16.204.34) |
| 18:56.34 | coreyfarrell | I think https://gerrit.asterisk.org/10479 is backward compatible. anyone have an opinion about cherry-picking it to 13/16? |
| 18:57.58 | kharwell | I think I'd be fine with that |
| 19:20.09 | gtjoseph | I'd be ok with it going back to 16 but am concerned with it in 13 "just in case" |
| 19:23.33 | *** join/#asterisk-dev CELYA_1 (~Thunderbi@lfbn-1-5361-63.w90-105.abo.wanadoo.fr) |
| 20:06.56 | kharwell | Looks like the new hard drives did not fix the ast_restart test. Failed again and I got a similar backtrace where it looks as though sqlite3 is blocking on fsync |
| 20:07.27 | kharwell | Also I noted this in the test's log file reported by sipp (I don't recall seeing it before): "Overload warning: the minor watchdog timer 500ms has been tripped" Also saw one for major |
| 20:11.27 | kharwell | I'm not sure why this seems to be happening only for the ast_restart test. Or if the timing of that particular test is just right to trigger the condition with the container. |
| 20:12.14 | kharwell | seemingly it only also occurs when other tests/containers are actively executing other tests or testsuite items |
| 20:13.12 | kharwell | Anyone have any thoughts on what to do or how to move forward with this? Or other things to check? If it is a problem with running the test in a container only at this point I'm not sure we have a good way to work around that |
| 20:14.36 | gtjoseph | so we really believe that the problem is the fsync to astdb? |
| 20:18.16 | file | that would indeed block stuff |
| 20:20.13 | *** join/#asterisk-dev dandann00dle (~dandann00@2601:840:8401:d7ee:cd87:f9f3:b237:db82) |
| 20:20.45 | kharwell | so far I've collected at least 2-3 backtraces when the test fails. Each one showed a thread waiting on fysnc and then other threads waiting on that thread or threads waiting on that thread |
| 20:21.33 | gtjoseph | hmmm |
| 20:21.38 | kharwell | Also the initial DEBUG_THREADS lock output showed similar |
| 20:22.13 | gtjoseph | when you run it, are you clearing out the old astdb before running again? |
| 20:22.48 | kharwell | doubtful? The testsuite would have to do that and I am not aware if it does |
| 20:22.58 | gtjoseph | it's inside a container |
| 20:23.04 | gtjoseph | so it's a new environment envery time |
| 20:24.19 | kharwell | not when I have been running it for my testing. However on valid runs it should successfully unsubscribe. So should be fairly cleared out |
| 20:24.44 | gtjoseph | yes but cleaned out isn't the same as not there |
| 20:25.11 | gtjoseph | i'm just thinking out loud |
| 20:25.30 | kharwell | okay I'll delete it and re-run the test in a loop and see if it triggers again. Maybe something was left from previous testing/failures |
| 20:25.45 | gtjoseph | what's the pass/fail rate now? |
| 20:26.08 | gtjoseph | delete the file at the top of every loop |
| 20:26.13 | kharwell | This time it triggered on the 64th run |
| 20:26.16 | file | DON'T ELETE IT |
| 20:26.33 | gtjoseph | wrong file :) |
| 20:30.23 | cresl1n | coreyfarrell: I'd be ok if https://gerrit.asterisk.org/#/c/asterisk/+/10479/ went into 13/16 |
| 20:31.01 | cresl1n | I was trying to figure out why 13 should be excluded, and if we feel that it's safe enough to be merged into 16 I don't see why we shouldn't put it into 13 as well |
| 20:31.39 | gtjoseph | 13 is LTS 16 isn't was my only thinking |
| 20:31.48 | cresl1n | It doesn't break API and shouldn't bust the world if we've reviewed it enough and decided that it's ready to merge it into a mainline branch |
| 20:31.53 | gtjoseph | make samples |
| 20:31.55 | cresl1n | also 16 is LTS as well |
| 20:31.56 | cresl1n | :-) |
| 20:31.57 | gtjoseph | oops |
| 20:32.10 | gtjoseph | oh yeah. |
| 20:32.11 | kharwell | gtjoseph: deleting the astdb.sqlite3 won't work/matter. It's created new for every test run. At least each test run has its own db file |
| 20:32.13 | file | LTS = Looks Too Salty |
| 20:32.35 | cresl1n | so if we're ok with busting 16 LTS we should be ok with busting 13 LTS as well |
| 20:32.36 | cresl1n | IMHO |
| 20:32.45 | gtjoseph | kharwell: yea i know. i wanted to see if it had anything to do with the f i l e creation itself. |
| 20:33.02 | gtjoseph | cresl1n: i'm fine with it then |
| 20:33.07 | cresl1n | (presumably it won't bust, since the test suite passes on it) |
| 20:33.13 | cresl1n | but in any case... |
| 20:33.17 | cresl1n | that's what I meant |
| 20:33.22 | file | dream the dream! |
| 20:34.05 | kharwell | gtjoseph: I'm missing something. If it's created for each test run there is nothing to delete prior to running it? |
| 20:34.24 | gtjoseph | yeah, the previous one |
| 20:34.31 | kharwell | cresl1n: no worries if stuff breaks we know who to blame |
| 20:34.37 | cresl1n | heh |
| 20:35.10 | gtjoseph | kharwell: it's created by asterisk. i just wanted to force asterisk to create it every time |
| 20:35.12 | file | the people who reviewed it? |
| 20:35.14 | file | :D |
| 20:35.22 | kharwell | gtjoseph: which previous one? For a given test run there is no previous astdb.sqlite fil e |
| 20:35.47 | gtjoseph | how are you running the test? |
| 20:35.51 | kharwell | gtjoseph: right it should already be doing that from what I can tell or understand how the testsuite works |
| 20:36.13 | kharwell | gtjoseph: I use the following command: |
| 20:36.14 | kharwell | while ./runtests.py -t tests/channels/pjsip/subscriptions/ast_restart/; do sleep 1; done |
| 20:36.27 | gtjoseph | where is asterisk installed? |
| 20:36.42 | kharwell | in the default location |
| 20:36.56 | kharwell | //usr |
| 20:37.33 | kharwell | there is no sqlite file under /var/lib/asterisk, so it must be creating a new on for each run and not copying or using the system one |
| 20:38.32 | kharwell | file: I'll blame Canadian healthcare |
| 20:38.55 | gtjoseph | ok |
| 20:49.19 | gtjoseph | kharwell: |
| 20:49.22 | gtjoseph | oops |
| 20:49.36 | gtjoseph | what happens if you just start and stop asterisk in a loop? |
| 20:50.35 | kharwell | I'll give it a shot and see. Want me to delete the db fil e I assume everytime? |
| 20:51.00 | gtjoseph | yeah, then see if it fails at all. |
| 20:51.25 | gtjoseph | should be faster than running teh test anyway. |
| 20:52.21 | gtjoseph | you could also have a separate sipp scenario sending a continual stream of subscribes as well |
| 20:52.56 | gtjoseph | what i really want to know is if the behavior manifests outside the testsuite. |
| 20:53.51 | kharwell | ya I'll start with asterisk |
| 20:58.22 | seanbright | kharwell: where's your BT? |
| 21:00.38 | kharwell | seanbright: here is the last one - https://pastebin.com/fN5an9TQ |
| 21:00.55 | seanbright | what specific version of sqlite3? |
| 21:01.32 | kharwell | 3.7.17 |
| 21:02.54 | seanbright | thanks |
| 21:15.22 | seanbright | kharwell: what distro? |
| 21:16.09 | kharwell | CentOS Linux release 7.4.1708 (Core) |
| 21:21.14 | coreyfarrell | gtjoseph: looks like you didn't notice my comment about '#define ASTMM_LIBC ASTMM_IGNORE' for Asterisk 16+ - https://gerrit.asterisk.org/10606 |
| 21:21.25 | gtjoseph | nope |
| 21:22.45 | seanbright | kinda weird that 'fsync' in your backtrace is coming from libpthread |
| 21:27.20 | seanbright | meh, libpthread exports it, so whatev. |
| 22:05.07 | *** join/#asterisk-dev sh_smith (~sh_smith@cpe-76-174-26-91.socal.res.rr.com) |
| 22:12.52 | *** join/#asterisk-dev coreyfarrell (~coreyfarr@24-181-237-90.dhcp.oxfr.ma.charter.com) |
| 22:12.52 | *** mode/#asterisk-dev [+o coreyfarrell] by ChanServ |
| 22:24.43 | *** join/#asterisk-dev leifmadsen (~leifmadse@asterisk/documenteur-extraordinaire/blitzrage) |
| 22:24.43 | *** mode/#asterisk-dev [+o leifmadsen] by ChanServ |
| 22:31.37 | *** join/#asterisk-dev kharwell (kharwell@nat/digium/x-bxburfrgkmtrtphb) |
| 22:31.37 | *** mode/#asterisk-dev [+o kharwell] by ChanServ |