| 04:53.25 | *** join/#ldstech psa (~yomode@50.35.199.100) |
| 14:11.44 | *** join/#ldstech scgallafent (~scgallafe@c-67-160-61-229.hsd1.wa.comcast.net) |
| 14:11.44 | *** mode/#ldstech [+o scgallafent] by ChanServ |
| 14:43.30 | *** join/#ldstech tdeforest (tdeforest@conference/ldstech/x-fpqpdcxqlskdofsh) |
| 14:43.30 | *** mode/#ldstech [+o tdeforest] by ChanServ |
| 14:45.28 | *** join/#ldstech ChadF (~ChadF@75-162-239-127.slkc.qwest.net) |
| 14:47.35 | *** join/#ldstech josephscott (~josephsco@208.95.156.134) |
| 14:47.56 | *** join/#ldstech smazga (~mckay@64.55.45.194) |
| 14:53.57 | *** join/#ldstech TurboLDS (~quassel@67-2-159-89.slkc.qwest.net) |
| 14:56.31 | *** join/#ldstech zakzakattack (zakzakatta@conference/ldstech/x-uqucqognmjltsxsr) |
| 15:00.07 | *** join/#ldstech ekeating (~Dennis@99-54-46-12.lightspeed.snjsca.sbcglobal.net) |
| 15:00.18 | ekeating | ~meetstart <LDSTech Dev Scrum> |
| 15:00.18 | infobot | ********** Start of the <LDSTech Dev Scrum> Meeting on 2012.08.27 15:00:18 ********** |
| 15:00.32 | ekeating | Godd Morning everyone |
| 15:00.35 | scgallafent | Moin |
| 15:00.36 | ekeating | that's good not godd |
| 15:00.39 | TurboLDS | Greetings |
| 15:00.46 | tdeforest | howdy |
| 15:01.12 | ChadF | HI |
| 15:01.28 | ekeating | scgallafent, let's begin with you |
| 15:01.35 | scgallafent | I spent some of my time over the weekend working on Matchmaker, but most of it fighting with the production server. |
| 15:01.35 | scgallafent | Load spiked on Saturday and was causing problems all weekend. |
| 15:01.35 | scgallafent | I expect there will be more server tweaking today and hope to spend some time on Matchmaker also. |
| 15:01.35 | scgallafent | I have two remaining items open for the current sprint: |
| 15:01.35 | scgallafent | https://tech.lds.org/jira/browse/LDSTECH-362 - Probably won't be able to complete because the header needed isn't available in production. |
| 15:01.38 | scgallafent | https://tech.lds.org/jira/browse/LDSTECH-774 - Have been working on Fortify issues between other items. |
| 15:01.39 | scgallafent | That is all. |
| 15:02.18 | ekeating | thank you |
| 15:02.22 | ekeating | tdeforest, |
| 15:02.48 | tdeforest | Had a great vacation last week! |
| 15:03.00 | tdeforest | still moving my stuff in this morning to the new office space |
| 15:03.08 | ekeating | where did you go? |
| 15:03.34 | tdeforest | i was assigned a bunch of tickets while i was out. jdavid26 had a baby last week and ChadF gave me his tickets |
| 15:03.40 | ChadF | :) |
| 15:03.53 | tdeforest | i'll be focusing on those tickets once i'm set up |
| 15:04.07 | tdeforest | talent matching will take a back seat while i get those tickets done for this sprint |
| 15:04.13 | ekeating | See what happens when your on vacation :-) |
| 15:04.18 | tdeforest | :) |
| 15:04.30 | tdeforest | glad to see the couch in smootar's office |
| 15:04.49 | tdeforest | i think that is all for me |
| 15:04.55 | ekeating | thanks |
| 15:04.56 | *** join/#ldstech LDSTech (LDSTech@conference/ldstech/x-jipdzhpmruoliafs) |
| 15:04.57 | *** mode/#ldstech [+o LDSTech] by ChanServ |
| 15:05.02 | ekeating | ChadF, How did the move go? |
| 15:05.15 | ChadF | fine - |
| 15:05.32 | ChadF | chaotic, but new office is nice |
| 15:06.20 | ChadF | my back is thrown out (not from the move) - completely locked up - spent Fri afternoon, Sat, and Sunday flat on my back. excruciating pain if I am sitting or standing. |
| 15:06.41 | ChadF | I rebuilt the CSM spreadsheet Friday to fix several issues with it |
| 15:07.06 | ChadF | I'll be rebuilding our form for CSM prospects today, and a new form for ICS managers to list their new opps |
| 15:07.15 | ChadF | more Jira work as well - |
| 15:07.19 | ChadF | that's it for me today |
| 15:07.22 | scgallafent | would like to see those forms.... |
| 15:07.36 | ekeating | Been there, done that. I remember curling up in a ball when relatives were here and couldn't even get up to say goodbye. :( |
| 15:07.50 | *** join/#ldstech TimRiker (TimRiker@bzflag/projectlead/TimRiker) |
| 15:07.50 | *** mode/#ldstech [+o TimRiker] by ChanServ |
| 15:08.07 | ekeating | morning TimRiker |
| 15:08.16 | ekeating | do you have a report for today? |
| 15:08.23 | ChadF | ok scgallafent - they're simple google forms, but they enable us to push the data entry to the prospects and hiring managers |
| 15:08.27 | TimRiker | moin |
| 15:09.20 | TimRiker | wiki is still down. going to spend as much time today as needed to track down what's wrong with the server. |
| 15:09.31 | TimRiker | cpu load is way too high, and we don't know why. |
| 15:09.56 | TimRiker | might try moving to another server to see if we get the same results. ie: copy data to staging server etc. |
| 15:10.08 | scgallafent | I saw it down at 0.68 overnight. Didn't last. :( |
| 15:10.11 | TimRiker | also going to dig in and confirm that memcached is working correctly. |
| 15:10.14 | ekeating | if it isn't one thing, it's another :) |
| 15:10.43 | TimRiker | scgallafent and I have put quite a bit of time into it already. still no solution. |
| 15:11.40 | TimRiker | I miss my window view. I live in a cave now. guess it's just in time for Halloween. :) |
| 15:12.21 | TimRiker | that's it for me |
| 15:12.31 | tdeforest | TimRiker: put some out door scenes on one of those many monitors you have |
| 15:12.39 | ekeating | watch out for the bats |
| 15:12.40 | tdeforest | :) |
| 15:13.05 | ekeating | any other reports for today? |
| 15:13.51 | zakzakattack | well i had a good move on friday... but today someone tried to take overmy desk over the weekend . i have to put my desk back together |
| 15:14.21 | ekeating | mist be gremlins |
| 15:14.30 | ekeating | mist=must |
| 15:14.53 | ekeating | I'm a little confused (not unusual). Will there be a staff meeting tomorrow or not? |
| 15:15.24 | ekeating | Alan said something about an "all hands" meeting |
| 15:17.16 | ekeating | Unfortunately we're over time so I'll end this meeting. Have a good day |
| 15:17.28 | scgallafent | I still show staff meeting, although it's being held in 3166 instead of 1358. I don't see ICS all hands on the calendar. |
| 15:17.31 | ekeating | ~meetend <LDSTech Dev Scrum> |
| 15:17.31 | infobot | ********** End of the <LDSTech Dev Scrum> Meeting on 2012.08.27 15:17:31 ********** |
| 15:17.46 | ekeating | thanks scgallafent |
| 15:17.56 | scgallafent | Probably ought to ask smootar later. |
| 15:18.04 | ekeating | will do |
| 15:18.22 | scgallafent | TimRiker, load was down to an inspiring 0.68 overnight. It has been climbing back up this morning. |
| 15:18.49 | scgallafent | I tried reenabling the wiki at 0.68 and things spiked pretty quickly. |
| 15:19.08 | scgallafent | Any brilliant ideas other than move to a different server and see if the problem persists? |
| 15:19.22 | scgallafent | I haven't heard anything from dlhace. |
| 15:23.08 | *** part/#ldstech TurboLDS (~quassel@67-2-159-89.slkc.qwest.net) |
| 15:33.04 | *** join/#ldstech johnsonth (JohnsonTH@conference/ldstech/x-lseelwnromiaapxf) |
| 15:46.55 | *** part/#ldstech scgallafent (~scgallafe@c-67-160-61-229.hsd1.wa.comcast.net) |
| 15:47.01 | *** join/#ldstech scgallafent (~scgallafe@c-67-160-61-229.hsd1.wa.comcast.net) |
| 15:47.01 | *** mode/#ldstech [+o scgallafent] by ChanServ |
| 15:47.14 | *** join/#ldstech cognifloyd (~cognifloy@pat00.orem.infotraxsys.com) |
| 15:47.15 | *** mode/#ldstech [+v cognifloyd] by ChanServ |
| 15:49.47 | TimRiker | scgallafent, going to try messing with memcached a bit. |
| 15:49.59 | scgallafent | OK. |
| 15:51.01 | *** join/#ldstech smootar (smootar@conference/ldstech/x-oonnczsatonevjpf) |
| 15:51.31 | *** mode/#ldstech [+o smootar] by ChanServ |
| 15:51.34 | TimRiker | scgallafent, I had moved the redirect to localsettings as it catches other entry points into the wiki there but not in index.php |
| 15:52.07 | scgallafent | When I looked at it yesterday, there was a delay of a couple of seconds of processing before it hit the redirect. |
| 15:52.39 | scgallafent | LocalSettings happens after a boatload of other stuff. |
| 15:52.45 | scgallafent | How are they getting in without going through index.php? |
| 15:53.44 | scgallafent | It doesn't hurt to have it in both places. |
| 15:53.45 | TimRiker | load.php is the one I was seeing in the logs |
| 15:53.51 | scgallafent | adds it back to LocalSettings |
| 15:54.02 | TimRiker | I just took it out... |
| 15:54.11 | TimRiker | running with memcached disabled as a test |
| 15:54.55 | TimRiker | load heads right back up... :( |
| 15:54.58 | TimRiker | disabling again. |
| 15:55.20 | scgallafent | I'm watching vmstat. It's definitely processor. |
| 15:55.37 | scgallafent | I was seeing a lot of block input/output yesterday, but the backup was running. |
| 15:55.58 | scgallafent | Very little block I/O today, but processes waiting just spiked to 30 and then fell back off. |
| 16:00.17 | TimRiker | running a find... expect io |
| 16:03.36 | scgallafent | Big jump in waiting requests. Did you just start your find? |
| 16:04.12 | TimRiker | it's done now... |
| 16:04.28 | TimRiker | was done before the spike. not sure what caused it |
| 16:04.29 | scgallafent | Starting to fall off. |
| 16:05.17 | scgallafent | I don't see a huge spike in HTTP requests. |
| 16:05.30 | scgallafent | Have to run drop off a key. Back in ten. |
| 16:06.20 | TimRiker | some processes don't map to usernames correctly. uid 68 and 206 |
| 16:21.07 | scgallafent | I'm back. Daughter locked her keys in the car. |
| 16:21.22 | TimRiker | fun fun |
| 16:24.13 | scgallafent | I did some testing with serving static files yesterday (wiki/images/....) and didn't see a huge spike. It looked like it was only when httpd invoked PHP on those pages. |
| 16:25.05 | TimRiker | nods |
| 16:28.52 | scgallafent | TimRiker, spoike about twenty seconds ago. Was that something you did? |
| 16:29.11 | scgallafent | s/spoike/spike/ |
| 16:29.32 | TimRiker | nope. heading to devotional for a 1/2 or so... |
| 16:48.03 | *** join/#ldstech tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 16:56.31 | *** join/#ldstech smootar_ (smootar@conference/ldstech/x-doqfwcpyulhluigg) |
| 16:56.32 | *** mode/#ldstech [+o smootar_] by ChanServ |
| 17:02.36 | *** join/#ldstech zakzakattack (zakzakatta@conference/ldstech/x-wxqkpvolkpkjpwyi) |
| 17:04.14 | *** join/#ldstech tdeforest (tdeforest@conference/ldstech/x-njjweauogyxkssvg) |
| 17:04.14 | *** mode/#ldstech [+o tdeforest] by ChanServ |
| 17:07.00 | TimRiker | scgallafent, got the video hardware in "our" office. wanna call in? |
| 17:07.16 | scgallafent | Give me a minute to wrap up a phone call and I will |
| 17:22.07 | scgallafent | says hi to himself |
| 17:32.54 | TimRiker | time dd if=/dev/zero of=/dev/null bs=1G count=30 |
| 17:42.17 | *** join/#ldstech smootar (smootar@conference/ldstech/x-dcsybrglnidxgrji) |
| 17:42.17 | *** mode/#ldstech [+o smootar] by ChanServ |
| 17:45.23 | TimRiker | scgallafent, what's the exact sysbench command you used? |
| 17:45.41 | scgallafent | sysbench --test=cpu --cpu-max-prime=1000 run |
| 17:45.43 | scgallafent | niced, of course |
| 17:46.02 | scgallafent | Or un-niced, as the case may be |
| 17:46.21 | TimRiker | total time: 0.4876s on l7869 |
| 17:46.45 | scgallafent | 5.0239s on L12426, 5.0333s on L12427 |
| 17:46.58 | TimRiker | again more than a factor of 10 |
| 17:47.44 | scgallafent | Sounds like the ops/VM group needs to look at something. |
| 17:55.57 | TimRiker | talked to dlhace again. he's trying to get some answers |
| 17:56.14 | scgallafent | OK |
| 17:58.38 | TimRiker | perhaps I should convert l5488 to rhel and then move back there. :) |
| 18:03.15 | tdeforest | TimRiker, scgallafent i posted a note about the outage on the ldstech blog |
| 18:12.18 | TimRiker | tdeforest, thx. waiting to hear from the server team |
| 18:13.08 | TimRiker | maybe I should move tech.lds.org over to my raspberry pi. It might perform better there. |
| 18:13.15 | scgallafent | :) |
| 18:14.57 | tdeforest | heh |
| 18:27.35 | *** join/#ldstech smootar_ (smootar@conference/ldstech/x-hmarpcijyiqxbczz) |
| 18:27.36 | *** mode/#ldstech [+o smootar_] by ChanServ |
| 18:27.59 | TimRiker | scgallafent, probably want to whitelist tdeforest and smootar |
| 18:28.07 | josephscott | scgallafent: I'm curious to know what the results of your framework tests were |
| 18:30.27 | scgallafent | josephscott, on a call with TimRiker. I'll chase you down in a minute. |
| 18:30.54 | josephscott | np |
| 18:37.26 | TimRiker | so, how do I call you? |
| 18:37.35 | scgallafent | looks |
| 18:39.48 | TimRiker | don't see you. your calls come from a gateway and I can't call that back. |
| 18:40.20 | scgallafent | You can't dial a PVC extension directly? |
| 18:41.03 | TimRiker | dunno. what's you extension? just your username? I don't see your name listed in the directory, though I do see smootar |
| 18:41.22 | scgallafent | What does it show as smootar's extension? End in 040? |
| 18:42.19 | TimRiker | yep |
| 18:42.54 | scgallafent | I'm not sure why I'm not listed. Can you call smootar? |
| 18:43.41 | TimRiker | I can dial myself by extension. I tried smootar but he might not be in his office. |
| 18:44.07 | scgallafent | If you can dial yourself by extension, you should be able to dial me. |
| 19:12.25 | *** join/#ldstech LDSTechRPI (pi@conference/ldstech/x-zkegzvjvznarkvga) |
| 19:12.25 | *** mode/#ldstech [+o LDSTechRPI] by ChanServ |
| 20:00.28 | *** join/#ldstech smootar (smootar@conference/ldstech/x-rzaobfbybvewrmsx) |
| 20:00.34 | *** mode/#ldstech [+o smootar] by ChanServ |
| 20:06.32 | smootar | I see missed calls from PVC. |
| 20:06.47 | smootar | I was not in my office at the time. |
| 20:07.24 | TimRiker | trying things out. scgallafent is virtually here. |
| 20:07.30 | scgallafent | We're in 3162. |
| 20:07.31 | smootar | I can hear scgallafent over the pvc |
| 20:07.37 | TimRiker | :) |
| 20:07.39 | scgallafent | I think that's my office number. |
| 20:07.40 | smootar | Hello |
| 20:07.58 | scgallafent | You should come into here and we'll talk to you. |
| 20:08.06 | smootar | Just starting a meeting with chetan |
| 20:08.25 | scgallafent | Then you probably shouldn't come in here. |
| 20:19.31 | *** join/#ldstech ChadF (~ChadF@75-162-239-127.slkc.qwest.net) |
| 21:01.30 | TimRiker | call in again? kevin is here |
| 21:09.39 | TimRiker | headed out to my kids school. call my cell if the server team gets around to us. |
| 21:09.53 | *** join/#ldstech josephscott (~josephsco@208.95.156.134) |
| 21:46.00 | tdeforest | scgallafent: how goes the war? |
| 21:46.30 | scgallafent | Which part? My part is going fine. Waiting for the server team ... not so much. |
| 22:09.57 | *** join/#ldstech smootar_ (smootar@conference/ldstech/x-bzggzktzezhiawhw) |
| 22:09.57 | *** mode/#ldstech [+o smootar_] by ChanServ |
| 22:11.01 | tdeforest | scgallafent: thanks for the update. |
| 22:11.14 | scgallafent | Unfortunately, not much update. |
| 22:11.30 | tdeforest | i've heard tehre are issues with the forum but it doesn't seem to have any problems |
| 22:25.58 | *** join/#ldstech smootar (smootar@conference/ldstech/x-lezwvilejxyozmca) |
| 22:25.59 | *** mode/#ldstech [+o smootar] by ChanServ |
| 22:35.56 | scgallafent | tdeforest, there are intermittent issues with the forum. The server is performing poorly. The wiki is the first thing to go since it's got the most complex code. |
| 22:36.07 | scgallafent | Once the load starts to spike, the forum starts having problems. |
| 22:36.20 | tdeforest | i see |
| 22:36.27 | scgallafent | TimRiker and I did some troubleshooting today and handed it off to the server team and dlhace. |
| 22:36.47 | scgallafent | Now we're waiting for some help from them on why we're seeing a performance difference between servers. |
| 22:37.28 | scgallafent | doesn't much like the waiting |
| 22:48.19 | tdeforest | agrees with scgallafent on the waiting |
| 22:48.41 | tdeforest | scgallafent: i grabbed your green cable when i moved offices |
| 22:48.52 | scgallafent | Thanks. |
| 22:58.55 | *** join/#ldstech smootar_ (smootar@conference/ldstech/x-xvcejqnpgnhzpsfc) |
| 22:58.55 | *** mode/#ldstech [+o smootar_] by ChanServ |
| 22:59.21 | tdeforest | heads out and hopes that the server team will come to life soon |
| 23:13.21 | *** join/#ldstech ChadF (~ChadF@75-162-239-127.slkc.qwest.net) |