| 00:02.41 | harleypig | Doing a straight compare on the env output doesn't work ... I have too much customization setup on serverA |
| 00:03.00 | harleypig | but I'm looking for anything that seems weird on serverralph |
| 00:03.08 | jlp_znc | well, only specific variables should go through |
| 00:03.28 | jlp_znc | compare your sshd_config and ssh_config to see what gets passed. mostly things like LANG, LC_TYPE, etc. |
| 00:03.48 | jlp_znc | have you tried with some generic user (like, create a new user with a clean environment just for testing)? |
| 00:04.37 | harleypig | I haven't ... but this is a brand new installation. |
| 00:06.50 | harleypig | wait ... you mean a new user on serverA and serverRalph? |
| 00:07.49 | jlp_znc | yeah |
| 00:07.55 | jlp_znc | and host |
| 00:13.37 | harleypig | I created a barebones user called testssh on all 3 servers. |
| 00:14.03 | harleypig | serverA still works and connects to user testssh (with password 12345) |
| 00:14.21 | harleypig | serverRalph still does not work when connecting to user testssh (with password 12345) |
| 00:15.24 | fugue88 | harleypig: For kicks and giggles, try this on serverRalph: "ssh localhost". Does that work? I'm still suspicious of your terminal. |
| 00:16.01 | fugue88 | I'm anticipating that you'll be asked for a password. |
| 00:16.36 | harleypig | Yeah, a prompt for a password was given, and it worked. |
| 00:16.52 | fugue88 | kk |
| 00:17.06 | jlp_znc | I still think running sshd -p 2222 -ddde on host would be useful |
| 00:17.52 | harleypig | sshd re-exec requires execution with an absolute path |
| 00:18.06 | jlp_znc | so /usr/sbin/sshd -p 2222 -ddde |
| 00:20.27 | harleypig | Hmmm ... mm_auth_password: user not authenticated |
| 00:21.43 | harleypig | wtf? when trying ssh -p 2222 testssh@host from serverA I got connection refused. |
| 00:22.23 | *** join/#utah fugue88 (~dsowen@fugue88.ws) |
| 00:22.44 | jlp_znc | oh, -e probably only runs one connection |
| 00:22.49 | jlp_znc | restart it |
| 00:24.25 | jlp_znc | turn PamAuth off, then try it again |
| 00:25.08 | harleypig | It worked after restarting it, from serverA |
| 00:25.12 | jlp_znc | /usr/sbin/sshd -p 2222 -ddde -o UsePAM=no |
| 00:25.17 | harleypig | trying with PamAuth off |
| 00:26.11 | harleypig | there it is. |
| 00:26.27 | jlp_znc | so it works without PAM, but fails with PAM? |
| 00:26.29 | harleypig | from serverralph, with pam off on the host I was able to connect. |
| 00:26.38 | harleypig | yeah |
| 00:26.44 | harleypig | but only from serverralph |
| 00:26.59 | jlp_znc | there were some really old bugs in PAM (like, from 2003), but I would think they were all fixed by now |
| 00:27.06 | jlp_znc | must be something else in the pam config |
| 00:27.27 | jlp_znc | take a gander through /etc/pam.d/system-auth |
| 00:29.04 | harleypig | That's here: https://www.irccloud.com/pastebin/5QfHQtol |
| 00:29.47 | jlp_znc | yeah, I don't see anything weird in there |
| 00:32.59 | harleypig | So, with pam off, login just compares the password against what's in /etc/shadow, correct? |
| 00:33.21 | harleypig | But with pam on, the password lines are used |
| 00:33.30 | harleypig | err ... the password lines in system-auth |
| 00:33.32 | jlp_znc | I think so |
| 00:36.25 | harleypig | wait ... if I'm logging in via ssh, would /etc/pam.d/login be used? |
| 00:37.02 | jlp_znc | no, /etc/pam.d/sshd |
| 00:37.28 | jlp_znc | my sshd includes password-auth instead of system-auth |
| 00:37.44 | jlp_znc | but I don't see anything in password-auth that should cause the failure you are seeing |
| 00:37.49 | jlp_znc | I'm still blaming the DNS :-) |
| 00:46.55 | *** join/#utah undertakingyou (~undertaki@c-67-171-113-60.hsd1.ut.comcast.net) |
| 00:48.11 | jlp_znc | so what's pam_hulk? |
| 00:52.26 | harleypig | Ok, I figured out how to debug pam. I get this: PAM-hulk[4638]: Brute force detection active: 580 LOGIN DENIED -- TOO MANY FAILURES -- IP TEMP BANNED |
| 00:52.36 | harleypig | pam_hulk appears to be a cpanel module |
| 00:52.48 | jlp_znc | I don't have pam_hulk on my machine, so no man page for it :-( |
| 00:53.14 | harleypig | yeah, there's no man page for it on the container either. |
| 00:54.35 | jlp_znc | and if I search for it, I get the incredible hulk or "hulk: pam anderson wants to tongue wrestle" :-( |
| 00:55.05 | harleypig | :D |
| 00:55.10 | harleypig | search for 'pam pam_hulk.so' |
| 00:55.27 | jlp_znc | is it a cpanel module? |
| 00:55.43 | harleypig | ah ... here it is, cphulk |
| 00:55.47 | harleypig | yeah, it looks like it. |
| 00:55.54 | jlp_znc | ugh |
| 00:56.04 | jlp_znc | /usr/local/cpanel/bin/cphulk_pam_ctl --disable |
| 00:56.09 | jlp_znc | then stop |
| 00:56.11 | jlp_znc | :-) |
| 00:56.34 | jlp_znc | okay, harleypig, I've got to go catch a train. looks like you are on the right track now. |
| 00:56.40 | jlp_znc | catch you later |
| 00:56.50 | harleypig | thank you for your help |
| 00:57.22 | jlp_znc | no problem, I'm glad we isolated it |
| 00:57.56 | harleypig | feh ... it turns out all I need to do is /scripts/cphulkdwhitelist ip |
| 01:37.29 | *** join/#utah Gate (~gate@162.243.149.109) |
| 01:37.29 | *** join/#utah Gate (~gate@unaffiliated/gate) |
| 01:42.18 | *** join/#utah StealthTaco_lptp (~E_lptp@67-2-184-175.slkc.qwest.net) |
| 02:40.20 | *** join/#utah olli_ (~olli@c-71-195-232-74.hsd1.ut.comcast.net) |
| 02:44.02 | *** join/#utah vissborg (~viskestel@li607-220.members.linode.com) |
| 03:33.03 | *** join/#utah Magmu (~Magmu@66.7.124.45) |
| 03:49.08 | *** join/#utah janderson (~janderson@50-202-143-210-static.hfc.comcastbusiness.net) |
| 04:21.57 | *** join/#utah yukaia_ (~yukaia@174-23-112-47.slkc.qwest.net) |
| 05:07.44 | *** join/#utah bgerber (~gerberb@suse103.zenez.com) |
| 06:07.04 | *** join/#utah jalbretsen (~jalbretse@75-169-1-241.slkc.qwest.net) |
| 06:34.47 | jxqz | cphulk.so, srsly? |
| 06:43.45 | *** join/#utah T4rku5 (~74rku5@unaffiliated/shunkydave) |
| 07:04.05 | *** join/#utah T4rku5 (~74rku5@unaffiliated/shunkydave) |
| 10:53.46 | *** join/#utah yukaia__work (~yukaia@198.183.219.9) |
| 11:03.18 | *** join/#utah infobot (ibot@rikers.org) |
| 11:03.18 | *** topic/#utah is Utah related discussion including user groups | logs @ http://ibot.rikers.org/%23utah/ | See also ##lds | OpenWest Conference sometime 2014! openwest.org | https://fiber.google.com/cities/provo/ |
| 11:03.18 | *** mode/#utah [+o infobot] by ChanServ |
| 11:25.40 | *** join/#utah infobot (ibot@rikers.org) |
| 11:25.40 | *** topic/#utah is Utah related discussion including user groups | logs @ http://ibot.rikers.org/%23utah/ | See also ##lds | OpenWest Conference sometime 2014! openwest.org | https://fiber.google.com/cities/provo/ |
| 11:25.40 | *** mode/#utah [+o infobot] by ChanServ |
| 11:32.28 | *** join/#utah Migs (~migs@205.122.50.108) |
| 11:41.43 | *** join/#utah yukaia_ (~yukaia@174-23-112-47.slkc.qwest.net) |
| 12:37.45 | *** join/#utah StealthTaco_lptp (~E_lptp@67-2-184-175.slkc.qwest.net) |
| 12:44.34 | *** join/#utah olli_ (~olli@c-71-195-232-74.hsd1.ut.comcast.net) |
| 12:57.07 | *** join/#utah macnewbold (~macnewbol@c-24-2-87-227.hsd1.ut.comcast.net) |
| 13:43.10 | *** join/#utah pdayton (~pjdayton@63.251.sfcn.org) |
| 13:50.16 | *** join/#utah josephscott (~josephsco@192.0.65.252) |
| 14:09.58 | *** join/#utah WebBugsByte (~webbugsby@c-76-23-60-71.hsd1.ut.comcast.net) |
| 14:19.13 | *** join/#utah macnewbold (~macnewbol@c-24-2-87-227.hsd1.ut.comcast.net) |
| 14:29.53 | *** join/#utah T4rku5 (~74rku5@unaffiliated/shunkydave) |
| 15:00.36 | *** join/#utah tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 15:06.57 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 15:32.10 | *** join/#utah manithree (~blr@li4-76.members.linode.com) |
| 15:42.20 | eightyeight | http://ipfs.io/ |
| 15:42.52 | *** join/#utah jalbretsen (~jalbretse@sonic.itransact.com) |
| 15:49.47 | *** join/#utah neybar_work (~quassel@74-220-196-120.unifiedlayer.com) |
| 15:53.13 | *** join/#utah utlemming (~ben@97-118-224-106.hlrn.qwest.net) |
| 15:53.14 | *** join/#utah utlemming (~ben@ubuntu/member/utlemming) |
| 15:57.35 | jima | eightyeight: thank goodness no one in here cares about network storage |
| 15:57.50 | *** join/#utah peno (~irc4me@c-98-202-160-116.hsd1.ut.comcast.net) |
| 15:59.13 | eightyeight | it's an interesting concept, i think |
| 16:01.45 | *** join/#utah macnewbold1 (~macnewbol@inferno.apxalarm.com) |
| 16:06.48 | *** join/#utah Oberon4278 (~dcrooksto@38.112.155.163) |
| 16:09.45 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 16:12.17 | Migs | http://3dprint.com/17339/3d-printed-quadcopter-elf/ |
| 16:12.37 | Migs | two of my favorite things, mini-quadcopters and Google Cardboard, together at last! |
| 16:14.53 | *** join/#utah jsmith` (~jsmith@207.198.119.103) |
| 16:15.13 | *** join/#utah jima_ (jima@laughton.us) |
| 16:16.41 | *** join/#utah opapo_ (~quassel@204.113.97.239) |
| 16:17.52 | *** join/#utah Tene (~tene@mail.digitalkingdom.org) |
| 16:17.52 | *** join/#utah Tene (~tene@poipu/supporter/slacker/tene) |
| 16:17.54 | *** join/#utah tristanbob_l (~tristanbo@137.190.208.10) |
| 16:18.42 | Oberon4278 | We've got some ham radio operators here right? |
| 16:21.41 | programmerq | yup |
| 16:21.56 | eightyeight | Oberon4278: yeah. what's up? |
| 16:21.57 | Oberon4278 | Are you one of them? :D |
| 16:22.08 | Oberon4278 | So, I'm trying to tell a war story and I want to make sure I get the details right. |
| 16:22.15 | Oberon4278 | Or at least plausible. |
| 16:22.58 | Oberon4278 | I was in Iraq, working with Shadow UAVs, to which we connected via a directional antenna |
| 16:23.50 | Oberon4278 | and for a couple of days we had fairly serious problems with our signal that we eventually traced to solar flares that were causing broad-spectrum EM interference |
| 16:24.33 | Oberon4278 | if I recall correctly, the interference was measured in decibels - is that correct? |
| 16:25.11 | programmerq | yeah, that sounds right. |
| 16:25.19 | *** join/#utah kim27 (~kim27@2604:a880:800:10::bc:6001) |
| 16:25.19 | *** join/#utah kim27 (~kim27@unaffiliated/kim27) |
| 16:25.29 | Oberon4278 | also, iirc, we had about 2.5 times the normal amount of background noise |
| 16:25.38 | Oberon4278 | is that even possible just from solar flares? |
| 16:31.41 | youtah | #HashTag |
| 16:33.55 | jima | youtah: /join #HashTag |
| 16:34.08 | jima | hash tag...is that a game played in amsterdam? |
| 16:44.51 | eightyeight | #hashtag https://www.youtube.com/watch?v=57dzaMaouXA |
| 16:47.06 | jima | guesses (without looking) weird al |
| 16:48.02 | eightyeight | nope |
| 16:48.09 | jsmith | jima: Funny you should say that.... guess where I am today... |
| 16:48.19 | eightyeight | weird al is still a thing these days? |
| 16:48.21 | jima | in weird al's house? |
| 16:48.25 | jsmith | jima: (and no, I didn't play -- I played the "Sherlocked" escape game instead) |
| 16:48.32 | jima | eightyeight: how's the weather under that rock? |
| 16:48.55 | eightyeight | the last weird al i knew was vhf, or uhf, or whatever it was |
| 16:49.05 | jima | uhf, iirc |
| 16:49.17 | eightyeight | oh wait. he did a spoof on lourde |
| 16:49.19 | jima | here's a little update: he also has a music career. |
| 16:49.33 | eightyeight | yeah. never got into it. |
| 16:49.37 | eightyeight | my brother did though |
| 16:50.26 | eightyeight | weird al "foil" https://www.youtube.com/watch?v=urglg3WimHA |
| 16:50.35 | eightyeight | the other was jimmy fallon and justin timberlake |
| 16:50.43 | jima | he did a batch of recent videos |
| 16:50.49 | jima | including, yes, foil |
| 17:01.17 | manithree | I have to google the original songs Al parodies these days. I don't know any of them but "White and Nerdy" was hilarious. |
| 17:01.39 | jima | i usually at least know the titles |
| 17:01.47 | jima | (like ridin' dirty, iirc) |
| 17:02.07 | manithree | I had never heard of Ridin' Dirty or The Benjamins |
| 17:02.20 | manithree | It's all about the benjamins, I guess |
| 17:07.29 | Oberon4278 | So, is 100dBm an insane amount of noise to expect from a solar flare? Is that even how you'd measure it? |
| 17:51.09 | youtah | lol jima |
| 17:53.54 | jlp_znc | Oberon4278: did you ever figure out your core file problem from the other day? |
| 17:54.21 | Oberon4278 | nope |
| 17:54.31 | Oberon4278 | well, I mean, the overarching problem yes |
| 17:54.37 | Oberon4278 | but not via the core file |
| 17:55.02 | Oberon4278 | I started asking around about it and someone said "Dude if you want to see the lock table you can just run a cstat report. It's in section 5." |
| 17:55.20 | Oberon4278 | I mean I guess it did come from the core file, but it turns out we have a tool that just does it. |
| 17:55.28 | jlp_znc | cool. |
| 17:55.54 | Oberon4278 | Not *nearly* as cool as being a l33t h@xz0r! |
| 17:56.22 | Oberon4278 | I keep trying to do these awesome obscure things and then someone's like "Yeah Bob over in accounting wrote a script for that back when he was in development, don't waste your time shaving yaks." |
| 17:56.44 | jlp_znc | I was going to say that shared memory segments could be still active in the system. ipcs would show you where they were and you could attach to one from another program. also, the core file is pretty useless without the symbol table from the oringinal executable. with that symbol table, however, you can do all kinds of stuff with a core file. |
| 17:56.48 | Oberon4278 | Or, "That's a neat idea but our customers would never allow us to connect directly to their system to get that data in real time." |
| 17:57.09 | Oberon4278 | They killed the locks in question, so the data we needed wouldn't be there any more. |
| 17:57.41 | Oberon4278 | He took the core file (with shared memory) before killing them specifically so we'd have good data for a root cause analysis, but the orphaned locks were causing serious problems in their production environment so they had to go. |
| 17:57.49 | jlp_znc | ahh |
| 17:58.05 | Oberon4278 | We have the original executable and all the code used to produce it in our libraries. |
| 17:58.39 | Oberon4278 | The longer I work in IT the more the benefits of a code repository become celar. |
| 17:58.42 | Oberon4278 | clear, too. |
| 17:59.17 | *** join/#utah emcnabb (~emcnabb@cpe-107-015-189-013.nc.res.rr.com) |
| 17:59.17 | *** mode/#utah [+v emcnabb] by ChanServ |
| 18:24.51 | *** join/#utah Oberon (~dcrooksto@38.112.155.163) |
| 18:33.04 | jxqz | rockstar ninja data scientist |
| 18:33.20 | jxqz | I think that's my new job title |
| 18:34.27 | jxqz | being a programmer is awesome because it means you can irrationally inflate your largely meaningless title drolly |
| 18:35.24 | Oberon | God I hate it when I see job postings saying they're looking for "rock stars" or "ninjas". |
| 18:36.51 | Oberon | Few things make me lose interest faster. |
| 18:39.59 | jxqz | haha |
| 18:40.48 | jxqz | "do you just love staying up to 3:00 AM rocking out code for our customers?" |
| 18:41.04 | Oberon4278 | Maybe they're just targeting a different demographic. |
| 18:41.22 | Oberon4278 | Specifically, one that's younger. |
| 18:42.31 | jxqz | exploitation |
| 18:42.48 | jxqz | they're exploiting a younger demographic |
| 18:42.59 | jxqz | "we'll pay you $18/hour!" |
| 18:43.14 | jxqz | is cynical |
| 18:44.06 | Oberon4278 | Cynical, realistic -- six of one, etc. |
| 18:46.41 | Oberon4278 | I feel like not caring (in this case, that our documentation is shit) should be a skill, and that I should be able to practice that skill. |
| 18:46.49 | Oberon4278 | But no matter what I try, I still care. |
| 18:54.48 | *** join/#utah josephscott (~josephsco@proxy.automattic.com) |
| 18:57.54 | *** join/#utah dremex (~dremex@www.xmech.net) |
| 19:03.31 | *** join/#utah undertakingyou (~undertaki@neutroninteractive-xmission.xmission.com) |
| 19:11.01 | *** join/#utah dremex (~dremex@www.xmech.net) |
| 19:12.08 | *** join/#utah StealthTaco_Work (~StealthTa@198.183.219.9) |
| 19:13.53 | jxqz | keep exercising |
| 19:14.03 | jxqz | you'll get better |
| 19:36.55 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 19:47.51 | RyanE | "Idiot-ignoring Ninja" since '93. |
| 19:48.28 | jxqz | what do ninjas even do? |
| 19:48.37 | RyanE | Ignore idiots, mostly. |
| 19:48.44 | jxqz | haxor code beyond all possible recognition |
| 19:48.49 | jxqz | and attack rms at night |
| 19:48.51 | RyanE | They sneak around silently. |
| 19:48.55 | jxqz | anything else? |
| 19:48.58 | RyanE | lots of sneaking. |
| 19:49.14 | jxqz | sneaking code in |
| 19:49.16 | jxqz | and out |
| 19:49.22 | jxqz | sneaking code around |
| 19:49.24 | RyanE | yeah, great idea. |
| 19:49.25 | RyanE | :| |
| 19:52.05 | Oberon4278 | Ninjas flip out and kill people, guys. |
| 19:52.36 | RyanE | so we're really talking about co-workers who sneak code into the codebase without warning, and eventually "go postal". |
| 19:52.39 | RyanE | perfect. |
| 19:53.05 | jxqz | taking the code all the way to the post office |
| 19:53.19 | jxqz | what if some spills out on the way? |
| 19:54.08 | Oberon4278 | http://www.realultimatepower.net/ninja/ninja2.htm |
| 19:54.09 | Oberon4278 | There. |
| 19:54.13 | Oberon4278 | All you need to know about ninjas. |
| 19:58.47 | *** join/#utah smcquay (~smcquay@17.226.13.44) |
| 20:23.28 | *** join/#utah emcnabb (~emcnabb@152.16.191.33) |
| 20:23.29 | *** mode/#utah [+v emcnabb] by ChanServ |
| 20:37.10 | *** join/#utah tristanbob (~quassel@2001:1948:212:1010:1daa:ea41:db2a:6111) |
| 20:49.29 | *** join/#utah smcquay (~smcquay@17.226.13.44) |
| 21:00.11 | *** join/#utah emcnabb (~emcnabb@152.16.191.33) |
| 21:00.11 | *** mode/#utah [+v emcnabb] by ChanServ |
| 21:13.30 | *** join/#utah pdayton (~pjdayton@63.251.sfcn.org) |
| 21:25.37 | *** join/#utah youtah (~youtah@199.192.165.212) |
| 21:52.52 | *** join/#utah youtah (~youtah@69-195-66-178.unifiedlayer.com) |
| 21:59.50 | *** join/#utah utlemming (~ben@97-118-224-106.hlrn.qwest.net) |
| 21:59.50 | *** join/#utah utlemming (~ben@ubuntu/member/utlemming) |
| 22:43.37 | *** join/#utah pdayton (~pjdayton@63.251.sfcn.org) |
| 22:43.53 | *** join/#utah emcnabb (~emcnabb@152.16.191.33) |
| 22:43.53 | *** mode/#utah [+v emcnabb] by ChanServ |
| 22:58.36 | *** join/#utah fugue88 (~dsowen@fugue88.ws) |
| 23:01.15 | *** join/#utah emcnabb (~emcnabb@152.16.191.33) |
| 23:01.15 | *** mode/#utah [+v emcnabb] by ChanServ |
| 23:03.29 | *** join/#utah fugue88 (~dsowen@fugue88.ws) |
| 23:34.47 | *** part/#utah Oberon4278 (~dcrooksto@38.112.155.163) |
| 23:43.23 | *** join/#utah davido__ (~davido@c-50-168-241-168.hsd1.ut.comcast.net) |