| 00:52.39 | shalkie | harleypig: I applaud your bravery for making such a bold announcement. :-P |
| 00:57.36 | *** join/#utah davido_ (~davido@173-165-131-122-utah.hfc.comcastbusiness.net) |
| 01:00.13 | harleypig | oi=08435n |
| 01:12.42 | *** join/#utah kendsnyder (~Adium@67-2-115-168.slkc.qwest.net) |
| 01:27.45 | *** join/#utah OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 01:36.43 | *** join/#utah yukaia-work (~yukaia@198.183.219.9) |
| 03:08.11 | *** join/#utah olli (~olli@c-71-195-232-74.hsd1.ut.comcast.net) |
| 03:48.15 | *** join/#utah gr3yr0n1n (~gr3yr0n1n@c-174-52-46-96.hsd1.ut.comcast.net) |
| 03:48.26 | *** part/#utah gr3yr0n1n (~gr3yr0n1n@c-174-52-46-96.hsd1.ut.comcast.net) |
| 03:53.28 | jlp_vnc | I think maybe harleypig's cat got at the computer. It has now ordered 5000 catnip mice from amazon to be delivered overnight. Wheeee! |
| 03:54.49 | harleypig | kdfg=9je |
| 04:12.19 | *** join/#utah davido_ (~davido@c-50-168-241-168.hsd1.ut.comcast.net) |
| 04:16.14 | *** join/#utah yukaia_work (~yukaia@198.183.219.9) |
| 04:49.19 | *** join/#utah jalbretsen (~jalbretse@70-58-32-202.slkc.qwest.net) |
| 05:05.32 | *** join/#utah bonez2046 (~LiveWell@173-14-239-230-Utah.hfc.comcastbusiness.net) |
| 05:09.10 | bonez2046 | how can I copy text from a page several commands for ./configure xxxxxxxx to my command prompt? I copy/highlight and then the mouse won't paste it to the prompt to run it |
| 05:18.12 | programmerq | bonez2046: shift+insert, middle click? |
| 05:18.39 | fadein | programmerq: he gone |
| 05:18.48 | fadein | I was going to suggest `xclip -o` |
| 05:18.50 | programmerq | yeah, I just missed him. |
| 05:24.01 | supaplex | oink |
| 05:44.28 | *** join/#utah OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 06:19.32 | *** join/#utah jalbretsen (~jalbretse@70-58-32-202.slkc.qwest.net) |
| 06:54.22 | *** join/#utah OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 07:19.02 | *** join/#utah spangborn (~spangborn@216.51.232.249) |
| 07:33.03 | *** join/#utah siel (~siel@unaffiliated/motley) |
| 07:36.33 | *** join/#utah spangborn (~spangborn@216.51.232.249) |
| 08:55.08 | *** join/#utah Greagle (~greagle@c-76-23-60-71.hsd1.ut.comcast.net) |
| 09:20.54 | *** join/#utah siel (~siel@unaffiliated/motley) |
| 12:21.26 | *** join/#utah olli (~olli@c-71-195-232-74.hsd1.ut.comcast.net) |
| 12:34.21 | *** join/#utah neybar_work (~quassel@74-220-196-120.unifiedlayer.com) |
| 12:52.30 | *** join/#utah jima (jima@laughton.us) |
| 13:11.28 | *** join/#utah siel (~siel@unaffiliated/motley) |
| 13:30.07 | *** join/#utah resplin (~richard@esplins.org) |
| 13:53.27 | *** join/#utah T4rku5 (~74rku5@c-174-52-141-215.hsd1.ut.comcast.net) |
| 14:03.16 | *** join/#utah Migs (~Migs@205.122.50.64) |
| 14:03.33 | *** join/#utah T4rku5 (~74rku5@unaffiliated/shunkydave) |
| 14:08.47 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 14:11.59 | *** join/#utah smcquay (~smcquay@17.226.13.44) |
| 14:48.51 | *** join/#utah Tristan_ (~Tristan@c-50-186-103-12.hsd1.ut.comcast.net) |
| 15:15.30 | jxqz | basepi: sudo make me a sandwich |
| 15:16.16 | basepi | Luckily, I removed you from the sudoers file yesterday. |
| 15:16.40 | basepi | permission DENIED |
| 15:16.52 | jxqz | eightyeight: `rm -fr /*` should get around '--no-preserve-root' because you're removing 2nd level dirs, not the top dir, right? |
| 15:17.04 | jxqz | unlike just `rm -fr /` |
| 15:17.21 | jxqz | maybe you should try it just to verify |
| 15:17.56 | jxqz | on a busy, critical server so that the results cannot be misconstrued |
| 15:18.41 | jxqz | basepi: darn |
| 15:18.59 | jxqz | basepi: you don't need to see my identification |
| 15:24.34 | *** join/#utah kendsnyder (~Adium@67.214.245.39) |
| 15:27.41 | *** join/#utah TimRiker (~TimRiker@bzflag/projectlead/TimRiker) |
| 15:27.41 | *** mode/#utah [+o TimRiker] by ChanServ |
| 15:31.34 | *** join/#utah pdayton (~pjdayton@67.21.63.146) |
| 15:39.55 | *** join/#utah tristanbob__l (~tristanbo@137.190.208.10) |
| 15:40.38 | eightyeight | jxqz: tias |
| 15:41.41 | *** join/#utah d3c4f (~d3c4f@198.183.219.10) |
| 15:42.59 | jxqz | eightyeight: utias |
| 15:50.18 | *** join/#utah SunSparc (~jonathan@162.219.43.50) |
| 15:56.55 | *** join/#utah pdayton (~pjdayton@67.21.63.146) |
| 16:00.29 | *** join/#utah emcnabb (~emcnabb@cpe-107-015-189-013.nc.res.rr.com) |
| 16:00.29 | *** mode/#utah [+v emcnabb] by ChanServ |
| 16:04.17 | *** join/#utah tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 16:13.55 | *** join/#utah manithree (~blr@li4-76.members.linode.com) |
| 16:22.22 | *** join/#utah manithree (~blr@li4-76.members.linode.com) |
| 16:39.06 | *** join/#utah d3c4f (~d3c4f@198.183.219.10) |
| 16:39.32 | *** join/#utah yukaia-work (~yukaia@198.183.219.9) |
| 16:45.25 | *** join/#utah josephscott (~josephsco@192.0.80.201) |
| 16:49.31 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 16:53.49 | *** join/#utah opapo (~quassel@204.113.97.234) |
| 16:55.23 | *** join/#utah shalkie (shalkie@theadamsresidence.net) |
| 16:55.35 | shalkie | jxqz: on a RHEL/Cent 6 box I can definitively tell you that you can run just rm -fr /* |
| 17:02.47 | fungus | what's more fun is to: dd if=/dev/zero of=/dev/sda |
| 17:03.12 | fungus | much more spectacular failures |
| 17:04.18 | jima | fungus: i won't deny having done that before...after making dang sure i was on the right server. |
| 17:05.19 | fungus | lol |
| 17:38.09 | *** join/#utah undertakingyou (~undertaki@neutroninteractive-xmission.xmission.com) |
| 17:42.04 | shalkie | fungus: You don't want to use /dev/urandom or even shred? Seems like that could be way more interesting. :) |
| 17:58.39 | *** join/#utah tristanbob__l (~tristanbo@137.190.208.10) |
| 18:00.28 | *** join/#utah opapo (~quassel@204.113.97.239) |
| 18:37.53 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 19:26.00 | *** join/#utah neonotanderson (~neonotand@c-76-23-5-72.hsd1.ut.comcast.net) |
| 20:38.54 | *** join/#utah OldCoder (~OldCoder@unaffiliated/oldcoder) |
| 20:46.19 | rsimpkins | Is there ANYONE in utah who runs a Hadoop cluster(s) containing 1PB+? |
| 20:47.15 | jxqz | rsimpkins: the NSA claims to have 1 YiB |
| 20:47.27 | jxqz | you should call them up |
| 20:47.36 | rsimpkins | I need to find someone who has 10+ years of general ops experience along with 2+ years Hadoop experience. |
| 20:48.09 | jxqz | yet another job description for which no one who ever lived qualifies |
| 20:48.23 | rsimpkins | Why would no one ever live qualify? |
| 20:48.37 | rsimpkins | Because they have given up on life after 1 year with Hadoop? |
| 20:48.39 | jxqz | ok, maybe 5 qualified people |
| 20:48.58 | rsimpkins | Right. 5 qualified people. I have a position for one of those five people, and the salary to match. |
| 20:49.25 | jxqz | I'm just guessing that because hadoop is still a relatively new buzzwordy util that most people don't have more than a few weeks experience |
| 20:49.57 | rsimpkins | We've talked with many candidates with 2+ years Hadoop experience, but none of them (so far) that also have the combined ops history. |
| 20:50.15 | rsimpkins | We may just have to train them. |
| 20:50.22 | rsimpkins | One direction or the other. |
| 20:50.36 | jxqz | the invisible hand of the market strikes again |
| 20:52.22 | joshh | rsimpkins: our hadoop guys do all the non-basic ops for it, but they don't really touch any non-data stuff like generalists might |
| 20:53.23 | joshh | they do have lots of general ops/sysadmin experience though |
| 20:53.48 | joshh | mostly older guys who got into "big data" within the last few years |
| 20:54.04 | rsimpkins | joshh: Do any of them want to come work in a big glass building in Lehi? |
| 20:54.16 | joshh | doubtful |
| 20:54.45 | rsimpkins | But we have... a rock wall! I mean - isn't that a perk a very experienced ops guy would want to use? |
| 20:55.10 | jxqz | should have been an adobe wall |
| 20:55.13 | joshh | heh |
| 20:55.26 | rsimpkins | No Adobe at Adobe. |
| 20:55.31 | jlp_vnc | heh, experienced ops guys are too old to climb rock walls |
| 20:55.52 | rsimpkins | We have... a treadmill too. Several of them in fact. And a yoga instructor. |
| 20:56.01 | jlp_vnc | how about free lunch? |
| 20:56.02 | rsimpkins | What ops guy wouldn't want to take advantage of THAT!? :) |
| 20:56.05 | jlp_vnc | and snacks? |
| 20:56.18 | jlp_vnc | that's how you're going to attract the experienced ops guy |
| 20:56.35 | joshh | they take advantage of the wine, healthcare, domestic partner benefits, etc |
| 20:56.39 | rsimpkins | snacks, yes - but they are lame. free lunch? No, sadly. But I will buy them lunch if they want to have lunch with me. |
| 20:56.50 | joshh | not so much the massages and juice catering |
| 20:57.31 | joshh | i only take advantage of the kegs, which im about to do, since it's friday and almost 2 |
| 20:57.42 | jlp_vnc | I may know one or two engineers who might be interested |
| 20:57.43 | rsimpkins | 'free lunch with Ryan' - that might drive people away. Don't tell anyone that. |
| 20:58.09 | jlp_vnc | which is more valuable to you, the 2 years of hadoop experience or the 10+ years of devops experience? |
| 20:58.21 | rsimpkins | jlp_vnc: If they are interested I can help them shortcut the usual corporate BS and talk directly with the hiring manager. Just let me know. :) |
| 20:58.30 | jlp_vnc | (if you can only have one or the other) |
| 20:58.56 | rsimpkins | Now that we have fully pillaged Bluehost of all its talent, we are favoring the Hadoop guy. |
| 20:59.12 | jlp_vnc | I see. okay, let me ask around. |
| 20:59.22 | jlp_vnc | is there an official posting for this yet? |
| 20:59.26 | rsimpkins | Yes. |
| 20:59.35 | rsimpkins | Somehwere... in the cloud. :) |
| 21:00.03 | jlp_vnc | I just PMed you my e-mail. send it there when you find it and I'll pass it on to some people |
| 21:00.12 | rsimpkins | Will do. |
| 21:01.22 | jlp_vnc | prepares to snarf a strawberry cheesecake sandwich (dessert at lunch today) |
| 21:09.48 | SunSparc | I need a linux guru who knows networking aspects of linux down to the kernal level. My company would be willing to pay for some consulting about optimizing network activity of an application we are building. |
| 21:12.00 | jlp_vnc | what kind of problem are you seeing? |
| 21:14.27 | SunSparc | One of the quirks we're seeing is that we can't get our server to accept() connections quickly |
| 21:14.43 | SunSparc | sometimes it'll accept (linux accept call) about thousands in few milliseconds |
| 21:14.58 | SunSparc | then, it'll take multiple seconds to accept the next batch of tcp connections |
| 21:15.15 | joshh | and it's not full? |
| 21:15.26 | SunSparc | we've looked at various sysctl.conf settings as well as soconnmax and ulimit |
| 21:15.57 | SunSparc | off the top of my head, I don't know the command to check the tcp accept queue length |
| 21:16.09 | SunSparc | also, right now the test server only receives the connection and then closes it |
| 21:16.33 | SunSparc | all of this is done using epoll on and multiple server threads so it should be accepting tens of thousands of new connections per second |
| 21:16.56 | SunSparc | it's a Core i7 1270v3 on dedicated hardware (benchmark client and test server) with 16 GB of RAM |
| 21:17.04 | SunSparc | and a gigabit connection |
| 21:17.12 | SunSparc | the link shows it's only using a few Kb/s |
| 21:17.24 | SunSparc | and the CPU is sitting almost idle |
| 21:17.43 | SunSparc | I've turned of irqbalance and pointed network interrupts to cpu0 |
| 21:18.20 | SunSparc | most of what I'm fighting now is knowing exactly where the bottlenecks are |
| 21:18.26 | SunSparc | and figuring out a way to track them down |
| 21:18.29 | joshh | are the connection attempts constant? |
| 21:18.46 | SunSparc | right now they're more bursty |
| 21:18.59 | SunSparc | so I'll try a few thousand and wait for them to all connection and measure the time |
| 21:19.06 | SunSparc | once that round is done, I'll try it again |
| 21:19.21 | SunSparc | I'd like to get something where i have a long-running system to see where we can saturate things |
| 21:19.26 | SunSparc | long-running benchmark, that is |
| 21:19.54 | jlp_vnc | I'm assuming you have multiple client machines making the connections |
| 21:19.57 | joshh | is the client plugged into the server directly or through a stateful device/firewall? how many new sessions at a time? |
| 21:20.15 | SunSparc | right now, I'm using a single client for testing, so it could be a bottleneck |
| 21:20.24 | SunSparc | the client is plugged into a switch |
| 21:20.44 | SunSparc | yes, the switch could be the bottleneck, but not likely. |
| 21:20.49 | joshh | no firewalls though? |
| 21:20.54 | SunSparc | nope |
| 21:21.01 | SunSparc | ufw is disabled (Ubuntu) and iptables are clear |
| 21:21.21 | joshh | sometimes they can only create and allow so many new flows per second through them |
| 21:21.22 | SunSparc | the iptables are configured to let everything through without filtering |
| 21:21.55 | *** join/#utah Greagle (~greagle@c-76-23-60-71.hsd1.ut.comcast.net) |
| 21:24.09 | joshh | how many thousands of connections? |
| 21:28.01 | jxqz | what about your nic? |
| 21:28.14 | SunSparc | great questions. It's an Intel Gigabit |
| 21:28.21 | SunSparc | I don't have the model number in front of me |
| 21:28.25 | jxqz | well supported? |
| 21:28.38 | jxqz | you could check out kernel driver options |
| 21:28.39 | SunSparc | basically I'm trying to get about 4K new TCP connections per second |
| 21:30.08 | joshh | does the client get an error? like econnrefused? |
| 21:30.38 | joshh | or it works but just takes a while |
| 21:31.01 | SunSparc | Good question. Not sure. Seems like it just takes a while. |
| 21:31.18 | SunSparc | Thanks guys, jlp is going to take a closer look for us. |
| 21:31.27 | SunSparc | I appreciate the input. |
| 21:31.32 | joshh | heh nice |
| 21:31.41 | SunSparc | :) |
| 21:32.59 | *** join/#utah macnewbold (~macnewbol@inferno.apxalarm.com) |
| 21:36.41 | *** join/#utah mrpull (~jpollmann@216.51.56.98) |
| 21:40.35 | *** join/#utah Tristan_ (~Tristan@c-50-186-103-12.hsd1.ut.comcast.net) |
| 22:00.08 | levi | Pretty much all intel gigabit nics are well-supported. |
| 22:03.57 | joshh | sometimes i have needed to disable TSO and whatnot on them though, to fix bizarre issues |
| 22:04.24 | *** join/#utah smcquay (~smcquay@17.244.44.51) |
| 22:06.34 | *** join/#utah smcquay_ (~smcquay@17.244.44.51) |
| 22:07.47 | joshh | in fact, you might want to try that SunSparc |
| 22:08.51 | SunSparc | makes note of joshh's comment and googles him some TSO... |
| 22:09.52 | joshh | you do it with the ethtool program |
| 22:10.04 | SunSparc | Ah, thank you. |
| 22:10.40 | joshh | http://blogs.citrix.com/2011/04/14/using-ethtool-settings-to-address-network-performance-issues/ |
| 22:10.53 | SunSparc | My google found the same. :) |
| 22:11.11 | SunSparc | Looks like good stuff. Thanks :) |
| 22:12.15 | joshh | try it without tso and gso |
| 22:13.01 | levi | Possibly related to interrupt throttling, too? |
| 22:13.21 | SunSparc | Wow, that is a robust tool. I am adding that one to my toolbelt. |
| 22:14.32 | SunSparc | levi: Definitely something we will be looking at. |
| 22:14.42 | SunSparc | Good tip |
| 22:15.50 | levi | It's in "dynamic conservative" mode by default; if your traffic is not queued to a low-latency queue (which it probably isn't) you'll be stuck at 4000/s |
| 22:16.23 | SunSparc | 4000 connections? |
| 22:16.28 | levi | Interrupts |
| 22:16.34 | SunSparc | Ok |
| 22:16.47 | levi | The ethernet driver knows nothing about TCP connections, generally. |
| 22:17.09 | SunSparc | Different level, right. |
| 22:18.00 | joshh | but tso is for tcp |
| 22:18.17 | jxqz | yeah, ethtool is awesome |
| 22:18.27 | jxqz | just like everything else that comes from kernel devs |
| 22:18.32 | jxqz | linux, git, etc. |
| 22:18.54 | joshh | it amazes how fast git replaced everything |
| 22:18.57 | SunSparc | All hail the kernel devs. |
| 22:19.25 | joshh | almost nothing new doesn't use it these days |
| 22:19.33 | joshh | of course github is probably a factor also |
| 22:19.44 | levi | I'm not quite sure I'd hail all of linux, git, etc as unreservedly awesome. |
| 22:19.54 | joshh | it makes stuff like sourceforge seem like something from the vaccuum tubes era |
| 22:20.18 | jxqz | levi: oh sure, don't unconditionally worship kernel devs |
| 22:20.43 | levi | I think github was a pretty big factor in the move of everything open source to git these days. |
| 22:20.45 | joshh | linux simply sucks the least overall |
| 22:21.07 | levi | Linux sucks the least in some dimensions. It sucks quite a lot in other dimensions. |
| 22:21.14 | jxqz | is github itself even open source? |
| 22:21.35 | levi | Nope, all the web infrastructure and whatnot for github is closed source. |
| 22:22.06 | jxqz | git by itself will get you an amazing amount of awesomeness |
| 22:22.24 | joshh | we use github enterprise internally, and it's definitely locked down |
| 22:22.50 | jxqz | generally I've found github to be a spiffy front end to the well designed git backend |
| 22:23.05 | levi | It's not exactly rocket science stuff, though. It was just good enough and easy/cheap enough when it came out, and developed well over time. |
| 22:23.11 | joshh | github also sucks the least |
| 22:23.44 | joshh | i wish it was easier to find gists and taht i could see other admins repos as a site admin, but that's about it |
| 22:23.47 | jxqz | at my last place we used assembla, which wasn't bad, but not as special as github |
| 22:24.56 | jxqz | eventually I'd like to move most if not all of my stuff to my own private repo |
| 22:25.05 | jxqz | since I don't need a web gui to use git |
| 22:25.30 | levi | How did the cool kids get to be the cool kids? They're sure not always the best kids when measured by metrics other than popularity. |
| 22:25.46 | jxqz | levi: chaos |
| 22:25.54 | jxqz | that'w what makes it so 'fun' |
| 22:26.34 | levi | jxqz: gitolite is handy for managing a bunch of repos in a command-line manner. |
| 22:26.35 | joshh | its like mcdonalds being the most popular burger joint, or budweiser being the most popular beer |
| 22:26.55 | joshh | except github is actually good |
| 22:27.13 | jxqz | when your sample space is so large or time scale so short, noise, rather than metrics tend to select the winners |
| 22:27.26 | levi | McDonalds is "good" in that it is consistent. You always know exactly what you're going to get. |
| 22:27.54 | joshh | mcdonalds isn't considered a cool kid though... just.. i don't even know |
| 22:28.21 | joshh | bad example perhaps |
| 22:28.26 | levi | I'm not talking about hipster-cool, I'm talking about popularity. |
| 22:29.28 | levi | Anyway, it was not meant to be a very deep analogy. |
| 22:30.25 | levi | So, the Atlassian set of tools aren't terrible, either, if you want something more "featureful" than github. |
| 22:31.23 | levi | And there's always gerrit, too. |
| 22:32.17 | joshh | we use gerrit and github, but only jira from atlassian |
| 22:32.27 | joshh | well, and hipchat, as of the other day |
| 22:33.54 | levi | That's quite the mishmash. |
| 22:34.40 | joshh | our flow is push to gerrit, after a +2 you can merge it to github, and jira is just for all our tickets, some of which gerrit comments/resolves |
| 22:35.16 | joshh | if there's a jira in a commit msg |
| 22:36.18 | jxqz | have you tried remine or openproject? |
| 22:36.25 | *** part/#utah Nemus (~Nemus@unaffiliated/nemus) |
| 22:37.32 | joshh | not me |
| 22:37.59 | joshh | but im actually mostly an ops/network guy so i don't normally decide those things |
| 22:38.14 | jxqz | anything but bugzilla |
| 22:46.05 | *** join/#utah Nemus (~Nemus@unaffiliated/nemus) |
| 22:47.13 | jxqz | in trying to apply a coupon code at newegg, they won't let me use it unless I sign up for their marketing spam |
| 22:47.23 | *** part/#utah manithree (~blr@li4-76.members.linode.com) |
| 22:47.27 | jxqz | not a big deal, I guess, I can unsubscribe later, right? |
| 22:47.43 | joshh | well, you also have to wait for jenkins to comment in gerrit that the tests passed |
| 22:54.58 | SunSparc | jxqz, does your email system allow plus addresses? |
| 22:55.17 | SunSparc | like: jxqz+newegg@domain.com |
| 23:03.41 | SunSparc | Have a great weekend all! |
| 23:18.38 | *** join/#utah neybar_work (~quassel@74-220-196-120.unifiedlayer.com) |
| 23:24.34 | *** join/#utah Jayce (~jayce@74-220-196-98.unifiedlayer.com) |
| 23:48.45 | *** join/#utah kendsnyder (~Adium@67-2-115-168.slkc.qwest.net) |
| 23:55.00 | *** join/#utah janderson (~janderson@50-202-143-210-static.hfc.comcastbusiness.net) |