| 03:03.37 | *** join/#utah tristanbob_h (~Tristan@65-130-226-43.slkc.qwest.net) |
| 04:52.01 | klys | ~slashdot |
| 05:21.07 | *** join/#utah Azitrex (~Azitrex@unaffiliated/azitrex) |
| 06:47.35 | *** join/#utah TodPunk (sid145942@gateway/web/irccloud.com/x-luxioslsudpieqwt) |
| 06:47.39 | *** join/#utah spangborn (sid186075@gateway/web/irccloud.com/x-mvyxpmestkqfurlx) |
| 06:48.05 | *** join/#utah Alekane_ (sid331868@gateway/web/irccloud.com/x-qhqbjhurvofgytjd) |
| 06:48.09 | *** join/#utah romanovic (sid355365@gateway/web/irccloud.com/x-qbabcjrezyvakmlv) |
| 06:48.26 | *** join/#utah k-bake (sid331735@gateway/web/irccloud.com/x-tkrkcljdbpbpouwf) |
| 10:51.57 | *** join/#utah emcnabb (~emcnabb@hs-nc-1fff767f54-29977-1.tingfiber.com) |
| 10:57.17 | *** join/#utah Azitrex_ (~Azitrex@unaffiliated/azitrex) |
| 11:01.19 | *** join/#utah Azitrex_ (~Azitrex@unaffiliated/azitrex) |
| 13:14.56 | *** join/#utah nagefire (~nagefire@66-219-236-161.dynamic.ip.veracitynetworks.com) |
| 13:48.10 | beelzebob | I'm looking for an open source config/parameter store. Something close to what I imagine aws parameter store is (haven't used it personally). I want version control, and an easy api to GET parameters from a hierarchy |
| 13:49.04 | beelzebob | e.g. curl https://host:port/servers/foo/application/db_name would return the value for db_name |
| 13:49.36 | beelzebob | PUTting values also a plus! path based access control a plus! Queries that return multiplie values likely a requirement |
| 13:49.55 | beelzebob | Surely something like this already exists, but my keywords are failing me. Any humans with some help or ideas? |
| 13:52.29 | beelzebob | consul is close, but I don't need service discovery, and I *do* need version control vault is also close, but these aren't secrets (and also need version control) |
| 13:54.04 | beelzebob | a git repository arranged with directories as paths, files as keys with their contents as values, is *very* close - but searching and PUTing are somewhat problematic, or at least require extra effort |
| 14:05.48 | *** join/#utah josephscott (~josephsco@74.114.153.251) |
| 14:58.12 | *** join/#utah braxtron (~braxtron@216.160.248.178) |
| 15:00.29 | *** join/#utah legreathambino (~braxtron@216.160.248.178) |
| 15:41.35 | *** join/#utah tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 15:46.46 | jlp_znc | catches up |
| 15:47.02 | jlp_znc | jfindlay: I use mosh all the time. |
| 15:49.39 | fozzmoo | moshes all the time |
| 16:12.55 | *** join/#utah onesluttychicken (~braxtron@216.160.248.178) |
| 16:16.10 | youtah | Heh |
| 16:16.27 | youtah | Personal space heaters work great for testing UPS's and you need a load |
| 16:22.04 | jfindlay | a space heater means that your building's HVAC is a fail |
| 16:24.28 | jfindlay | beelzebob: if you're using salt you could try sdb: https://docs.saltstack.com/en/latest/topics/sdb/index.html |
| 16:31.13 | asoc | beelzebob: https://github.com/breser/git2consul might also give you the history functionality with consul but it is another thing to have running. |
| 16:34.25 | asoc | You might also take a look at InfluxDB since it was built for time series data it should give you basic history functionality and has a http api that worked well for a simple project I did a while back. |
| 16:46.58 | youtah | jfindlay, not even close. It's called we have some people who want the temps set at 67F in the winter, and receptionists that want it 87F |
| 16:52.37 | beelzebob | jfindlay: not using salt, but thanks |
| 16:53.09 | jfindlay | youtah: you have people fail |
| 16:53.12 | beelzebob | asoc: thanks for the tip with git2consul, that looks very interesting... |
| 16:54.36 | asoc | NP I havenât tried it though so my suggestion comes with no warranty ;-) I just had seen a link to it on a big for consul asking for change history to be implemented. |
| 16:55.48 | jfindlay | with a config store, I think you need to be able to structure it yourself because everyone's needs are going to be so different |
| 16:56.07 | spangborn | youtah: did you see Kaskade is coming to SLC? |
| 16:56.18 | spangborn | https://twitter.com/SLCSounds/status/1130495363628978176 |
| 16:59.38 | youtah | I did not |
| 16:59.49 | youtah | it would be fun to see him again in concert |
| 17:10.23 | romanovic | why, mac os, why. why does mac os feel the need to resize every window down to ~1cm square every time I close and reopen the lid |
| 17:11.07 | romanovic | I suspect it has something to do with connecting and disconnecting my external display, but I've been doing that for 2 years and this problem only started recently |
| 17:22.02 | jlp_znc | romanovic: I have an app called Stay that puts my windows back where I told it to any time the geometry of my desktop changes. I think it was $10, but I bought it so long ago that I don't remember for sure. |
| 17:23.51 | jlp_znc | https://cordlessdog.com/stay/ |
| 17:24.19 | jlp_znc | guess it's $15 |
| 18:11.36 | romanovic | thanks jlp_znc, I'll check it out |
| 18:15.12 | *** join/#utah goozbach (~goozbach@fedora/goozbach) |
| 18:15.12 | *** mode/#utah [+v goozbach] by ChanServ |
| 18:16.52 | goozbach | mornin |
| 18:18.19 | jfindlay | goozbach: what config database would you use? |
| 18:19.25 | goozbach | config db for what? |
| 18:19.33 | goozbach | like storing configs for servers? |
| 18:20.39 | jfindlay | yeah, something like that |
| 18:22.43 | goozbach | ansible |
| 18:22.48 | goozbach | group_vars |
| 18:22.53 | goozbach | and ansible-vault for secrets |
| 18:23.38 | goozbach | HashiCorp Vault for secrets for apps |
| 18:24.00 | goozbach | depending on the architecture I'd use etcd maybe |
| 18:24.15 | goozbach | consul is a good option in some cases |
| 18:24.20 | goozbach | it really depenes |
| 18:24.29 | goozbach | *depends |
| 18:24.37 | joshh | have you looked at zookeeper? |
| 18:25.06 | goozbach | I've just finished up a big contract which taught about zookeeper/kafka et al |
| 18:25.08 | goozbach | so yup |
| 18:25.46 | goozbach | it's useful for a datastore but it's persinckity |
| 18:25.53 | goozbach | also a GIANT resource hog |
| 18:26.06 | goozbach | if you're looking for clustering data store, cassandra is an option |
| 18:26.13 | goozbach | still java tho |
| 18:26.47 | goozbach | 98% of things you don't need a fully n+1 fault-tolerant data source, you just need a postgres or mysql DB |
| 18:26.52 | goozbach | hell even sqlite |
| 18:27.12 | goozbach | I'd like to do a presentation on "Do you really need Kubernetes?" |
| 18:27.17 | goozbach | it's the new hotness |
| 18:27.36 | goozbach | but if you're a 3person shop with a small startup idea... |
| 18:28.47 | jfindlay | the latest in IT solutions powered by buzzword |
| 18:28.48 | goozbach | just run GCP container engine if you want some more control, or use something like openshift or heroku |
| 18:29.16 | goozbach | hell, just a single EC2 instances using ansible or something to deploy |
| 18:29.38 | goozbach | the problem isn't the tech most times it's the workflow |
| 18:29.58 | goozbach | gitlab runner which does a git checkout and restarts a webserver is sufficent to "move fast and break things" |
| 18:30.11 | jfindlay | but a database of some kind is better than a plain text file |
| 18:30.36 | goozbach | is it tho? :) |
| 18:30.44 | goozbach | really depends on what you're doing |
| 18:31.08 | jfindlay | it is if your text file has config data on everything for more than 10 systems |
| 18:31.44 | jfindlay | a text file runs out of scale at that point I think |
| 18:32.06 | goozbach | "We need machine learning!" -- "how many customers do you have right now?" -- "well 200, but..." -- "no you need cat /var/log/httpd/*.log piped to sed and awk" |
| 18:32.49 | goozbach | jfindlay: I've got a pretty rock solid ansible layout which has maybe 30 files (vars and playbooks combined) |
| 18:32.56 | goozbach | that has scaled to thousands of machines |
| 18:33.22 | goozbach | 20'ish of those playbooks are specific apps deploys/configs |
| 18:33.23 | jfindlay | well, git is technically a blockchain, so you get that for free |
| 18:33.43 | goozbach | store the changes in git, poof done |
| 18:33.58 | goozbach | if you have a lot of dynamic scaling issues, maybe terraform is better for you. |
| 18:34.18 | goozbach | again, small number of files, spins up nodes, ansible confiures them. |
| 18:34.30 | goozbach | backing this with anything other than flat files is silly |
| 18:34.42 | jfindlay | I like to use salt+salt-cloud for orchestration |
| 18:35.03 | goozbach | feel free to use a dynamic inventory (AWS/GCP/Digital Ocean) python scrpt to keep track of hosts using tags |
| 18:35.43 | goozbach | but the *CONFIGS* in flat files backed by git is good for everyone short of amafacegoogsoft |
| 18:39.10 | jfindlay | so you'd recommend flat files over vault/consul/zookeeper/ectd/etc? |
| 18:39.49 | goozbach | again it depends on what you're storing |
| 18:40.00 | goozbach | I'd say spin up vault for storing secrets almost always |
| 18:40.06 | jfindlay | well, yes |
| 18:40.09 | goozbach | but that's not "server config" |
| 18:40.13 | jfindlay | but for non secret data |
| 18:40.14 | goozbach | that's "app config" |
| 18:40.45 | goozbach | I'd use etcd for larger installs of app configs |
| 18:40.54 | jfindlay | is /etc/apache/apache.conf.d an app config or a server config? |
| 18:41.18 | jfindlay | what's a server config? Role and network data? |
| 18:41.37 | goozbach | anything that ships with OS is server config |
| 18:42.08 | jfindlay | but that all comes with the builtin OS setup |
| 18:42.27 | jfindlay | so why would you need to store it externally? |
| 18:42.29 | goozbach | the settings that tells apache where the mod_perl directory is server config |
| 18:42.51 | goozbach | somewhere in the middle between app and server config is things like database passwords |
| 18:42.52 | jfindlay | so localizing paths to the host OS? |
| 18:43.12 | goozbach | and things like api keys for third party APIs is 100% app config |
| 18:43.29 | jfindlay | why distinguish between app and server configs? |
| 18:43.45 | goozbach | so the access to them can be decided differently |
| 18:44.00 | jfindlay | user access or system access? |
| 18:44.02 | goozbach | my job in ops: spin up a server/platform so it can host an app |
| 18:44.18 | goozbach | your job in dev: configure said app to do what it's supposed to |
| 18:44.22 | jfindlay | so you don't have complete vertical ownership? |
| 18:44.43 | goozbach | ops historically does |
| 18:44.58 | goozbach | but I'm trying to get out of it |
| 18:45.21 | goozbach | if I have templates and playbooks that spin up your platform (and I manage them as an ops person) |
| 18:45.35 | goozbach | I can write tooling that allows you to do "self service" |
| 18:45.50 | goozbach | eg (I need a new rails app platform, here's the API to do that) |
| 18:46.09 | goozbach | my rails app needs a database |
| 18:46.30 | goozbach | ok, here's an ops tool/api that spins up a database and here's your username and password |
| 18:46.43 | goozbach | my rails app needs to connect to 3rd party |
| 18:46.53 | goozbach | ok here's a config database (vault) for your app |
| 18:47.05 | goozbach | the database password can exist in vault |
| 18:47.12 | goozbach | or can exist in my ops tool |
| 18:47.18 | goozbach | but the app needs it |
| 18:47.31 | goozbach | the app and user don't need the root password to the machine it's running on |
| 18:47.39 | goozbach | but that should be somewhere? no? |
| 18:48.00 | jfindlay | interesting |
| 18:48.02 | goozbach | that's where the ops datastore comes into play (I choose ansible cause it's what I know) |
| 18:48.22 | goozbach | if the app config is separate i can give devs access to change things |
| 18:48.50 | goozbach | ohh the 3rd party api password changed (here's two ways to deal with that) |
| 18:49.05 | goozbach | 1) put in a ticket/call/text/whatever an ops guy to change it |
| 18:49.07 | jfindlay | do you still provide the pipeline for the dev resources, like they use your automation platform to deliver new versions? |
| 18:49.29 | goozbach | 2) login with my creds to the app config store (which I have RBAC to just the subset of stuff of my app) and make the change |
| 18:49.39 | goozbach | I'd take 2 any day |
| 18:49.50 | goozbach | cause as an operator I want to get out of the app business |
| 18:49.53 | goozbach | yes |
| 18:50.10 | goozbach | I provide the gitlab pipeline, the hosted gitlab repo, and the tooling to do the above |
| 18:50.17 | goozbach | I get paged if any of that breaks |
| 18:50.26 | jfindlay | 1) is retrogressive even when that's how things were done |
| 18:50.36 | jfindlay | call the operator to talk to someone |
| 18:51.04 | goozbach | I shouldn't be gettign paged when hey, "this dev forgot a field that 3rd party api needs" |
| 18:51.05 | jfindlay | well, you've sold me. I'd buy your consulting service |
| 18:51.14 | goozbach | that should page the dev who broke the build |
| 18:51.43 | goozbach | my job as a consultant is to use the right tool for the job to make your business needs easier |
| 18:51.52 | jfindlay | someday I'll have my ops skills as tight |
| 18:51.53 | goozbach | not to do process/tech for process tech sake |
| 18:52.22 | goozbach | remove barriers, increase flow |
| 18:52.45 | goozbach | if you've not read _The Phoenix Project_ I'd suggest you do so *THIS WEEK* |
| 18:52.55 | goozbach | then _The Devops Handbook_ |
| 18:53.10 | goozbach | _Activator_ by Utah's own Brian Gilespee is another good one |
| 18:53.48 | beelzebob | I recently read The Phoenix Project |
| 18:54.06 | beelzebob | I just came away from it with a resentment for guys named 'Brent' |
| 18:54.15 | jfindlay | lol |
| 18:54.27 | jfindlay | maps to a dilbert character |
| 18:54.38 | goozbach | then read the bible: _The practice of systems and network adminstration_ (old testament), and _The practice of Cloud Systems Administration_ (new testament) |
| 18:55.21 | goozbach | there's a couple blog posts maybe even a small podcast miniseries in this dicussion |
| 18:55.33 | goozbach | it's been heavy on my mind lately |
| 18:55.34 | beelzebob | hey goozbach you left the other day before hearing me mention #fslc, a small lonely leftover of a bygone era |
| 18:55.44 | goozbach | yeah it's dead jim |
| 18:55.46 | goozbach | sadness |
| 18:55.53 | goozbach | i miss griznog and the gang |
| 18:56.21 | beelzebob | I'm talking about #fslc on freenode, not fslc.usu.edu |
| 18:56.41 | beelzebob | now featuring more sadness and loneliness |
| 18:57.15 | beelzebob | none of the personalities you know and love, and more of the empty spaces |
| 18:58.34 | beelzebob | I also hope to watch your presentation tomorrow night in some form or another, so hopefully fozzmoo can film |
| 19:00.49 | goozbach | you can come eat pizza and watch live! :P |
| 19:02.32 | goozbach | https://www.amazon.com/Thomas-A.-Limoncelli/e/B004J0QIVM/ref=dp_byline_cont_book_1 |
| 19:03.50 | beelzebob | it is a bit of a drive |
| 19:06.33 | beelzebob | (you may recall I have relocated to your old stomping grounds in GA) |
| 19:06.46 | beelzebob | (or may not, I dunno) |
| 19:09.46 | goozbach | oh right |
| 19:09.48 | goozbach | wherebouts? |
| 19:10.05 | beelzebob | Athens |
| 19:10.10 | goozbach | frontier has some good deals right now :) |
| 19:10.15 | goozbach | that is my old stomping grounds |
| 19:10.19 | beelzebob | indeed |
| 19:10.52 | beelzebob | I noticed your name on the chugalug presenters page some time back |
| 19:11.14 | goozbach | yup |
| 19:14.00 | beelzebob | I was asking about app config services before you showed up, if you have any input... |
| 19:14.36 | goozbach | it really depends on the platform, but I'd use vault as a first strike |
| 19:14.42 | goozbach | consul+vault is nice |
| 19:15.01 | beelzebob | I'm looking for something consul like (consul might be overkill) to serve out app config. The closest I've got is an appropriately laid out git repo served via http |
| 19:15.14 | beelzebob | asoc mentioned git2consol, which looks interesting |
| 19:15.32 | goozbach | etcd works, but it's a little lower level (plumbing not porcelean) |
| 19:15.53 | goozbach | you running kubernetes? |
| 19:15.54 | beelzebob | vault is also close, but I want version control, and these aren't secrets |
| 19:16.10 | goozbach | newer version of vault has secret history |
| 19:16.30 | goozbach | ansible :) |
| 19:16.35 | beelzebob | no, I'm running old school, apache vhosts + tomcat |
| 19:16.42 | goozbach | ohh then ansible all the way |
| 19:16.49 | goozbach | hands down |
| 19:17.00 | beelzebob | oddly enough, my predecessor has built a config framework using ansible |
| 19:17.11 | goozbach | if you have zero ansible infrastructure, I've got a good layout to use |
| 19:17.20 | beelzebob | but... this is kind of a different use case |
| 19:17.23 | goozbach | three big "gotchas" |
| 19:17.39 | goozbach | 1) use yaml inventory format |
| 19:18.00 | goozbach | 2) one "inv/" folder per environment (dev, pre-prod, prod) |
| 19:18.13 | goozbach | 3) symlinks to make "global" group_vars files work |
| 19:18.16 | joshh | for toggling configs without a redeploy/restart, using git plus either a config pusher to an update endpoint, or a git poller and updater in the app, is a perfectly fine pattern, when the config needs to be treated like source code |
| 19:18.33 | goozbach | yeah I'd get this working manually |
| 19:18.41 | goozbach | then I'd do a gitlab runner to apply automaticly |
| 19:18.57 | beelzebob | There are several different groups (humans and krom willing, machines) that need to occasionaly update parts of the config |
| 19:19.01 | joshh | if your app just needs to watch something and then update, i'd say consul or zk, but this does seem like a pretty small and simple use case here |
| 19:19.23 | joshh | ansible could do something like lay down new files and execute restarts or whatever really well |
| 19:19.31 | beelzebob | the app is not smart enough at this point to watch anything |
| 19:19.35 | goozbach | if humans are part of the mix, consul |
| 19:19.47 | goozbach | you get env_consul for "free" |
| 19:19.51 | goozbach | which can do the watchign for you |
| 19:20.02 | goozbach | it might be overkill, but it's still sounds like the right tool ATM |
| 19:20.55 | beelzebob | it may end up being that way |
| 19:21.21 | beelzebob | the main config for 100+ sites is stored in a tab delimeted file currently, which is difficult to get machines to edit properly |
| 19:21.50 | beelzebob | so I want to store these values in a heirachy that can be pushed/pulled more granularly |
| 19:22.10 | beelzebob | as well as store other related information |
| 19:22.25 | goozbach | a properly formated group_vars might be the right step |
| 19:22.43 | goozbach | but then again if you need to change values, setup consul as a datasource and ansible as a template engine |
| 19:22.51 | beelzebob | a possible consul snag (and I'm not a consul user) is that I'd like to be able to query this structure dynamically |
| 19:23.23 | beelzebob | to answer questions like, "how many values are using key X?" |
| 19:25.13 | beelzebob | apache kinto is similar in this way, but also way overblown |
| 20:16.15 | *** join/#utah josephscott (~josephsco@192.0.124.251) |
| 20:36.26 | youtah | throws goozbach into log rotation |
| 20:40.05 | goozbach | ~lart youtah |
| 20:40.05 | infobot | slaps youtah around with a large trout |
| 21:25.39 | *** join/#utah josephscott (~josephsco@74.114.153.251) |
| 21:45.22 | *** join/#utah tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 21:47.20 | youtah | flings old food truck food at goozbach |
| 22:24.35 | *** join/#utah friocorte (~goozbach@vpn.creditera.com) |
| 22:24.35 | *** join/#utah friocorte (~goozbach@fedora/goozbach) |
| 22:24.35 | *** mode/#utah [+v friocorte] by ChanServ |
| 22:29.05 | jfindlay | one of the worst things I have to deal with is vendor softwares |
| 22:29.11 | jfindlay | they are usually a dung heap |
| 22:29.36 | jfindlay | "enterprise" |
| 23:18.36 | jfindlay | is there a reason I can ping hosts on the local network but nothing beyond? |
| 23:18.52 | jfindlay | this CentOS 6 VM is being stubborn about the network |
| 23:19.23 | jfindlay | It's not configuring correctly either from the dhcp server or with static configs |
| 23:37.58 | jfindlay | of course it was a hidden vsphere bug |
| 23:38.03 | jfindlay | lamers |