| 01:48.19 | rsimpkins | youtah: Proof I was on IRC. Only been... years? |
| 03:27.29 | *** join/#utah tiwula (~lane@174-23-85-23.slkc.qwest.net) |
| 09:55.30 | klys | ~slashdot |
| 10:06.26 | klys | https://coinmarketcap.com/coins/ |
| 15:51.53 | *** join/#utah tiwula (~lane@70.89.246.241-Busname-UT.hfc.comcastbusiness.net) |
| 17:03.58 | youtah | WOO HOO |
| 17:04.12 | youtah | wait, dangit, he's gone |
| 17:40.11 | *** join/#utah keldwud (ae34be84@gateway/web/freenode/ip.174.52.190.132) |
| 17:49.27 | keldwud | I'm looking for some resources for best practices for building a staging environment and also anecdotal stories for how others have solved the deployment pipeline in their own environments. I'd really like to do an A/B type deal where staging becomes production and vice versa as all the dev work is piped into the staging environment and once it's production ready, switch my pointers such as domain names to the staging, then making the |
| 17:49.38 | keldwud | but I'm sure there are other ways |
| 17:50.23 | programmerq | DNS switch is unreliable since it depends on well-behaved clients. better to use load balancers in front of your A/B and switch the LB's upstream that way. |
| 17:50.49 | keldwud | oh perfect, I knew I was missing an important layer, I just couldn't quite put my finger on it |
| 17:51.10 | spangborn | Sounds interesting |
| 17:51.11 | programmerq | I would avoid calling it staging/production-- call it A/B or red/green or green/blue. To me, staging is a reference copy that lets you repro production issues. This, of course, isn't true for all staging environments |
| 17:51.28 | programmerq | but blue/green is more clear if you're doing that type of deploy (IMO, anyway) |
| 17:51.52 | keldwud | and A/B is generally used with website devs for UI stuff |
| 17:51.59 | keldwud | I also like blue/green |
| 17:52.19 | keldwud | what are the advantages of doing dev/staging/prod versus blue/green? |
| 17:52.36 | keldwud | is there any reason why I'd want to *not* do blue/green, basically |
| 17:53.08 | programmerq | I mean, you can "solve" any disadvantages by adding more environments. |
| 17:53.29 | programmerq | so maybe you have blue and green, which are both "prod", but you also have a staging blue/green |
| 17:53.51 | programmerq | it's a tradeoff between how much you need out of your environments and how much work you are able to put into them. |
| 17:54.29 | keldwud | but I'm understanding correctly that if my blue is live then my green is effectively the staging portion of my pipeline, right? |
| 17:54.33 | programmerq | I was on a project where I had something like 65 environments I managed. it was mostly automatable, so it was doable. it gave the various teams plenty of flexibility. |
| 17:55.31 | programmerq | yes and no, depending on what exactly your process is. if your devs are literally going in and mucking with it live, then it no longer serves as a reference environment for production. |
| 17:55.47 | programmerq | so if you run into some critical error on production, you may not be able to use that staging to repro |
| 17:55.55 | programmerq | and the consequences of that may vary wildly |
| 17:56.09 | programmerq | again, depending on what other environments you have and how likely something like that may happen. |
| 17:56.21 | keldwud | only have dev and prod currently |
| 17:56.22 | programmerq | not everyone needs a "prod reference" type environment |
| 17:56.57 | programmerq | if it's all scriptable and you can bring up a fresh environment in 12 minutes to serve as an on-demand reference, then it's probably no big deal. |
| 17:57.15 | keldwud | I'll definitely need a prod-reference type environment for my QA contractors |
| 17:57.41 | programmerq | prod-reference can be handy for things like security scans too |
| 17:58.03 | keldwud | true, I need that too since I'm also responsible for infosec now |
| 17:58.07 | programmerq | I'm in a happy place if I have a script that will take a db dump of prod and then anonymize it so it can be used without PII of the users. |
| 17:58.24 | keldwud | I don't have to worry about PII fortunately |
| 17:58.33 | programmerq | do you have email addresses? |
| 17:58.42 | keldwud | ahh, yeah |
| 17:58.45 | keldwud | good call |
| 17:59.39 | keldwud | ok so blue/green should work for me, I can dump live db into the non-live and let them run their QA on the new code |
| 17:59.40 | programmerq | adding a db migration should include reviewing the sanitizing script and auditing its results against the actual prod db. |
| 18:00.15 | programmerq | developers might even end up importing the sanitized db to their dev envs on their workstations. |
| 18:00.35 | keldwud | so the non-live should be separated from live like dev and prod are separated? |
| 18:00.44 | *** join/#utah SunSparc (~jonathan@45.56.57.85) |
| 18:00.54 | keldwud | I was operating under the assumption that blue/green would both be prod-like environments and could be treated the same |
| 18:01.58 | keldwud | but whichever solution I use, I'm definitely exporting prod data into the staging or non-live environment |
| 18:02.51 | keldwud | I had *thought* maybe I could have both blue and green pointing to the same DB with one not having write permissions |
| 18:03.25 | keldwud | and then when I'm ready to switch the non-live to live, I could turn on write permissions on one and then turn them off on the other |
| 18:03.36 | keldwud | is that an incorrect way to think about this? |
| 18:03.47 | keldwud | or is that a path worth investigating? |
| 18:05.00 | programmerq | in a blue/green, you may end up having both the blue and the green using the same exact DB |
| 18:05.24 | programmerq | the scrubbed prod db copy is more useful for non-prod environments-- you can check it in to VCS and pass it around. your external contractors will be able to get a copy of it |
| 18:05.57 | programmerq | another approach may end up being to have blue and green use a separate db, but one is the slave of another. |
| 18:06.08 | programmerq | you can flip the master/slave role as part of the switchover. |
| 18:06.22 | programmerq | it's a tricky question to answer. |
| 18:06.30 | programmerq | what sort of DB migrations do you generally have during a deploy? |
| 18:06.34 | keldwud | yeah, that's why I'm finding different perspectives |
| 18:06.51 | programmerq | are they safe to apply to a read slave? sane defaults in new columns? |
| 18:06.59 | keldwud | I don't have any DB migrations during deploy that I'm aware of |
| 18:07.12 | programmerq | what happens when you do have a dev that needs a change to the schema? |
| 18:07.18 | programmerq | it's a relatively common problem |
| 18:07.32 | keldwud | oh yeah, the schemas do get changed often. I see the chatter when it happens |
| 18:07.45 | programmerq | that usually gets rolled into a deploy scenario |
| 18:08.06 | programmerq | for large tables, than can take a very long time too |
| 18:08.54 | keldwud | ok so that's something I'll have to dig into before I can plan properly looks like. [ ] learn how my DB schemas are currently deployed |
| 18:09.04 | keldwud | makin my checklists :) |
| 18:11.41 | keldwud | programmerq: is all of your experience from trial and error or did you have any learning resources you used when obtaining your current knowledge? |
| 18:11.47 | programmerq | If you will be at the space on Friday around 2-4pm I can talk about this sort of stuff more in person. |
| 18:12.10 | keldwud | yeah, I'll take a work-from-home day this Friday |
| 18:12.13 | keldwud | I appreciate it |
| 18:12.13 | programmerq | keldwud⸠I've done a ton of deploys on a bunch of projects. Django and Rails for many of them. |
| 18:12.29 | keldwud | mine is docker, python and postgres |
| 18:12.38 | programmerq | db migrations turned into the bane of my existence for a while |
| 18:13.04 | programmerq | 5 million rows, and the devs want to write a migration that loads each record individually and do ruby or python logic on it and then move onto the next. |
| 18:13.07 | keldwud | yeah, the DB portion was definitely looking a bit intimidating for me |
| 18:13.20 | keldwud | my dbs aren't super large, though, so that's good |
| 18:13.30 | programmerq | then all managers everywhere are like HEY WHY IS THE DEPLOY TAKING SO LONG |
| 18:13.49 | programmerq | and I was like "same reason as last time, devs wrote a shitty db migration and didn't talk to the dba, me, or anyone else" |
| 18:15.34 | keldwud | I'll have most of my plan written by Friday, I hope, so that would be a great opportunity to be like "you missed ALL THESE THINGS!" |
| 18:15.35 | keldwud | lol |
| 18:16.11 | beelzebob | keldwud: something to keep in mind about A/B blue/green - it can be ambiguous which is 'stage' and which is 'prod' at any given time |
| 18:16.26 | beelzebob | that sort of confusion can cause issues |
| 18:16.37 | keldwud | beelzebob: so I'll need a robust naming schema |
| 18:16.47 | keldwud | or something that shouts "HEY I'M PRODUCTION!" |
| 18:17.16 | keldwud | hmm, I could put them on separate networks and then move my gateway back and forth between the two maybe |
| 18:17.26 | keldwud | that way once it's live and in prod, nobody can touch it |
| 18:17.31 | spangborn | Yeah but then how can you tell which is which |
| 18:17.36 | spangborn | as a developer |
| 18:17.43 | spangborn | I'd worry about deploying shit to the wrong box |
| 18:17.55 | beelzebob | however :) but it is inevitable that a dev will not get the memo and come in, "I pushed my changes to B - that's still stage, yeah?" |
| 18:18.11 | keldwud | I could solve that at the network layer, couldn't I? |
| 18:18.20 | beelzebob | ..for a liberal definition of 'push changes' |
| 18:18.28 | spangborn | beelzebob: That'd be my fear |
| 18:18.30 | keldwud | if they can't access the network on the live/prod version, then they can't push code |
| 18:18.42 | spangborn | But then how do you debug prod |
| 18:18.46 | spangborn | or check logs in prod |
| 18:18.50 | keldwud | spangborn: lol |
| 18:18.54 | keldwud | that's what staging is for |
| 18:18.57 | keldwud | and dev |
| 18:19.01 | beelzebob | right - but the challenge for A/B is that switch/restraint changes all the time |
| 18:19.13 | spangborn | You've never had to figure out what's going on in production? |
| 18:19.15 | beelzebob | not intractible, but more moving pieces |
| 18:19.40 | keldwud | spangborn: I have sentry logs for my application level errors. devs can always access that |
| 18:19.49 | spangborn | Seems like this could cause a bigger fire than you're trying to solve |
| 18:20.14 | beelzebob | in a traditional dev->stage->prod setup, those boundaries are static, and can be done once and left alone |
| 18:20.19 | keldwud | spangborn: that's why I'm chatting with you guys :) to make sure I temper my thoughts with common sense |
| 18:20.34 | spangborn | I mean, it's an interesting idea |
| 18:20.47 | spangborn | I'm just sure some idiot will cause a bigger problem |
| 18:20.53 | keldwud | ok so that's a good argument for traditional dev/stage/prod, switching context a lot can cause confustion |
| 18:20.55 | spangborn | I should know, occasionally being that idiot |
| 18:21.36 | beelzebob | I like the A/B idea though, and for deployment/testing it has a lot of merit |
| 18:22.45 | spangborn | Sounds like you need an A/B test of the A/B environment |
| 18:22.45 | beelzebob | there will always be a need for a 'prod-like' environment that is not prod |
| 18:22.48 | programmerq | yeah, the projects I've been on with blue/green it was constantly "which one are we doing? k" |
| 18:23.04 | programmerq | but some companies absolutely swear by the practice. |
| 18:27.39 | manithree | It's kind of an important part of immutable infrastructure |
| 18:29.17 | beelzebob | should you go with the A/B stuff, be sure to apply liberal amounts of automation that can't be subverted :) (and doesn't break) |
| 18:29.44 | beelzebob | but like all things, as people have said, it depends a lot on your constraints |
| 18:31.06 | beelzebob | if you've got a giant relational DB that requires a lot of schema changes, your answer will be a lot different than if you've got a small nosql that doesn't change much |
| 18:31.54 | beelzebob | cultural dev/qa practices factor in a lot too |
| 18:33.08 | beelzebob | ...and downtime tolerance |
| 18:37.41 | romanovic | opinions on git branching model - for a project with infrequent releases and no CD, is there anything more practical than the master/develop git-flow-style model? |
| 18:38.23 | keldwud | traditional dev/stage/prod will require less up front cost though, won't it |
| 18:40.34 | programmerq | keldwud⸠yes. The major disadvantage to a blue/green style deploy model is definitely the extra infrastructure cost. For some companies, that is a nonfactor, however. |
| 18:40.53 | keldwud | goals are to identify bugs earlier in the dev lifecycle and to allow contractors to do QA work |
| 18:41.27 | keldwud | as the platform engineer, I'd *like* to do the blue/green because it appeals to my goal of zero downtime deploys |
| 18:41.36 | programmerq | Arguably, the tooling around a blue/green style deploy is a higher cost to get developed upfront. |
| 18:41.52 | keldwud | and I'm absolutely okay with the higher upfront cost |
| 18:42.09 | programmerq | it is possible to achieve 0 downtime deploy without blue/green. |
| 18:42.42 | keldwud | and have the room to do it. but I don't want to get stuck in the mindset of "I have to do blue/green" just because I want to and ignore that traditional dev/stage/prod could solve it more easily |
| 18:43.18 | keldwud | I'm thinking the differences would be maybe 2 months vs 1 month of planning and deploying |
| 18:43.42 | keldwud | I have less than half a dozen nodes that I need to worry about |
| 18:43.56 | keldwud | as far as my deployment pipeline goes, that is |
| 18:44.06 | programmerq | what sort of load balancers do you have? |
| 18:44.24 | keldwud | docker swarm handles my load balancing |
| 18:44.49 | keldwud | my application is a stack of containerized microservices |
| 18:44.59 | programmerq | ah, so just the built-in round robin ipvs/routing mesh stuff |
| 18:45.04 | keldwud | yeah |
| 18:45.07 | programmerq | you will honestly want to have some sort of load balancer in front of that if you're going to do blue/green |
| 18:45.29 | keldwud | that might also help solve some of the DNS issues I've run into with docker swarm |
| 18:45.36 | programmerq | you could have your "blue" service and "green" service listening on different ports, and switching would just be changing the port in the LB config. |
| 18:46.02 | programmerq | the routing mesh is really designed to simplify ingress load balancer setup. you can use it directly, but meh. |
| 18:54.41 | keldwud | but yeah, my personal goal while here is to implement 0 downtime deploys. so I should probably invest the up front cost to blue/green instead of traditional dev/stage/prod |
| 19:04.27 | keldwud | boo @ more shootings! |
| 19:23.25 | *** join/#utah josephscott (~josephsco@2601:681:8001:c340:e569:a876:8976:73c4) |
| 19:38.24 | jfindlay | I don't think there is a universal solution for every project/company |
| 19:38.45 | jfindlay | one deployment method could be totally right in one situation and totally wrong in another |
| 19:39.20 | jfindlay | I think you have to use your intuition to know what bests serves the project's goals |
| 19:40.03 | jfindlay | basically use good tools and good judgement |
| 19:41.11 | jfindlay | and yes, culture is a valid influence on those decisions (to an extent) |
| 20:11.41 | keldwud | oh hey jfindlay long time no chat |
| 20:12.06 | keldwud | and thank you for the feedback programmerq, beelzebob and spangborn |
| 20:12.20 | spangborn | keldwud: Oh it's just my bullshit 2 cents |
| 20:12.23 | keldwud | oh and thanks to jfindlay too. meant to add him in the above statement |
| 20:12.27 | spangborn | But you're welcome ;p |
| 20:12.50 | keldwud | spangborn: yeah, yours was a bit more $0.02 cents versus programmerq's $2.00 :) |
| 20:12.58 | spangborn | lol fair point |
| 20:13.44 | keldwud | but during these planning and research phases, many perceptions are better than fewer perceptions so your $0.02 still counts |
| 20:13.56 | keldwud | still has value |
| 20:14.37 | spangborn | Hahaha |
| 20:16.32 | keldwud | so after checking my bias at the door and realizing that zero-downtime deployments aren't the be-all end-all in my environment, (downtime during deploys has virtually no impact) I've decided to move forward with traditional dev/stage/prod. |
| 20:17.00 | keldwud | I still plan on taking you up on that lunch meeting this Friday, though, programmerq. |
| 20:17.17 | keldwud | make sure I've dotted all my i's and crossed all my t's |
| 20:38.07 | keldwud | ok so one thing I don't like about different environments is that they always get out of sync with each other. Maybe there's a way I can just spin up my staging environment every time I need it. |
| 20:40.21 | keldwud | ooor |
| 20:40.30 | keldwud | maybe I can just test in prod |
| 20:40.40 | spangborn | I don't always test my code |
| 20:40.44 | spangborn | but when I do, I test it in prod |
| 20:55.51 | keldwud | anybody ever use AWS opsworks? |
| 20:56.50 | youtah | http://www.lung.org/about-us/blog/2016/07/popcorn-lung-risk-ecigs.html << well, that's scary |
| 21:46.47 | *** join/#utah SunSparc (~jonathan@45.56.57.85) |
| 22:08.59 | *** join/#utah jnbek (~jnbek@pdpc/supporter/active/jnbek) |
| 22:50.10 | *** join/#utah jab416171 (~jab416171@c-67-172-234-125.hsd1.ut.comcast.net) |
| 23:33.47 | youtah | https://i.imgur.com/tRsVu08.gif |
| 23:46.38 | *** join/#utah keldwud (ae34be84@gateway/web/freenode/ip.174.52.190.132) |
