Jamie Duncan: Yes. Hey, and thanks for joining us on episode
11 of the K files. My name is Jamie Duncan, I work at Google
Cloud with me all, as always is tark Islam.
Tariq Islam: Hey everyone, tarc Islam, also with Google Cloud,
Jamie Duncan: and john Osborne,
John Osborne: head runs is john Osborne with Red Hat.
Jamie Duncan: JOHN is currently enjoying a Stella Artois from
his home studio, and I'm a little jealous of him,
John Osborne: well deserved.
Jamie Duncan: It's Tuesday, and john said it's already been a
run. So this is not going to be a coop con recap episode,
because I think all three of us combined have had a chance to
watch 17 minutes of coop con content. And since it ended last
week,
John Osborne: I tried. I don't know what you guys, I just, it's
the whole idea of blocking out my calendar when I'm home. And
then someone's like, I have this really high priority thing. And
it's like, well, I'm sorry, I'm watching YouTube, like it
doesn't, just doesn't, doesn't really work like when you're at
it, as well as when you're actually physically at
somewhere. So
Jamie Duncan: yeah, yeah, I've run in I have this, I don't have
the attention span for a virtual conference. I really don't. And
Chris short, said it pretty well. He had a thread on
Twitter, where he's kind of said, you know, this is my last
virtual event. And he laid out some pretty good logic and some
stuff I didn't wholly agree with. But I certainly understand
where it's coming from. And I agree with the sentiment in
totality. So So no, coop con recap. We're, I'm sure we will
find as the content gets onto YouTube and gets out there.
Well, I plan on watching a ton of it, there were a ton of good
talks, but so far have watched almost none.
John Osborne: I do feel like one thing I will say about coop con,
I do feel like in the future, they'll probably have both modes
where you can go there. But also it is in a good experience for
people that want to be home. And it was cool. That was if you
wanted to stream it it was you know, I think it was $100 or
something which you know, right. And normally it's like $2,000 or
something like that. And plus travel plus, travel. Yeah, all
that stuff. So at least I made it Mark accessible to more
people. But
Jamie Duncan: yeah, making it democratizing in that way is
really cool. But just being a virtual event was a little too
rough.
Tariq Islam: I think it comes down to delivery as well. I
mean, for me, just you mentioning that travel and the
time off and the hotel stay and then all that just exhausted me
just
Jamie Duncan: talking about it. But we are taking on something
pretty ambitious. not going to get into the huge chunk of it,
the short version is for the next several episodes, we are
going to be tackling the majority of what ci CD means in
the Kubernetes space today. And we're still settling some of the
details. But it's going to be five or six episodes. And we
have a question for people. And we'll probably throw a Twitter
poll out there. JOHN will have details on that later. But this
first episode, we're gonna get down I think first and foremost,
we have to define what ci CD means. I always we always, we
tend to start with definitions here. And then we're going to
get into the backbones, the platforms on which you can do ci
CD. And really kind of get down into the nuts and bolts a little
bit and talk about some of the pros and the cons and what would
we using today. So who wants to take a stab at defining ci CD?
Tariq Islam: Let me start by saying that in the minds of many
folks, you know, see ICD is sometimes conflated with DevOps.
And I think it's important for us to start defining c ICD by
first defining what it's not. And see, ICD is definitely not
DevOps, but it certainly falls under the auspices of DevOps. I
think I think we'll start teasing out DevOps through the
course of these episodes. Because they are inherently
interrelated. And again, I think ci CD represents a subset
manifestation of a DevOps dev sec Ops, Dev, appsec, ham
sandwich. We put it culture, but I think, I think I think it's
important again, for the listeners to just settle, you
know, the backdrop here is that they are, they are very separate
topics to discuss. And with C ICD, and how we're going to
approach it, it is about the tools. But you know, and john
made this great point earlier. It's also about the process
around those tools and and the impact that all of that has on
the people. So you can start seeing where that bleeds into
the DevOps part of it. But it's not it's not everything. DevOps.
I know that sounds weird, but I think it'll become clearer as we
as we go through these episodes.
John Osborne: Yeah, I mean, if I always tell people like if
DevOps was just purely a technical problem, it already
would have been solved, right? The hard part is the people. But
I think at the same time, if you look at what you can do in an
organization for most people, it is they can automate something.
And that's why we I think, why we end up talking about ci CD so
much, because it is a, it's more tangible in the sense that more
people can make a difference through automation, and some of
these some of these practices, whereas, you know, defining
outcomes, and you know, reorg, doing reorg is when necessary,
and some of the other cross functional team stuff ends up
being, you know, there's only a small number of people in
organizations that can change some of those things that you
that ended up need to be changed at some point. So
Jamie Duncan: yes, it sounds like we're gonna be talking a
lot about technical debt from a couple of different angles.
Tariq Islam: I think we have to Yep.
Jamie Duncan: Yeah. And it's kind of interesting. In I had
this discussion, I forget who I was talking to whether it was on
Twitter or in like an internal chat room or what. But um,
technical debt takes on this negative connotation
immediately. But when you're are doing something like john said,
when you are making these positive changes, because if
you're a completely manual shop that runs off with three ring
binders, and three gay and conference calls, going to
getting into, you know, automating your pipelines for
continuous deployment is taking on new technical debt. And there
are good kinds of technical debt to take on, you know, it's like,
there is, you know, there's payday loan technical debt of
people making poor choices, and everyone having to endure them,
you know, for a long time, organizationally. But there's
also mortgage technical debt, where the debt is, you know, of
a low enough interest rate to push the to push it a little too
far. But it It provides a lot of benefit for what you're taking
the dead on for, and ci CD, a CD platform. Most of the time,
unless it's just a really poor implementation is good technical
debt to take on.
John Osborne: Yeah, I think too, like, you know, talking about
technical debt. You think about agile and some other things that
also, you know, there's a Venn diagram somewhere out there, but
that get entwined with things like ci CD sometimes in Sprint's
and these things, and there is almost this feeling out there. I
think that's honestly probably justified that a lot of times,
we get so focused on features that we're not adequately
prioritizing. I'm totally gonna go off the rails in this
episode. But we're not prior, we're not prioritizing our
technical debt and our testing and good architecture diagrams
and documentation and those things because you start doing
two week sprints and things like that it gets it does get, if you
do that poorly, it actually can be worse than waterfall if you
don't actually start. You know, prioritizing things the way they
should. And I will say to for as a side tangent, we'll put it in
the show notes, there's a new really good book out called
project to product on a flow framework. It's this kind of
prescriptive plan for value stream management. And the the
author also spoke at the DevOps enterprise summit, He's really
good. I've never tried this, but just reading it and in watching
it, it makes sense and resonated with me a little bit in terms
of, you know, how much should you how much of your Sprint's
should you focus on things like reducing your technical debt and
new features? And what makes and when should you adjust that and
why and it all sort of made sense to me? Yeah, I
Jamie Duncan: mean, the traditional, you know, the old
trope is that features drive revenue. But even with
Kubernetes, we've gotten to the point where like cube one died,
18 was very much a bug fix release, there were very few new
features, very few things moved from alpha to beta, or beta to
GA. And like you're talking about john, like, when you're in
these two week sprints, you can fixate on the wrong things, and
really drive your product into an unhealthy state.
Tariq Islam: So I guess then, the goal of properly done CIC D
is to avoid, you know, forgetting the forest. Right,
and only focusing on the trees. Yeah. And that's, and I like
your call out about technical debt, Jamie. But I do I do have
a recent story. Actually, I was working with an organization
that they were still packaging up now actually, to this day,
they are still packaging up their, their binaries in a zip
folder and emailing them to their ops team for deployment.
John Osborne: I've seen that all the time. So
Tariq Islam: yeah, it was Yeah,
John Osborne: I just moving it around, you know.
Jamie Duncan: Not even SFTP Yeah, not even.
Tariq Islam: Yeah, I see that all the time. So but but that's
where folks are right. And I think i think i think CIC D it
to John's point, it really does require a cultural shift just
like Kubernetes adoption does that we've talked about before.
I mean, this is we're talking about not just automation to
solve problems, but this is this is, you know, organizational
infrastructure that you're going to build on top of.
John Osborne: I'm really getting off the rails In this episode,
by the way, I feel will
Jamie Duncan: keep you on track, we'll keep you on
John Osborne: Kubernetes like actually makes a lot of sense
here. And there's other technologies too, but you know,
it's, we talk about, I think, Craig Mcluckie talked about this
like last year, but you know, the move from ticket driven to
API driven well, instead of putting in a ticket for saying
you need, what storage and how many replicas you want, you
know, that can be part of the API that developers hand over to
operations teams where they can say, here's my deployment, I
need, I need, you know, this persistent volume and need to
run this number of replicas within this.
Tariq Islam: So that's an interesting point. Because
Because I don't think there's a singular formula for ci CD. I
mean, there's like, okay, build your app, run your test, create
your artifact, you know, deploy to staging, you know, yada yada,
yada deploy to production, Canary, whatever, and then
you're good to go. But that can manifest itself in many
different ways, where different organizations define the lines
between their teams differently. And I've seen ci CD work where
developers are reaching over to the right, where they are
dealing with, you know, API driven requests, or that that
engagement pattern is definitely something that I've seen work,
but it requires a lot of, I think, a high degree of maturity
that most probably don't have.
Jamie Duncan: And there's a there's a pretty important
distinction as well, that I want to draw in to try to segue this
into talking about sort of how you get started, like talking
like using target customer, you know, whose zip file, you know,
emailing zip files, to ops people to deploy to deploy new
versions of things. Were john, where you pointed out, you see
the principles and Kubernetes very much our peanut butter and
jelly kind of scenario, chocolate and peanut butter.
Apparently, peanut butter goes with everything, really. But
Tariq Islam: it really including me, oh, by the way, believe it
or not,
Jamie Duncan: yeah, I've seen it, and not in that camp. But
let's just take a second to recognize the universal love of
peanut butter. But there, the there is a big difference,
because like you can't be a little bit Kubernetes, you can
be a little bit pregnant. Once you stand up a Kubernetes
cluster. To deploy an application effectively, you've
got to do all of it, you've got to containerize it, you've got
to figure out config maps, you've got to get your ingress
and your networking and your storage squared away, you can't
do a little bit of Kubernetes, you don't have to put all of
your applications on Kubernetes. But to do our application on
cube, you got to do it all the way. CDs a little different,
because that team that is you know, receiving zip files in
their inbox to do to run a maintenance event. They can
start down a road and and CD principals can be taken on one
or five at a time. And all of them provide benefit. You don't
have to go from zero to 100, like you do with an app in coop.
You can build it up over time and drive that adoption drive
those cultural changes with the tooling.
Tariq Islam: And that's a good point. Because I think that
that's why I've we have seen organizations adopt c ICD far
before, or long before they adopt Kubernetes in that
holistic way. Right. And by the way, I'm totally stealing that
whole you can't be a little bit pregnant.
Jamie Duncan: Oh, you never heard that before.
Tariq Islam: I've never heard that one before.
John Osborne: Oh, are you serious? I've never get around
it for a long
Tariq Islam: time. Michelle, I'm sheltered. What can you say?
Jamie Duncan: Poor little Tarik,
John Osborne: you're a nice boy. I'm telling you, man. I'm
Tariq Islam: a nice boy, you know.
Jamie Duncan: So let's, let's talk a little bit about some of
the options. You know, this first episode we want to talk
about the definitions. Before we get to that I want to get into a
little bit of trivia with you too. And I it's one of those
that where I'm wondering if I if I was sheltered back in the day,
when the first place I was at that was doing anything
resembling a CI CD workflow, and they were doing it really
poorly. But I didn't know any better at the time. It was
continuous integration. Continuous Delivery, was what ci
CD stood for. Apparently now it's continuous integration,
continuous deployment.
Tariq Islam: I think that's still debatable.
Jamie Duncan: Is it so is delivery Am I am I the only is
that like this. Sinbad being the genie in the movie thing? Yeah.
John Osborne: Just you know, deployment to multiple I guess
if you're saying delivering it. I think it I think they changed
it when they started. The delivery started happening to
me. multiple targets. And okay, then they just started moving
into deployment. That was kind of my, my take on it.
Tariq Islam: I'm still a fan of delivery. Actually, I feel like
it's a little bit more generic.
Jamie Duncan: And I wonder if I wonder if there was like a book
written. I wonder if there's, I wonder if there are camps here
and we don't know about it. There are some blogs that talk
about the difference between delivery and deployment. I'll be
honest with you,
Tariq Islam: I don't think it matters.
John Osborne: I remember the first time that that's our it's
like,
Jamie Duncan: Reddit hated it.
John Osborne: The first organization I worked for that
did this. I was the first few years I was in the industry. I
just wrote code by myself in a in in isolation, and Tareq. And
I were joking about a month ago about how ahead some of these
people are they're like, speaking on stage at cube con
and stuff for over them. We were but the first organization I
went to that actually did anything like see ICD, honestly,
I, I was very probably immature. And I wasn't used to doing all
these things. And I remember I went down to this, this guy's
desk, and he's like, yeah, you know, we're gonna, we're gonna
check our code. And every night, we're gonna scan it with a run
these tests, if things break, we're gonna email you right
away. And I just remember thinking, chill out, man. I just
want to write some code. I don't want emails every night with
things breaking, because things always break. And you know, I
didn't, I didn't actually see later I moved into ops. And so I
kind of saw all of why that mattered. But when I was just in
that writing code mindset, I didn't really didn't see, at
that time, at least, while that stuff was so important.
Jamie Duncan: I remember getting all of those alerts. And they
were constant. And they were useless, because they were
poorly configured. Because we had no clue what we were doing.
And they just became white noise. And you ignored them.
That's,
John Osborne: that's what happens still happens. There is
no technology that solves that problem. I do think you need to
there's a, there's just the data you got to put the time in.
Tariq Islam: It's a it's a it's a maturity thing, right. It's an
organization learning about what it cares about, in terms of
delivering features and functions. It gets better over
time. But I think, and john, I think you mentioned this before
we started up, you know, how you measure your ci CD pipelines,
your automation infrastructure, that's, that's hugely important.
And you're not going to get it right the first time, you're not
going to get it right, the eighth time, you might get it
right, maybe the 18th time. But I think it's, it's part of that,
that ci CD journey. And you have to have the patience and the
discipline to kind of stick with it, and get to a point where
it's not just about the efficiency of the automation,
but the the data that you're getting back. Is it useful? And
is it? Is it making your processes better? Is it
impactful?
John Osborne: I mean, the general kind of pattern I see
is, organizations that, you know, initially, they really
just talk about inputs, you know, we've got to, we've got to
do cloud, we've got to do agile, you know, we've got to, we've
got we need the Kubernetes, given the Kubernetes. And then,
you know, once they get a little more mature, they start looking
at the outputs, like the, you know, the four key metrics from
the state of DevOps reports, the number of incidents, those
things, and then once the few organizations have had that have
gotten pretty mature, we'll talk about outcomes. So you know,
reducing costs and improving operations teams and those types
of things that actually make more of a difference. But
initially, it's like, you know, we've got a, we need the cloud.
I mean, it'd be the cloud. And but measurements do become
really important, I think, in general to, to understanding
success.
Tariq Islam: So this, this begs the question, then, you know,
what, what is it good? What is it good ci CD infrastructure
look like beyond just build test? stage, deploy, monitor.
John Osborne: So before Jamie answers to like, let me just
throw, let me just throw everything we did in the, in the
pre, the pre call of the window, cuz I want to bring something
up, see if this resonates with you guys. But there is a really
good article on scaling to 100,000 users, where they this,
I'll put it in the show notes. And it's nothing to do with ci
CD. But I thought we could actually use a similar format
for this episode is based off what Sam and Jamie say, kind of
jog my memory, but they kind of talked about, you know, building
something with one user on one machine and then going into 10
users and you split up the database and then 100 users and
then building up like, maybe we do want to work that way for a
CI CD, or we're talking about you know, if it's Let's just eat
what should you do and kind of build out to what is like a
large enterprise, but what should they be doing? Because I
do think that the answer differs.
Jamie Duncan: Yeah, if it's just me, my Git commits are going to
be purely curse words.
John Osborne: You can't look at my I have nothing. I have no
documentation on code.
Jamie Duncan: Right. Documentation will not exist.
Yeah. I think I like the idea of starting with one person. I
don't think it's particularly practical. He's you. It's one of
those, you know, it's a Dilbert joke.
Tariq Islam: You know, we're like, have you started working
yet? And no, I've spent seven days configuring My environment.
But but but let's go back to that example of of the
organization that still sending zip files in an email to the SIS
ops team that probably started with a single person somewhere.
Yeah, sure. All right. And they didn't try to scale around that.
I think that's a good approach.
John Osborne: I think what happens when you see those
things a lot of times is the hire at the top of the
organizations don't have any idea who's talking to each
other. And so what happens is, is on the ground floor, you
know, they needed to get this to the operations teams, and no one
really thought about silos talking to each other. So how do
you contact someone, especially, you know, all these things
process around for 20 years? So how did you how did you get
something similar? 20 years ago, you emailed it right? And so
nobody really cared to focus on doing an improvement of that. So
here we are, was, that's still our problem. Years later, yeah,
20 years later. Still,
Tariq Islam: I think I think one of the best ways to distill this
maybe is is looking at Conway's Law, perhaps because I think
cicd flies in the face of that a little bit. where, you know,
Conway's Law states that an organization is doomed or
destined, pick your word of choice there to to create
systems and software that mimic the communication patterns of
their, of their organization. See, ICD is at least properly
done is actually meant to break that or improve upon that. And I
think that's, that's what makes it hard to get right. But to
Jamie's point, there are areas of ci and CD, particularly on
the CI CD part where actually on both sides, where you can, you
can cherry pick a little bit on what you want to improve upon
before taking a bigger bite. And so I think I think, you know, if
an organization wants to do things better, it's a great
place to start. It's a it's a good path of least resistance.
And and once you really get really good at C, ICD and
automation, I think, I think appetites increase for more
advanced tooling and platforms, you know, Kubernetes, and the
like.
Jamie Duncan: So let's talk about getting started a little
bit, because you have to, you know, see the end, we were
talking a little bit about this, before we hit the record button,
CD is the tooling. And, you know, that's really where the
the sort of the line of demarcation begins, but between
defining ci CD and defining DevOps, and they are very
knitted together, but see ICD very much represent a toolset
that you've adopted to get work done. Whereas DevOps is much
more philosophy. If you're adopting, taking that idea of
torrox example of the company that sends a zip file for over
email for deployment jobs. If the, if the visionary in that
group, we're starting to build out some sort of CD pipeline,
they would have to start with an engine, they would have to start
with a whatever you want to call it a backbone, a platform, an
engine to begin to integrate the CD steps. And it's that's the
first step they have to get on top of they have to take that
first step up.
Tariq Islam: Yeah, and and I'll go ahead and call this first one
out. And but before I call it out, I am going to call back to
a previous episode when we were talking about remember when we
talked about how Helm benefited from being kind of first to
land, and then being shoehorned for a number of use cases that
aren't so great anymore? Right. I think that's that's where a
lot of people landed with Jenkins. Jenkins has been around
forever. And it's a build server. Right? It it since
inception. And it's a very good one. But it's, it was almost
purely ci and because of its pluggable architecture, it
became, for a long time the de facto choice and I guess even to
this day, it's still kind of is despite it, there being you
know, hundreds of other options out there. It's just so so
ubiquitous,
John Osborne: it's a good point because I think if you look at
Jenkins is there's there's there's a lot to be said about
it but I think similar to home and you make a good point is
that it's it's the community of plugins and things that are
already out there I can go on to Jenkins and build something out
in an hour that plugs to my Slack channel and does all these
things because those plugins exist and they're easy to
integrate and so it's definitely not a what I would consider like
a cloud native build system but it there's so much there that
you can reuse out of the box that it we don't prefer that you
use at Red Hat we don't prefer that use Jenkins but we get it
if if if that's where you end up landing
Jamie Duncan: Yeah, the copy paste ability of Jenkins once
you have something built, you're able to duplicate it quickly and
enhance because what you find in a because it does very much
mimic your communication And patterns, most of your
applications deploy in a similar way. Once you get the first one
figured out, you know the first pill cost a million dollars a
second pill cost a nickel. Once you get that first pipeline
built out, you can reference it and just make the needed
modifications to build out a pretty mature Jenkins
environment. I deployed Jenkins Actually, this morning, just to
look because I hadn't looked at Jenkins in a long time I
deployed it into a Kubernetes cluster. And the process was
pretty good. You know, so I took on, I said, you know, when we
were planning this first episode out, I said, I would deploy
Jenkins and Spinnaker, and then add a Jenkins X to it this
morning when I was when I was tinkering around. Have those
three? Jenkins is the most straightforward thing to deploy.
Was it a Helm chart? Yeah, of course, it was.
Tariq Islam: I knew it.
John Osborne: Jenkins feels so 2005. But it does Oh my god, it
is.
Jamie Duncan: Like, it's, I'm stunned it how like, it's soon
Alright, so to deploy Jenkins, your there's a little bit of
ammo, it needs a service account inside the coop cluster. You
create a namespace, create a service account, run a Helm
chart, and your cluster is up and running takes three minutes,
I logged into that thing. And I was immediately in 2009. And the
only differences in the UI that I can ascertain after like a 10
minute analysis was they've made the font smaller. And the
graphics were updated about six like they it's like they took
the kitchen from 1998. And they painted the cabinets white and
put down new countertops. But it's the exact same kitchen
Tariq Islam: felt like coming home.
Jamie Duncan: Oh, like, like coming in back to the you know
the step dad that you never really liked them. But they're
very straightforward to deploy into a Kubernetes cluster and
took advantage. Like once it had that service account. It did
populate stuff. So like it was running builds inside the coop
cluster, it was doing everything pretty effectively, pretty
quickly. But I'm just like it was 12 years ago, when you click
new pipeline, you know, no matter which option you pick
whether it was one of the pre canned ones, or the DIY one. It
was 74 radio buttons, 92, checkboxes, and every one of
them brought up for other questions you had to answer. And
at the bottom was a text area that I wrote my shell script in?
Yeah, it is. And it was I was like, Oh, my it like the
workflow. And I guess at that point, because it is so
ubiquitous, it's rate of evolution has to be incredibly
slow. It's the generic general purpose, right? The old server,
everyone uses it. So everyone's holding on to the way they're
currently using it. I feel really sorry for cloudbees you
know, the people that do it professionally?
John Osborne: Yeah, there's something to that too. Because
if you look at the a lot of the way, you know, in practice, at
least what I've seen, tell me if you've seen this, but when
people get hired as a quote unquote DevOps engineer, what
they end up doing is just writing a lot of glucose, and
that glucose ends up often living as custom shell scripts
inside of Jenkins. Yeah, and yeah, that's a hard place to
live in, I think,
Jamie Duncan: yeah. It reminded It reminds me a lot of like, the
way pivotal stuff works. reminds me a lot of the way I did jobs
1213 years ago,
Tariq Islam: I will say that, you know, I'm a fan of knocking
on Jenkins, but for, for the, for the for the listeners. And I
still think that if if it does end up being Jenkins that you do
go with as your you're kind of getting started stepping stone,
so be it right. And if that's what that's what you're used to
get good with ci CD and automation and getting, getting
to a better place. Go for it. Now, that said, You are
absolutely going to hit, you know, major issues with respect
to scaling. You know, Jenkins is not exactly friendly to
resources. But that's something that that's something
John Osborne: you might care about day one.
Jamie Duncan: Yeah. And like you said, starting at one user, it
works just fine. Mm hmm. Yeah, yeah. It really scaling up in
102,000. I mean, let's all you know, I freely admit that the
modern internet owns lens, a large portion of its existence
to Jenkins, is you can't get that much software deployed
manually. We would still be working on your stuff from seven
years ago. We all have benefited from Jenkins globally as a
product that said, it's like Domino's Pizza. It is not the
best pizza in the town you live in, but it's going to show up
pretty consistently better than they're going. Right, exactly.
They're going to be there in a half hour and you're going to
eat it. Yeah. It's not the best pizza on the planet. You can get
a better experience. But, but you're going to pay for it and
you're going to have to jump through some hoops to get it. So
one of those better experiences. So what Jenkins ended up doing a
few years ago, and I don't know exactly when. But Jenkins then
is, is there and it has a sort of glacial evolutionary path,
you know, because everyone's using it. No one wants it to
change. So what they did was make a thing called Jenkins x.
It was back when I think putting x in front of at the end of
things was cool. That was actually a team that left Red
Hat to, to go build that. And if you remember, was it. Yep. It
was. Oh, yeah. Yeah, not.
John Osborne: So. And it wasn't the same Jenkins team. It was a
bunch of people that were really strong and on messaging and kind
of middleware. Some of the people that worked on are built
Apache camel, those types of people, and they left and did
their own thing. And they came up with Jenkins x.
Jamie Duncan: Yep. So Jenkins X has a great UI for Kubernetes
ties itself into octant, which is a really solid user
experience. There's an ops persona panel for for Jenkins x,
there's a dev persona panel for Jenkins x, really slick, really
modern, get a lot of work done really efficiently. So I think
under the covers, it's using a lot of the same state engine
tooling. It's a lot of the same engine of Jenkins, but it's
presented in a more modern fashion. And it's got more
modern hooks, like there's a more modern API that you can
interact with, and really deal with the tooling much more in a
much more modern cloud native way. deploying it to some of the
silly stuff I've ever seen in my life, I cannot figure out the
lot. The logic of how they came up with the deployment tooling
for Jenkins x. It starts out really cool. It starts out easy.
Like it's like 10 lines of JSON, that you drop into a terraform
template, and you run terraform in it and terraform apply. And
then it checks out a repository and it does Oh, no, then it does
some stuff. So it actually deploys a Kubernetes cluster on
your cloud provider. So like you tell them in that in that JSON,
you tell it what cloud project you're using in different ones,
like AWS looks a lot different than Google Cloud, all that. But
it stands up a Kubernetes cluster, and in that Kubernetes
clusters running just about nothing, then you have to go
download another binary, called j X on your local machine, run a
couple of more commands, one of which checks out a repository
from GitHub. And then that generates a yamo file that you
run another command on that thing goes and sticks the right
things in the Kubernetes cluster that you deployed 20 minutes
previously.
John Osborne: I say felted, just when you're explaining, Yeah,
Jamie Duncan: dude, I and
John Osborne: I looked at Jenkins x when I first started
that company. So I was interested in what they were
doing. But I haven't really followed it much the last couple
years.
Jamie Duncan: I don't understand how they got there. And the fun
thing is I pull down the latest version of everything. There is
some hardware inside some of that. And it's all terraform
under the covers the deployment tooling inside some of the
terraform, or inside some of the file somewhere inside this JSX
binary, it is hard coded to look to check out the master from
when it checks out this GitHub repo, and the master branch
doesn't exist anymore. And GitHub because they renamed it
to Maine, but the installer hasn't been updated. And I
pulled down the latest release binary to try it out for like
three different versions. So the installer for Jenkins x is
effectively broken right now.
Tariq Islam: So this is where managing your own
infrastructure. And the type of infrastructure you choose. As
soon as you get beyond a small team really starts to matter. I
Jamie Duncan: think so yeah. So when we were sketching this,
this episode out, we broke down three generic categories of CD
backbone CD platforms, Jenkins and Spinnaker would be the
number two one, but we're not talking to so Jenkins and
chicken neck Spinnaker, and I'm sure there are others are what
you would call traditional. And those are the you know, I'm
going to deploy a thing and whether it's deployed into
Kubernetes or deployed into a VM or wherever. It's a traditional
application, and I get a traditional experience from it.
The second type C FCD. backbone is one that's a little more
cloud native. And is tark. is do you want to get into that a
little bit.
Tariq Islam: Yes, so the cloud native ones, some some examples
here. But this is by no means an exhaustive list. In fact, this
is probably 1% of everything that's out there, you know,
tools like circle ci, GitHub actions, git lab, ci, Travis CI,
I think of those really comes out to, you know, GitHub and get
lab being the more popular ones. I've seen more folks, I've
actually seen quite a lot of Git lab.
Jamie Duncan: Yeah, it's really good. I used it a lot. We used
it a lot at VMware. really effective. Travis and circle I
think they were they were first to market in this space. And I
don't know who be who. But Travis and circle, were filling
the gap before get lab and GitHub had their own integrated
CD
Tariq Islam: platform. For my own edification, how are we
differentiating between traditional and cloud native? Is
it just how many years old they are? Or deployment model? Maybe
both? What makes a thing cloud native in this context?
John Osborne: I mean, I think, you know, if you look at tecton,
or some of the newer ways to build pipelines, it's that they
can one be event driven, in the sense that, you know, something
can trigger the pipeline to start or even instantiate from
scratch, right? And then, you know, does it run well, in
containers? Can you scale up scale down? Very easily? You
know, is it a lot of times now, we're also talking about is it
declarative in the sense that you set what the desired state
is going to be? And not just the steps to get there an imperative
model? But it's, you know, is it going to run on Kubernetes?
Well, is it going to run as a container? Well, those are the
types of things that I that I normally think about, I will say
to like, before, before I forget, you know, I think when
we talk about building, in a pipeline, any sort of scale like
and I would even say, once you get to like 10 nodes, if you're
building in your cluster, you probably want a dedicated
machine just for the builds, because that he comes so spiky,
that it really can destroy some of your, your, your node
resource usage, you know, things can spike up really high spike
back down, especially if you're doing Java, you know, start
doing Maven builds or even node j s, you know, Mo the NPM stuff.
You know, even go right, because it's it's statically compiled.
So the builds tend to be pretty compute resource intensive
first, but obviously, the runtime is very good, right. So
you know, you probably at any sort of even medium scale, you
probably want to look to start to have dedicated machines that
just do builds. This sounds
Tariq Islam: a lot like the problems that we've seen
organizations run into with, with Jenkins, right, is resource
contention spike Enos having to have dedicated infrastructure, I
think, you know, I don't know, maybe maybe cloud native is not
the most accurate term for this category, maybe it's the fact
that the design principles, the user experience, in these kind
of Next Gen. backbone, backbones are, are just better, you know,
a little bit more refined.
Jamie Duncan: And I'm thinking through, because I think I was
the one that wrote cloud native in the outline, so I'll take the
hit for that one.
John Osborne: Actually, I didn't even see that.
Jamie Duncan: But I'm gonna hold on to I'm not gonna let go of it
just yet. Because john tark, I think you're 100% on point is
that they are an evolution and how it works. The ones that we
the ones that we've taught that we've sort of enumerated circle
and Travis and get lab and get hubs integrated ones, and there
are a ton more and a ton of them are very platform specific, like
natla, fly and a bunch of others. Even if you're deploying
get lab on prem, and you do have to account for those build
resources. When I'm hooking Mike a Kubernetes, cluster into those
build resources, it takes me about 30 seconds. And the entire
experience is driven through the Git lab UI. And I effectively
just give Git lab permissions to go deploy workloads on my
cluster. And I can drop it into a namespace with quotas and
handle those resources that way. So we're Git lab, because it has
an on prem version is almost a hybrid between a traditional and
a cloud native. You do account for that to some degree with
with Jenkins. And I've actually ran in or give with Git lab. And
I've actually ran into that where, you know, my build server
ran out of resources a lot. It was some projects I worked on at
VMware. But with like GitHub, and Travis and circle, they're
all as a service. So you don't have to provide any of you don't
have to account for any of it. And the worst case scenario is
that you're waiting a little while longer on your build to
run if it's eating up a lot of stuff, or if the run queue is
just long with GitHub actions or, you know, in the Travis CI,
Travis is handling all that for you or circle ci is handling all
that for you.
John Osborne: They've done a really good job. Some of those
tools were Oh man,
Jamie Duncan: they are awesome.
John Osborne: They used to be a Git repo and now their ci CD
workflows. thinly disguised as a Git repo a Git repo. Yeah. But
they've done a good job for sure.
Jamie Duncan: Yeah. And the cool thing about all of these is the
way they present themselves to the user. And it's kind of the
reason I'm holding on to cloud native as them as a moniker for
them. Because all of them have the same workflow. Whereas
Jenkins, you've got to go into the Jenkins UI to find these
things, you know, define a pipeline to get all this stuff
hammered out all these buttons and checkboxes and text fields.
With all of the cloud native tools that we've talked that we
just mentioned, all of them are a single YAML file at the base
of your repository. And that your entire experience. That
means that it's massively powerful. It is Yep. And that's
what that's why I'm holding on to cloud native, because if I
drop a YAML, dot, Travis, ci dot YAML, at the top of my repo, and
I tell Travis that you have permissions to talk to this
repo, and I'm done.
Tariq Islam: So this begs the question, then is it worth, you
know, whether it's the organization that's still
emailing their zip files? Or maybe it's another organization
that's slightly more mature than that, or retelling them? Or we
issue like a peanut butter spread? Going back with peanut
butter here?
Jamie Duncan: Are we just telling everyone always goes
back to penguin?
Tariq Islam: are we telling folks, you know, just go
straight for the, the what we're calling cloud native, like the
circle ci, and the Travis CI is and get labs of the world? And
skipping over, over Jenkins? And just maybe, are we saying that
Jenkins is just legacy enough? That you can you can you can
forget about it and move into these newer tools?
John Osborne: I think it depends. Because if you look at,
you know, cloud native stuff, right, you know, you has all its
dependencies with it kind of handles restarts, well, you
know, it can run on containers, Kubernetes, it's highly
available, all that stuff. You know, that requires in some
cases, you know, so you're using tecton, as an example, right. So
you might have to write containers with automation in it
to make some of this stuff happen that you might have a
plugin out of the box with Jenkins. So it's, it is kind of
a your mileage may vary, depending on your use case, but
and not every org has people just sitting around that are
writing Glue Code in containers, to do all this stuff. So it's
kind of a, you know, you have to make your own decision. But I do
think that there's a certain amount of technical debt you
assume by by doing Greenfield Jenkins deployments at this
point.
Jamie Duncan: Yeah. And I think at least for me, I see them as
a, it's a case coverage problem is circle and Travis and GitHub
and get lab, they all have some limitations. If you're going to
use Travis or circle, or just about any of them, I think
you're going to be in the limitations aren't great. But
you're going to have to use some of their tooling. And they're
going to make some assumptions about what you're doing. And the
steps are going to be relative or somewhat limited compared to
what you can do in Travis, or what you can do in Jenkins,
where you know, the world is your is your checkbox. If you
find yourself straining against those limitations be because of
the way you've designed your application, or because of the
limitations or security needs you've got. You always have
Jenkins available as this infinitely customizable platform
that's going to get you where you need to go. But for that
problem, I don't know if it's 85, or 97 and a half. But for a
very high percentage of modern applications being written
today. These cloud native CD tools, fit your use case very
comfortably, and are a much lower barrier to entry into this
arena.
Tariq Islam: And I will say that of these, or at least of the
ones that I've seen, I think I think Git lab is probably the
furthest along as far as bringing together feature set
ease of use deployment pattern. You know it? Again, to your
point, Jamie, it's not it's not 100%. As far as customizability
the way Jenkins is, but I think I think for the vast majority of
organizations, they've I'm calling them out because you
know, they've done great work.
Jamie Duncan: Yeah, I agree. I like I said, I use it pretty
extensively at VMware, and have used it on and off before and
after. And I'd agree, yeah, they are the most mature of all the
cloud native ci, CD tooling. And it is very easy to use.
Tariq Islam: So let's talk about this this last category that
we've got here, managed, right and managed by managed and the
most sensitive to this term. Because I think there's a lot of
confusion around it. Anything for me, and we I don't I don't
want to make this a rat hole. But when I say manage I'm
talking about, you are not deploying this infrastructure or
this tool in any way, shape or form where you're having to
manage it. Right? There's no such thing as day to operations
here. This is a purely Oh serverless for form of, of ci CD
infrastructure, where you just kind of click on a button. And
and you get a thing. Yeah.
Jamie Duncan: Right. Yeah, I see them is in the what we're trying
to talk the examples of what we're talking about her, like
AWS code pipeline, Google's cloud build, we couldn't come up
with azures, I think they've called it Azure DevOps. Yeah,
it's their builds that
John Osborne: it's that that the TFS, the Team Foundation Server
stuff in Azure called Azure DevOps.
Jamie Duncan: Yeah. And really what you get as you get a pretty
configurable set of inputs. And you get a completely abstracted
build experience, and you get a less configurable set of
destinations or output of what it will what it will generate
for you and where it will put it. But that's really it, you
feed it your source code, you feed it some parameters, it does
some magic, and it drops the result of that magic in one of a
handful of locations.
Tariq Islam: So I think this one, this category is pretty
interesting, because I feel like it is the most scalable from one
to 100,000, just because of the nature of how its deployed, you
know, at least from the perspective of the user, it's
not something that's deployed. But you know, me as an
individual, I can use Cloud build, for example, or Azure
DevOps for just my own purposes. And then I could bring any of
these into a fully fledged enterprise organization, and use
them at scale as well. Now, you know, you do have to consider
which one of these you're going to pick, because the one that
you pick, is is going to is going to effectively tie you to
a cloud. Right, but, but i think i think i think listeners
should, you know, see the advantages of a managed offering
here, because of how quickly you can get up and running, and how
quickly you're able to scale and bring it to the rest of an
organization. But, again, you know, going back to the point
that Jamie made around technical debt, there's something to
consider.
John Osborne: So for my consistency, what you'll see
from adoption, but in my opinion, like ci CD tools in the
cloud, is managed services have some of the lowest adoption of
all the public cloud provider services. And I don't know, but
I think the reason for that is is, is people have a really hard
time paying to build their code. There's some there must be some
psychology there, I don't know. But pay as you go. And
obviously, you pay with, if you manage anything yourself, you
you pay to, in terms of in terms of all the people that you that
are managing those things, right. But it's not as apparent
as paying for a service. And I see challenges with with people
just mentally getting over that hurdle of paying to compile my
code. Have you seen that or such as something that that, that
I've seen?
Tariq Islam: I've actually not seen that yet. Although, again,
I do, I would have to agree with you that it's, it's not it,
there hasn't been a huge wave of adoption across these tools. But
I can, I can see, I can see why that would be prevalent. It's
one of those. It's one of those, like, you know, I'm going to do
it myself kind of thing. It's, it's this is low value task. And
and that's the that's the perception, right, that the
tasks of building, you know, compiling, building, you know,
all of that, all that sci fi stuff, especially, is deemed low
level. machinery. And, and and it's Yeah, I can, I can totally
get there.
Jamie Duncan: Yeah, my take on it was, I certainly agree that
it's not, you know, people aren't beating down the doors to
Google Cloud to use Cloud build. It's a great tool, and it does
exactly what it needs to do. But it's it's very much a value add
for someone who is sort of all in on AWS cloud, at least in the
public cloud space. I assumed that slower curve of adoption
was because people have relatively mature CD pipelines
because like we said, the beginning people have been doing
this for a long time. So if they've got something and it's
not broke, They don't see it as debt, they see it as providing
value. Why would they read? You know, why would they build
cycles to change it? And as you were talking, I started
wondering if that is if it's that as much, or if it's is, as
we were looking at these and talking about them, you know, as
you go from these traditional platforms like Jenkins, to these
cloud native platforms like Git lab, to these managed platforms,
like like AWS code pipeline, each step along the way, you
have to have good, better hygiene. Like you can do some
crazy woolly stuff inside Jenkins and get your application
to build.
John Osborne: Yeah, automation is hard. I mean,
Jamie Duncan: yeah, exactly. Like we said at the beginning,
but and there's lots of duct tape is strong.
John Osborne: Yeah. Duct Tape survival solutions, sometimes
Exactly. Oh, yeah.
Jamie Duncan: And Jenkins is really good duct tape. It's the
duct tape of the internet. It is needed. Yeah, the cloud native
tools, you can use a little less duct tape, you have to have a
little better hygiene, there are some more limitations. And the
Manage platforms, there are even more limitations. Like I said,
you have a more limited set of inputs, you have a more limited
set of outputs. And you don't have a lot of control of what
happens in the middle. So I wonder if it's because they're
just they have something they're happy with? Or it's because
they've got their house of cards duct taped up enough that they
don't want to touch it. That is that sort of limiting some of
that, that that adoption of these managed platforms? I don't
know, it's kind of an open ended question. I'm thinking about it,
you know, even as I say it out loud. But I'm, we're running a
little long here. Yeah, talking about this. So john, do you want
to give us give the skinny on what we're going to do for the
next five or six episodes, yes, and throw this idea out for
John Osborne: free. I will throw this out there too, before we
get into the agenda. But, you know, we talked a little bit
early on before we hit the record button about about
DevOps. And if it's even something that would would
resonate with the users here. You know, it's DevOps in
practice ends up being kind of a collection of practices and
methodologies around product flow. You know, things are out
there's a lot of people and process side to it. And so, you
know, with, would you be interested in the audience of
listening to practices that work, don't work around, you
know, moving things like institutionalized codified
processes into automation, or how to what works and doesn't
work with things like cross functional teams, ways that you
can avoid common pitfalls that we see in organizations, those
types of things? Or are you mostly focused, we'll put a
Twitter poll out, you can email us as well, or DMS, whatever.
But or you just focused on the tooling. And you want us to just
kind of stick with the more I guess, tangible things in
technology, you know, that's a, we're gonna leave that up to
you. If there's, if there's interest, we'll do it. If not,
we'll just we'll skip over it. But I think the next episode,
we're gonna go into ci, and some of the things from a depth
perspective, like code quality and testing and registries and
some of the open source projects that are getting pretty big
adoption there as well, especially from the from the dev
aspect to it, then we'll move into continuous deployment from
the ops perspective, we'll probably touch on some of the
paradigms that we talked about before, like get ops and open
policy agent, but more high level and where that fits in.
And then of course, you can't, can't talk about ci CD right now
without talking about dev sec ops and adding the security
aspect to it there. What we're seeing in terms of adoption, and
what kind of practices and policies are easy to implement
in ci, CD, which ones are harder and where some of the pitfalls
are. And then we're going to build our own pipeline
throughout this. And we're each going to pick your own tools and
talk about what we liked and didn't like, and, and we'll,
we'll show that off. And I dare say, Well, I guess it's we're
open source, we get to put it out there. But we'll have to
document our code this time, I guess, well, we'll do it just
just for this one time, we'll document that code. And we'll
put that up there. That's our plan. If you'd like to see
something different. You know, we're feedbacks always important
to us. So feel free to reach out. Thanks a lot, everybody.
Appreciate it. And
Jamie Duncan: thanks for joining us again.
John Osborne: Thanks, everyone. Stay safe, Happy Thanksgiving.