Tariq Islam 0:07 We do intros first, right? Yes. Jamie Duncan 0:10 Thanks for joining us on the K files. Each week we take on a different topic in the Kubernetes community and the IT industry. Tariq Islam 0:16 Depending on the week, we may find ourselves knee deep in kernel headers, magic quadrants or even startup pitch decks. John Osborne 0:23 Our goal is to close each file with a sense of completeness and satisfaction for us and anyone listening. Jamie Duncan 0:31 Jamie Duncan, I work for VMware. I was at Red Hat for a long time recently moved over to VMware and one of the staff Kubernetes architects there. I'm helping bring Kubernetes into all of the VMware products and I'm all about the fundamental knowledge and been dealing with Kubernetes I think we've all been dealing with Kubernetes Gods since what's been five years now. So john, you're next Who are you? Sure. So my name is john Osborne on the John Osborne 1:00 openshift lead for Red Hat public sector, I work as part of the pre sales teams, helping mostly federal government customers adopt Kubernetes, you know, either in public cloud or on premise. I think, when we talk about companies now, we're usually not even just talking about, you know, what they call entry, or like core Kubernetes really talking about the entire ecosystem. So what it amounts to is a lot of noise for a lot of people that are just kind of coming up on board with a project. So what I essentially do is help customers, you know, with best practices, kind of keep them with the guardrails on a little bit so they can focus on just what they would need to go to get started and build momentum. I think, you know, anytime you go into like a traditional organization, and they don't have cloud technologies, they don't do things like cloud native apps, and so forth. You know, it's really, you got to build progress, you got to build incremental wins. So I kind of help them really early on, you know, set the stage for architectural decisions, best practices, part of like a modernization strategy, or you know, sometimes Greenfield as well so that's kind of what I do. And in tark, what is what is your day to day Tariq Islam 2:00 My day to day Hey, everyone, I'm Tarek Islam, I've been milling about in Kubernetes land with john and Jamie here for going on five years now. But generally helping folks learn and adopt Kubernetes started this journey at Red Hat, actually, and eventually found my way over to Google Cloud where I'm at right now. And I do much of the same thing. A lot of educating different customers, different organizations, helping them to adopt Kubernetes anything in the, in the ecosystem. And at this point, I figured I'd start advertising my thoughts on it. So here we are. Jamie Duncan 2:36 And I think that's a good spot to pull in like a full disclosure. We all started together at Red Hat in the very early days of openshift. While Tarik and I have gone different paths over time, tartan to Google Cloud, and I'm a recent moved over to VMware so I'm still looking for where the bathrooms are. But I think anyone who has spent time at Red Hat you Understand that you never quite lose that point of view. It's always going to color the way we think about technology and the way we think about technology being consumed tart, do you think that's valid? You've been outside the fence longer than I have? Tariq Islam 3:12 Yeah, I think it's completely valid. I mean, the whole open source story that we've all been indoctrinated into Red Hat. I mean, we've we've seen that play out tenfold in the industry at this point, right? Red Hat for a long time as own that message, and coming outside of Red Hat come into Google, that message is is still here as well. And I think that's being brought into other you know, I guess what we would consider old school enterprises like the VMware is of the world and even the oracles of the world. They're starting to adopt and embrace these technologies. So it's essentially it's it's been really fun to see what we professionally grew up in, in our time at Red Hat and seeing that scale out with with really everyone that's coming to this ecosystem at this point. Jamie Duncan 3:55 It's pretty wild. And you can fault Red Hat or you can give them credit for anything that you choose to buy. But that idea if there's innovation happening in it, it's happening in open source. John Osborne 4:06 Yeah. And I think too, like, we all work at different vendors, and we have different competing stacks for, you know, Kubernetes and integration of some of those ecosystem components. But, you know, we've all been doing this for a long time now. So, yeah, there's customers that want to see, you know, how you're different than whatever Ks or whatever other distro, right? But for the, for what we've seen, like the customers that really are successful, it's really about kind of the partnership that you build with them, and actually, like the human human connection, and helping them you know, not just deploy a cluster, but help them make good decisions around getting their apps onto the cluster and making good decisions about their whole cloud strategy outside of what would just get deployed on Kubernetes. And that would involve like storage and networking and you know, what would be involved with like a region to region failover or an ability what you do in availability zone goes out those types of things. So it's, it's a lot more than just the technology I think, you know, when we look at adopting Kubernetes at the I mean, the irony of what you just said is that I think all three of us are introverts. But yet our jobs are to reach out to people and educate them and help them kind of work with their person at varying personalities at a given organization and given enterprise. But yeah, no, I love what we do. I mean, I think all of us are in a similar position right now, just at three different vendors at this point. Yeah, like we do too. It's, we are introverted. Like, I love going to talk to customers, but also, talking to customer all day is like exhausting to write, especially if I'm the only one, you know, especially if it's like back and forth for like, you know, six hours and stuff. But it's also the that's also the most rewarding aspect to it. But then I just have to go like sit in a dark room and like not talk to, like I got for it right. Jamie Duncan 5:40 Sitting in read Nietzsche for a couple hours. So that's, that's a pretty good segue into why we're doing this and why are we sitting down to do a podcast together? Tariq Islam 5:52 Yeah, I think what we're looking at here with this specific podcast, kind of one episode to the Nexus Coming from three different perspective, three different vendors, as Jamie mentioned, it's, it's, you know, we're, we're validating patterns, right, we're seeing patterns emerge. And you know, whether it's one vendor or the next, you know, we we have thoughts around, you know, this whole ecosystem and what I think is going to come out of this more so than anything else is a little bit of guidance, a little bit of information education on what patterns to pay attention to, and how to how to, you know, figure out the signal from the noise because, needless to say, the Kubernetes ecosystem is highly highly saturated at this point. John Osborne 6:31 Tareq and I were walking around coupon, not not this year, but actually the year before and you're like, you know, look at all these startups offering this very specialized, you know, aspect of the cloud native ecosystem. And you know, we don't think they'll be here next year but then like, I went to coop con this year, they were all still there. So it's still a very high note that would call like high noise environment, you know, especially if you look at probably the security aspect is like the most high noise environment and I play pretty heavily in this space like, you know, investigating a lot of the, you know, third party ecosystem providers. For, like container scanning and policies and so forth, and you know, it's hard there's so many choices out there and you know I'm in it quite a bit and I still don't know everything there is to know about it. There's still always like a new player with a new, you know, little specialized offering even in that little specialized segment. So it's, it's a lot. Jamie Duncan 7:19 Yeah, really like tarts phrase validating patterns, because that's really what we're looking to do for this episode, because we're getting our feet underneath ourselves. This morning, we all decided to grab two things from the news pile and just bring them so the the six most interesting things that we could find on our own Twitter echo chambers this morning are two topics for today. But down the road, we are going to have some Thunderdome moments where we're going to pick on each other's technology and debate the merits of whatever decisions were made and decide you know whether they were good ideas or not. And then I'm sure we'll have some people come in and help us around some technologies none of us can grok a particular thing go out and find someone who does and have them explain it to us and figure out you bring in some outside expertise, some outside knowledge. What other kinds of things do we want to do here? Like what other kind of, you know, cheese ball segments? Like, do we want to have like a cake eating contest? What do we want to do? John Osborne 8:17 I think you would argue about some stuff like I saw, you know, some of the articles that came on VMware around like bare metal versus hypervisor. And I was like, Oh, I could argue with Eric Thomas. Jamie Duncan 8:27 Are you talking about the 8%? thing? John Osborne 8:29 Yeah. Where they were like, Hey, we found this one specialized use case where every JVM has like 64 gigs of memory and if you do this Jamie Duncan 8:37 so I just started when that came out. On my little like, the about me email. Yeah, the introductory email for all the new people that accompany I'd said something in a follow on to that, about that. And then in parentheses, it was like TL Dr. numa helps, which is really what that research was. John Osborne 8:57 Oh, absolutely. Jamie Duncan 8:58 If you take something that doesn't do All the numa tagging and pinning, and then the VMware product does do all the numa tagging and pinning for you and make sure that it's on a local, close by CPU and it close by cache and all that. That's where the percent was. Absolutely. John Osborne 9:13 And I've seen them do things like that before. Jamie Duncan 9:14 Yeah. So what it says is numa is good. Yeah, you know, it's not, I don't think it's earth shattering. Although, you know, I was talking to somebody else at Red Hat recently. And they came from the chorus acquisition, and they were saying that they were hiring ops people that had never seen a physical server. So be surprised. You know, some of the younger crowd is not actually sure what numa is, you know, some of those. With cloud a lot of things get abstracted, especially if you're used to working in a data center where you have to worry about power and H back and rack bottlenecking different networks and stuff like that. And a lot of people that has been working cloud their whole career like don't necessarily know what those things are, and like what sockets are, for instance, interestingly, I was just on a call yesterday, where the engineering sneeze for Kubernetes like they just do sort of a Here's the cool things we're working on kind of thing internally, VMware. One of the things they brought up was that in cube one dot 17. It is topology aware. So you can add labels, their their labels specifics, and I can't remember the goals off in the show notes. When we put them out, I'll put the specific links. But in 117, and maybe one of y'all have looked at it, like you can specify rack and server or hypervisor, you can specify like topology elements, like the availability zones, whatever, down to very granular thing. It's a new feature in Windows John Osborne 10:34 17. That's interesting. You know, if you're used to cloud stuff like RAC affinity is not really something we're used to caring about. Jamie Duncan 10:41 It's important in the cloud to it's just, it's not important to you. And that's been my get off my lawn moment for years. John Osborne 10:47 Like, I guess you're talking about availability, you're probably talking like AZ affinity at that point. But usually, if you're talking like RAC affinity, it's more around like the bottlenecks that you have on an individual rack like an on premise environment so you're not as concerned about it. Cloud but of course, like bottlenecks are concerned, no matter where you are, right? Tariq Islam 11:03 They're just they're different kinds of bottlenecks. And I think this is actually the translation that we're seeing that I personally have a little bit of cognitive dissonance around this, you know, where you have these these up and coming engineers that have never seen or touched the server. They're bottlenecks are inherently and entirely different, conceptually, and how they would mitigate that and how they would observe that it's just a, it's just kind of a new, a new paradigm that we're that we're going to be running into at this point. John Osborne 11:29 Yeah, there's a lot of differences with cloud and Red Hat, you know, for us, like most of our customers are probably evenly split between public cloud and on premise, you know, especially if they're doing Greenfield that's probably most likely in public cloud. But you know, we have a lot of, you know, existing on premise customers that still have data centers and other things where they already own all that stuff. So they want to to run there, but I probably say maybe 60% public cloud 40% on premise just just up ballparking it not scientifically, but Jamie Duncan 11:58 that's interesting. That's gonna give me a whole new thing. To think about and ask questions about later, maybe we'll follow in that in the second episode, I'll find some more information about it and share that a little more. But I'll at least get a link into the show notes for this episode. So our first idea was to do like an Amazon reinvent retrospective, and just sort of our favorite things about reinvent. But the new cycle being the new cycle being what it is now, your reinvent might as well have been 100 years ago, what we decided John Osborne 12:23 there was a few things I want to talk about. Still, though, we don't have to go through the whole list. I mean, there was probably 100 hundred announcements. So Jamie Duncan 12:29 was the first day they had like 67 new products launched or something, some ridiculous number, that we can't not talk about some of that. That was gonna be the whole episode. So I think to bring it a little more current to have a little more flavor there. We decided to grab a couple interesting facts off of the news feed. And I think most of us just went looked at our Twitter because that's where the Kubernetes ecosystem lives, we'll find the most interesting things that are out there. So I think tark we're going to start with you on this one and hand the baton off. Tariq Islam 12:57 Well, actually, there was there was a couple things One of the things and we've discussed this recently, and actually this was an inspiration from you, Jamie was was the idea of custom controllers versus operators, right. And, frankly, speaking up until about last week, I was having issues figuring out or grokking, what was what and what I would use which one for. And I ran into a couple of posts the other day that there really clarify things a little bit. And I've been sharing those around as much as I can. But I think the idea behind custom controllers and operators, it's not really a vs here is I think, something that as, as this ecosystem matures, I think folks are really going to start settling on top of Kubernetes with these contracts in a really big way, both at the enterprise level startups. I mean, you name it, this is going to be kind of a new way of not just deploying things, but just interacting with with Kubernetes at large. Jamie Duncan 13:45 Can you answer like a real straightforward question? Possibly. Now that I've been unleashed into the wider Kubernetes world that you've been living in for several years now? Yeah. A little bit of sensory overload and my learning curve is just now kicking in. So I know what an operator is I've given talks about what operators are and why they're functional and why they're good. And I understand what a controller is I've never written a controller, is this a true statement? Does a controller have to have a custom resource definition? Tariq Islam 14:18 So I'll pull it right from from the source that I'm getting it from here. A custom controller is a controller that you would write that acts upon a standard Kubernetes resource. So that being a Kubernetes pod, a service and Ingress deployment, anything within the core Kubernetes API, whereas an operator is going to allow you to write a custom controller that acts on custom resources. So they leverage custom resource definitions. So CR DS as we as the as acronyms go, custom resource definitions apply to define any kind of resource that you want that the Cooper These API will then interact with. So if I wanted to create a Kubernetes resource called car, or, you know, plane and deploy that into my cluster, I would define that through a CR D. Now, if I wanted to do something more operational, and I think I'm preaching to the choir here, but if you want to do something more stateful or operational around how you manage that car or plane object within Kubernetes, that's where the operator comes into play. Okay, that's, that's as that's, that's my understanding. Jamie Duncan 15:29 Okay, I get there. But man, something that two weeks ago, I felt was a very cut and dry definition is not John Osborne 15:37 like, do you feel like it's kind of like semantics. So for, you know, people that have been doing this for too long, because the actual like benefit to the end user really doesn't really matter what we call it right. Jamie Duncan 15:47 Maybe the reason I think it's important though, is that signal to noise ratio that we were just talking about, there's so much information out there, and if they go and read article one, and they're talking they use the term operator or they use the term controller in one way. And then they go read article two and its use completely differently. They're just more confused for it. They went to get smarter, and they just got more confused. settling on vocabulary that sort of generally accepted to me is really important. I'm that person that's going to go read an article and they go read another article, and they've got opposing definitions. And I'm just gonna spiral into annoyance and pain and not want to learn anymore. John Osborne 16:25 Yeah, no, I agree with that. Tariq Islam 16:26 I feel like we should just rename these things. Instead of saying an operator versus a custom controller call it call it a native controller versus a custom controller. I mean, I don't know because native controllers would control native Kubernetes resources and custom controllers would control custom resources. I mean, that's John Osborne 16:41 the ship sailed. I think a little bit I mean, the marketing materials gotten out all the third party companies that have you know, down the operator stuff and really like you know, if I look at the business value of it, it's it is more around, you know, the Red Hat's VMware is in the Google's it's a little bit more for other. We've definitely been benefit, especially if you're running on premise. But you know, it's a little bit more for those third party resources or ISVs. They're going to deploy apps onto the cluster, because they know as long as they tightly coupled their app to Kubernetes, that it's going to work the same way. Whether you know, customer runs in AWS or on premise, I think that's kind of a real big business value for really the ISVs. And that's really why I think there's so many onboard right now. I think there's like, if you've got an operator hub, there's about over 100 operators right now. And it's mostly databases and things that require state but, you know, it would make sense for some, you know, for some customers to that, you know, might need, you know, some real legacy application that needs a lot of hand holding or something like that. Jamie Duncan 17:38 And, john, I don't want to get into the Thunderdome. Is redhead gonna try to monetize operator hub? John Osborne 17:42 No, I actually think we're donating it to cncf we're trying we're trying to Okay, yeah, Jamie Duncan 17:47 I knew there was discussion both ways. John Osborne 17:49 Yeah. So and I'm glad that that's the choice that was made. Now. I mean, Jamie Duncan 17:52 that'll help to be on John Osborne 17:53 Yeah, I mean, we'll have our own, you know, console and things that in openshift, where the operators have been better Did a little bit more but the operator hub we want to make completely community. So I think we're in the process of donating that Jamie Duncan 18:07 was that's what it like. I think like you said like the horses out of the barn a little. Yeah, if we could rewind time and come up with better terminology tark would be 100%. Right. But right now we're looking at what's in front of us. Some people are trying to avoid the word operator because they are not sure if Red Hat is going to try to corner that term. They don't want to get painted with that brush. John Osborne 18:29 Yeah, no, definitely not. Definitely not Jamie Duncan 18:31 awesome. But I hope that's the way it goes. Because I see a lot of value in it with that. So john, what new thing do you want to talk about? John Osborne 18:38 I know coupons like ancient news being like a month ago, but I just wanted to talk about one thing that I thought was pretty interesting from there and then a couple things from reinvent. So you know, at coop con, it was pretty interesting because, you know, I'm pretty introverted. But I did meet quite a number of people at coop con, everyone I met was all a first time visitor to coop con. You know, there's some stats that got sent out by seeing ncf that two thirds of the people that attended the conference, were first timers. And so I felt like there was this kind of huge dichotomy there where there was, you know, oh gee, cu people like us that have been there forever, that we're looking for one thing, and there was brand new people that were kind of new to this whole ecosystem that were looking for another. So there was a lot of people out there saying that that was like one of the keynotes was this was this like open tracing, open telemetry demo, where they were like live coding and go, and all the people that have been around the Kubernetes ecosystem for a long time thought that that was just like, the best keynote ever, whereas the other like two thirds of the people that were brand new or pretty lost, right? Because if you're coming from more of the traditional enterprise side, there was a lot of, you know, there were salespeople there, there were people who were traditional, like sis admins that really new to containers, people who were just kind of writing some code and kind of new to the whole Kubernetes thing. And they weren't writing go. So they were actually they Like some of the keynotes that some of the OD cu people did not think were that good. So there was like this keynote from Vicki Chung, which actually was really good around why we need Kubernetes when we have this other stuff that was working, and it was kind of on this evolution of, you know, as we build things, we end up needing things like load balancing, and replicas, and all these things. And, you know, that really resonated with some of the new people that were into the Kubernetes ecosystem. And there's some other talks, there's some other keynotes, you know, in a similar nature that were more high level that the newer people liked, whereas the Oji people were just wanting to, like, get into the weeds and like learn very, very specific new technologies and not necessarily like the why we need Kubernetes which is really what like new people look for, especially as more enterprises and more big customers sending they're sending non technical people as well. So it's kind of the balancing the two i thought was kind of the most interesting aspect to it. Tariq Islam 20:48 With more non technical folks coming into, into coop con, I feel like they just kind of have to suck it up. I mean, like, for lack of a better term, it is cube con right. It It is a technical convention for Kubernetes and the Kubernetes ecosystem. This is not exactly, you know, a sales kickoff or anything like that. I think it's meant to be technical, it should stay technical and folks that are, you know, there I think I think two layers of that there's folks that are new to the ecosystem that want to be technical. And then there are those folks that are going that are not technical, but are interested in the ecosystem. Now, there may be some overlap there. But I do think that you know, there should be some considerations in terms of sessions for folks that are new to the ecosystem net new but want to be technical, as far as folks that are, you know, coming there just because coop con and Kubernetes in general is like the next big up and coming thing, even after five years, they might just have to, you know, suffer through the the technical nitty gritty details of every session, but John Osborne 21:44 definitely should be tracks I felt like there definitely should be tracks and probably some better guardrails on like what a session actually entails. Now what do some deep dives and it was, there was kind of one on one and then you know, you go to another deep dive and it was like, specifically for Kubernetes contributors that were writing go every day and they just they wrote Go in the session so that there wasn't really a lot of like guardrails on what a deep dive was versus what a one on one was. And I felt like they did probably need more sessions on like, why Kubernetes or you know how this is going to help you in terms of like a modernization program and things like that. But overall, I thought it was it was a good run conference, it was just kinda like the dichotomy of the of the attendees and kind of trying to satisfy both of those was, is I think will be part of the bigger challenges going forward, especially you know, it's growing every year. So don't be surprised if next next year was 20,000 people, right, so Jamie Duncan 22:30 yeah, I know a lot of people were talking about the rejects conference being way more technical and kind of being that refuge. If you wanted, nuts and bolts super hackathon all the time kind of thing. My only thought is sort of non tech, the non technical track inside coop con if one exists or doesn't exist, as long as the as long as the barriers to entry are low. The most important Kubernetes contributor to me, is the next because that's where the next good idea is going to come from. And as long as it's someone who wants to get interested in it. Whether it's you know, release management, whether they're sort of a PM, whether they're a technical writer, whether they're whoever they are, as long as those barriers stay low. And one of the things Kubernetes has gotten right is, it tends to be a pretty inclusive community. Hmm. You know that though the Linux kernel mailing list, you're an idiot kind of mentality never developed inside Kubernetes, at least not to any degree that I've seen. Maybe I'm, maybe my privilege is showing that I've never seen it. I see Kubernetes as an inclusive commune, because it's tried to be from the outset. As long as that continues, as long as that you don't get lost in as long as you don't get lost in the noise. As long as you get lost in the crowd, then I'm game for it. I'm going to try anything else. John Osborne 23:43 Yeah, no, I agree. And the other aspects of too is like the sponsor, keynotes, you know, there was some sponsors that, you know, focused on upstream community, community stuff, whereas there was other sponsors that kind of focused on bad sales pitches. Yeah, so there's there's that aspect to it too, but overall, it was good conference. Some that's ancient Now, you know, the reinvent just kind of finished a couple weeks ago. But, and there's a couple things I want to bring up to the event to I don't know if you guys have any huge takeaways from there, but you're ready to switch gears for a sec. Jamie Duncan 24:10 One of the things that was interesting to me as I was kind of reading the news this morning was very micro and macro, the micro when I didn't think much of it when I first saw it, but then I stopped and thought about it for a few minutes, and it could actually be really awful or really awesome. And I haven't decided which yet, in the latest Kubernetes release, you can there. It's a pre alpha, so we're not gonna see it for a while in any measurable way, but you can put namespaces inside namespaces their hierarchical namespaces as pre alpha Tariq Islam 24:41 yo dawg, I heard you like namespaces Yeah, I Jamie Duncan 24:43 know. Right? The exhibit royalties are about to go through the roof. And the reason it's interesting to me and we were having this conversation a few weeks ago, and I'm not sure that we ever settled on it. Maybe we didn't. Maybe we didn't. But I've talked to a couple other people about the object that is the tenant of Kubernetes has Then kind of confusing, because everyone thought the namespace was the tenant object that if you had a multi tenant deployment of Kubernetes each customer, each tenant got a namespace. And that was how you isolated. And it turns out that isolating by namespaces is hard. John Osborne 25:14 I think if you go there was a couple there's a couple things that I think like if you go back and look at originally when they designed namespaces and you know, for those of you in the openshift world, that's we call it a project is really what's built on a namespace, but it was really built around teams, like one team would have a namespace or another team that have another namespace or lines of business, but not necessarily like multi tenant in terms of you know, this namespace is going to run production in this namespace is going to run test or this type, this namespace is going to run this sensitive workload and this other one won't. So those types of things is kind of one aspect to it. But then you know, the evolution of CR DS you know, everything is an extension and everything is a CRD on top of Kubernetes. Now kind of adds to that namespace trouble because you know, those CR DS are usually where they are like global resources, and then kind of what does that look Like, especially if you go to use something like Helm or something to deploy an app into your cluster, and I only have access to one namespace, but I also need to crud you know, crud capabilities on this on the CRD, right. And then before you know it, you need abilities to create our back or do something else that you really shouldn't have. And then, before you know it, you have a single tenant cluster. Jamie Duncan 26:20 And when that was a big reason behind cluster API, really what we're doing with cluster API, one cluster that can be managed by a team that can deploy multiple clusters, the cluster becomes the tenant object you guys want to know. So is that the tenant object and if that is the tenant object that now we're now we're cookie cutter and clusters out, for each each time we need a new tenant. Then why are we putting namespaces and namespaces for going northbound? Why are we going south? Tariq Islam 26:49 I hear that so this is an interesting development for me, right? Because one of the very first questions I ever got, like from the early days, right back when we were all at Red Hat early The first question I got, can you put namespaces inside of namespaces? flat out? No, five years later now suddenly, you know that that capability is real to your point about Jamie Duncan 27:09 right now, you're a liar five years later. Tariq Islam 27:11 Yeah. To your point about multi tenancy, though, John Osborne 27:14 but should you it's definitely still hard now. Tariq Islam 27:16 Yeah, absolutely. That's your point about multi tenancy though this isn't this is this is, I think, a very underdeveloped part of the ecosystem. And I know from, you know, here, here, where we are at Google, this is actually a very concerted effort on our part in terms of, you know, the fact that you can run clusters as the atomic level of tenancy or namespaces it's kind of one or the other, but you can even get even less granular than that. I mean, it's it. I think it's gonna depend on what people are more comfortable with. I mean, you'll find organizations and enterprises that are phenomenal about you know, declarative ops and, and making sure that they've got, you know, a good knowledge of our back end Kubernetes and how to isolate CR DS and scope things out, and then everyone else is probably going to be more comfortable with that. You know what I want to treat my clusters like cattle and I want to be able to make that my level of tenancy but it's going to be varying degrees of multi tenancy there's, you know, tons of there's tons of room for development in this space I think Jamie Duncan 28:13 like the thing that gave me shivers up my spine was that if namespaces inside namespaces like the are back for that is going to be like the next pearl like write once read never wait. I mean our back is already not the not the best club in Kubernetes is bag. I mean it gets it's something that a lot of people dog on a lot of people have trouble with. So they ignore it, which means it's even worse under. Now, you know that I think that's Tariq Islam 28:39 a failure on our part. Right? And when I say our I mean, I mean I'm talking about how we educate organizations, the role we you know, the industry, right, they're not I'm not just talking about the companies though but also us as folks going into an organization and teaching them about this stuff. I don't think we we we pay enough attention to to our backs I mean, it is actually conceptually simple. You know, I know it's a lot of gamble. Right? Everything is at this point. But I think just the big inhibitor here is understanding how the our back works. I mean, this is something that we see as a blocker for a lot of folks getting on boarded in the cloud in general. How does your Iam work? Right? And that typically takes a lot of education problem, and we don't have a lot of folks that can actually explain it. Not that I can. I'm just saying that it's it's just I feel like if folks could understand how to use Kubernetes our back and how it works. I think we could start getting into more granular levels of multi tenancy right now I just kind of default to you know, what, just have your own cluster, every line of business, have your own cluster in every team have your own cluster. It's just an easier answer. John Osborne 29:44 I think one of the problems with with multi tenancy, too, is there's really like this falsehood that's been around for Kubernetes for five years, is that you're going to get all this crazy density, and you'll be able to throw away all these physical servers if you move to containers. And yes, you can pack a lot more containers onto a server than you would with a VM. But mostly that's around that that's hard, right? Just moving to containers is not going to get you that. First of all, you need to know what your containers how much memory and CPU your containers need, which means you're probably going to have to, you know, run those containers for a couple months and collect, you know, metrics and telemetry and do standard deviations and those types of things to actually get that data because, you know, when vmworld people just guessed, you know, how many CPUs they need, and maybe they just doubled that and then rounded up from that and, you know, a Tomcat Tomcat instance running like a static website, you know, have like eight gigs, a had eight CPUs, and you know, something like that, where it was really unnecessary and moving to containers, they really didn't know how much the route needed. So kind of, you know, this whole idea of, I can go to containers and suddenly throw away a lot of servers really isn't true, at least day one. There's a lot of work to get there. And I think people they want to do multi tenancy because they feel like oh, well, I'll need another control plane and, you know, I'm really supposed to be getting all this density. By just magically move to containers, it's not happening. But then the other aspect of the multi tenancy is really not necessarily even a security boundary, but really kind of the day to day operations of it. So I think, you know, if you look at objectively I think kind of the industry looks at like the managed offerings as Gk being kind of a leader in that space. And you know, with Red Hat, what we're doing with, you know, operators is making it more of operator managed platform, and all that stuff is great, but if you look at, you know, Kubernetes, how it's grown, and that whole ecosystem when you start putting everything that you're running on Kubernetes with sto and you know, development tooling, and building apps and, you know, logging and metrics and all these things like that's more things you need to worry about when you upgrade and kind of having these kind of like huge mono clusters. You know, the more you add them, you can have any managed service in the world. You can have all these operators doing it. But upgrading 1000 node cluster is not going to be easy unless the cluster is running a very similar type of workload, right? Not running an entire it is not running all these different logins. solutions and metrics, solutions and all those types of things. I think kind of the, it makes more sense when you go to look at actually upgrading cluster a lot of stuff on it to have more, you know, smaller, specialized clusters. I think from that point of view, although I still think we're kind of figuring out what that means to manage multiple clusters at scale. So, Jamie Duncan 32:17 so like, it's a little feature that didn't get a lot of press. And I just kind of stumbled across it on some random Twitter account this morning. And it really got me thinking like what are historically I'm not a fan of having 17 solutions for everything. Because that I don't want to have to learn 17 solutions for everything. When I see sort of cluster API getting momentum, like I said, cluster API, and you could hear Tarik salivating. And cluster API is how VMware will be doing business with Kubernetes for the foreseeable future. But this little feature this idea, this pre alpha hierarchical namespace thing they got no press is just sort of a drop in from could really change some ideas. I don't know if I would ever truly recommend hierarchical namespaces as like a go to when a customer asks, hey, what level of multi tenancy should I have? or How should I organize my lines of businesses, I just, you know, the way the direction the industry is going, I feel like with Eks, and aka s and Gk, the new open shift architecture, you name it. I mean, we're treating Kubernetes as a bit of a commodity, right? It is really bubbling up that whole pets versus cattle thing from not just applications, but to the cluster level. So why am I going to go in and recommend you get really solid with hierarchical namespaces inside of a cluster when you don't have? John Osborne 33:35 Yeah, there's a lot of things that customers will ask me and I'll say, Get comfortable running for the platform for a year or two. And then we can have a discussion like in the VM thing is one of those things to where they'll say we run VMs. Now, should we just run this on bare metal? And I'm like, Well, no, but you need to a year or two experience running Kubernetes before you kind of consider going that route. And so I kind of feel a similar things about a lot of the more advanced aspects of Kubernetes like you almost have to have a look Kubernetes docks right now you almost have to have like tunnel vision around what you're going to look at and kind of ignore the other stuff. Because there's, there's so much there that it's gets to be a little bit hard to adopt. So you kind of have to just, you know, focus on the bare minimum. And I kind of feel like I help customers with that a little bit. sutar. What's the second thing that you picked this morning that you found interesting? Oh, this is a good one. From john, actually. So this is a tweet that john put out. Not too long ago. It was around Docker. And this is a this is a personal favorite, actually. So he'd mentioned that a few years ago, I concluded that Docker could not survive as an independent business. So obviously, this is a commentary, a remark on the fact that Docker ended up getting, I don't know, like pseudo acquired, kind of sort of acquired, I don't know what you want to call it by mirantis of all companies, and I hadn't heard about mirantis in years, but, but they they took Docker in and it's just it's almost like so it was almost like a feeling of closure right in a very Long in weird journey for for that vendor. But I felt that you know and john your tweet was was actually in response to to Solomon's tweet actually from yet I just wanted to let someone go around I just wanted to write I just wanted to set them off a little bit but you know, I think like yeah, I think you know that was interesting because you know, if you look at especially with with reinvent like with eBay they're still pushing ECS which is really what it is is Docker as a service you know accepted it's like Docker accepted as customers. And so you know, Docker is helping out the Tariq Islam 35:40 knife john John Osborne 35:44 Well, you know, Docker as a company didn't really have any customers but you know, ECS which is managed Docker offering and Amazon lives pretty strong, right, and has, you know, some decent, some decent traction. So, which is pretty interesting, too. If you kind of think about, you know, how Docker came up and started this whole thing. started this movement, you know, they're, let's face it, they're dead now, right and so, but Amazon has this kind of managed offering, which is a lot smaller subset of what you'd be able to do with Kubernetes. But they still have quite a bit of customers on it. So I guess that technology kind of lives on through through Amazon, you know, monetizing every open source. Technology, it's out there. So Tariq Islam 36:20 I mean, there was a few things to unpack. And that's, that's why I wanted to talk about this. Right. The first the first thing was, you know, I think it's important for folks listening, why did Docker end up the way it ended up? I mean, they had they had this incredible technology, right? containers, again, not not a new tech, right, very old, you know, I'm not going to bother going through the history of it, but they made it usable. They brought it to the user, they brought it to the enterprise. And what happened, like why why couldn't they? They make money? John Osborne 36:50 Well, they definitely did some things of course, you know, wrong. You know, with late adoption of Kubernetes. This tool, the way swarm has implemented some of those things, but I think a lot of people kind of feel like, you know, they had just kind of this arrogance around them where, you know, they weren't really listening to other people, or any other ideas. They, you know, of course, famously wore their shirts that they had at Red Hat summit, you know, kind of creating a pretty big enemy there as well. So there's just kind of the attitude that they have, or had around that space, but they never really differentiated, like they're open source versus their enterprise offering either. An open source is hard, right? Like I think, you know, Red Hat's kind of a one off in that space kind of monitoring, monetizing open source is definitely a challenge, especially with the cloud providers, you know, can create a pretty nice managed, you know, service on on top of that. So ECS is a great example of that, but they didn't really have like a monetization strategy or like an enterprise strategy. You know, they were focused on throwing huge Docker cons and you know, creating this big Docker Hub registry and so forth, but it didn't seem like they had been Like really idea on how to monetize that? Maybe a little simplistic. I don't think anything. I don't think any that's wrong. I think all of it played a role. When someone says what killed the Roman Empire? Like there are 7000 correct answers. The one that stands out to me the most is when I compare it to the previous revolution, looking at the virtualization revolution, you know, like, what, when VMware went to market, it took eight years for KVM, to be on par with performance and feature set to commoditize virtualization. And then it took another two or three years after that for everyone to realize it. So for a decade, more or less VMs did a great job of meeting customers where they are, especially on the upside, right, like once you actually spun up a VM, there was no difference than a physical server, right? It looked the exact same to you. And containers didn't do that in Dockers entire strategy was based around developers. And we know that you know, developers are great, but they don't always make the purchasing decisions in the enterprise, right? And so you have to To win over the ops folks as well, and they didn't really meet the operations folks where they were. And I think kind of that was a big aspect of that if you kind of look at Jamie Duncan 39:08 versus the VM adapt to creating an obstacle, but trying to sell it to developers is also a huge contributory factor. But the thing that blows me out of water is, you know, VMware simply got to print money for a decade. On VMs. before there was any realistic competition. When Docker dropped Docker, you know, when Docker released this container runtime, they were the first to market but it only took about 18 months for someone to have another one that was feature. It was a feature match. So it they didn't have that time to build a monetization strategy. So they had to try to have one on the fly. And monetizing a runtime. It's pork bellies. Like if you have a big enough scale, you can find a profit margin in it but the margins gonna be razor thin, and they were looking for windfall dollars. The thing that I see is that I think the velocity of it, just shoot them up. They help get them The ball the ball rolling down the hill, then they got rolled over by. To me, that's the one that jumps up. John Osborne 40:06 It's really hard to organically grow as fast as they would have had to. And if you look at a lot of the people ended up leaving, like the core Docker people, a lot of them left because of burnout, right? Because they were just, I'd heard stories from people that were literally traveling like 250 days a year, you know, a doctor, like in the early days, you know, they probably would have need to, you know, they famously turned down that $4 billion Microsoft offer but they probably would have had to get acquired by somebody that would have the scale to get them, you know, that that speed that value to market faster, because they just weren't gonna be able to do that. Jamie Duncan 40:40 Yeah, I've got a reputation online for being a Docker basher, so I'm not gonna weigh in too much more than that. So john, what's the thing you want to bring up? what's what's got your attention today? John Osborne 40:49 Yeah, I think, you know, reinvents, you know, they have this enterprise focus this year. And I thought it was really interesting because if you look at the cloud providers, a lot of what they're doing is kind of making this massive push on premise. And the three major cloud providers, you know, they have really this kind of global presence. At reinvent, you know, Amazon first announced this thing called Global accelerator. And what it is, is basically a way to route traffic from your own data center or wherever your own user is, into their network very fast. Now, Microsoft and Google, they already do this for free, actually, where, you know, if you look at the way traffic is routed around the internet uses BGP is kind of the lingua franca of the internet. And the way that that Amazon works is they basically route traffic over the public Internet, as far as it'll possibly go, until it gets to the region that the server serving, right? What Google and Microsoft do is they route traffic into the closest region to the end user. And then it travels throughout their backbone to get to the server. And that's how like, if you're familiar with like, you know, CloudFlare or Google DNS, you know, you type in 8888. It kind of works the same way globally, the way they They do it is once you, once you get the traffic into your own network, you can kind of do all these things with it, which is, you know, really interesting. And it might even go to different servers at that point. But I feel like with the global accelerator that Amazon released now to where you kind of like pay this extra money, and you know, everything gets into the Amazon network right away. It's barely traversing the public Internet at all. And this is how Google and Microsoft do it, too. And if you look at that, you know, it's basically just making a very small hop from wherever your end user is into their mesh. And then if you look at what they announced with outpost, which is basically this kind of like rack of servers, that extends a region on premise, and this announcement called local zone two, which is kind of just for LA at this point, but it's basically a bunch of outposts, you know, more racks of servers. It's like a local availability zone. You know, we're almost getting to the point where these public clouds are going to have their own internet, right, like they're barely or maybe even at some point, you won't even traverse the public Internet at all from your end users. To get to its destination, there's a lot of benefits around that, you know, you don't won't have to deal with things like DNS, you know, caching and Time flows and stuff as much, you know, once it gets gets into their network, but I felt like that was really interesting. You know, they announced with outpost, managed services around, you know, Eks and ECS, as well, although I still kind of feel like the interesting things around that were the control plane still lives in the region. So I wouldn't call it an edge offering. It's just part of when people say, like multi cloud or hybrid cloud, you know, I really think of the control plane, which is kind of managing that not where the data plane is. But you know, it provides some users like especially in regulated industries to do like some latency sensitive workloads and things like that. So I felt like that was pretty interesting that we want internet run by the public Internet, then China's kind of creating a secondary internet and then these public clouds are kind of creating almost their sub internets as well. So it's kind of interesting that Tariq Islam 43:54 I was going to say, since you since you triggered me the the fact that like You said that every provider is going to have their own internets, effectively how we've been operating way, way before I joined Google. I mean, we literally just turned on our infrastructure for enterprise consumption. But really, it's the same infrastructure that the internet has practically been running on for for all these years. But that being said, global accelerators is effectively meeting what Azure has what GCP has in terms of reducing the number of hops that it takes for request to get into a given providers network. And I think, you know, I don't know about Microsoft, I know for GCP, as you mentioned, it's free of charge. I don't know what the cost structure or cost model is for AWS. But to your point about parity between the cloud providers, I think there's absolutely truth in that. Right. I think, I guess that's part of why we're doubling down on Kubernetes and why I think it's important for a lot of vendors, I think we're seeing the same thing with VMware, right with Project tanzu. doubling down on the Kubernetes is really going to be where the new value add comes into play when we talk about okay provider x y&z See all have pretty much the same features and functions in terms of the value that I'm getting out of cloud. But now we need to make sure that I can move around. And so I think Kubernetes gets this, you know, a good a good portion of the way there. Jamie Duncan 45:14 The part that got me thinking was that you go from your ISP directly to the cloud providers, connectivity to their backbone. Like Tarik said, that's kind of been where we are for a while, like, when you're watching your Netflix videos, most of the large ISP is have giant racks and racks of Netflix, high speed connected into their own pointspread into their own pops. But like Tarik said, Google's network has been global for years. What we're seeing is the commoditization of it, and the being able to do that as a bootstrap startup, to be able to provide a global presence, which, when I was doing big web 1011 years ago, just getting a CDN to work was hard, doing like a global presence that had didn't had anything short of just massive comedic latency. was really, really hard to do. And now we're seeing it commoditized is pretty cool. fargate I've seen two takes on it. I've seen one take that says that's a really interesting idea. And then the other people say, holy crap, that is some serious ugly duct tape trying to glue Kubernetes into things where it shouldn't be glued in ways it shouldn't be glued. I haven't done enough research to have a real good opinion on it. I'm not a giant fan of serverless as a general purpose, business model, neither am Tariq Islam 46:26 I. But I do see the value for developers and developers only Frankly, I mean, I mean, I mean, fargate You know, when it's micro VM concept and things like that, it's great, where you can just bring your own container, right. That's effectively what we did with with cloud running k native. That's, that's, that's the whole idea. Now, I understand why AWS is doubling down on fargate. I just think that that's it's just whether you put it on Eks or not still a you know, a form of locking. I mean, that's the value of k native and I know Red Hat's looking at K native. I know everyone's looking at K native at this point. The idea that you can stand ups Something like a fargate on Kubernetes, wherever Kubernetes is supported, which is everywhere. That's, that's valuable, right. So now I can take my serverless experience for my developers and take it wherever I want to. I'm not really sure what fargate gets me above and beyond that, frankly, John Osborne 47:16 if you're running Kubernetes, I think that k native is going to get you especially for, you know, we say serverless, you know, I think event driven, right. And so with K native, you can kind of get that aspect to where you, you've spun down to zero, and then you have some web requests or something else that comes in and kind of spins up a container to serve that workload. And that's not really what fargate is, I watched their demo on it. And they basically have at least for the integration with Kubernetes, where, you know, you basically have labels and the mostly map to namespaces. And whenever you whenever you deploy a workload into your cluster, anything that goes into that namespace goes out into outside your cluster and into this big shared fargate cluster that Amazon runs It runs within a KVM instance are talking about using firecracker in the future, you're basically it runs in a micro VM. And then it has a VPC that kind of maps that back to your own cluster. That might get you something from a cost perspective. But there's lots of limitations around that with you know, it has to be stateless, and other things like that. And it's gonna run all the time. It's not necessarily event driven, it'll scale up for you, because you won't have to worry about things like the cluster auto scaling for nodes that's inside, that's inside a Kubernetes cluster. So might be able to scale a little bit better, but it's not necessarily event driven in that aspect. Jamie Duncan 48:33 This is a little bit of a tangent, and maybe we can follow up on this in the second episode, if it's interesting to y'all. But it's a open question I posed to some people and everyone knows it's happened, but we don't know why it happened or if it's really happened, or if it's just sort of, again, that whole half 17 solutions for everything. But a year ago, the really cool tech was all around coop vert, where we were taking containers. Taking pods and running a VM inside them. And couverte was the KVM based virtualization, but there are all sorts of different things where we were putting virtual machines inside pods. Now, and this is what we're doing with Project Pacific and pod VMs. Inside, you know, it VMware. It's firecracker, like you were talking about micro VMs, all this different stuff now, sort of the trend is we're gonna take a tiny virtual machine and put a pod inside it. When did that change? Like, where Why are we? How did we pull that one ad so fast and so quietly? Or did I just miss that meeting? Tariq Islam 49:35 I think we all missed that meeting, frankly. Jamie Duncan 49:38 But we did right. like everyone's got that initiative right now. Just a year ago or less. Cooper was the coolest ID Tariq Islam 49:45 Yeah, it was a flash in the pan. I just I don't think there was a good way to make it consumable Jamie Duncan 49:49 because VMs are really easy to make weird. Yeah, like five interfaces and 14 different discs and all these kind of weird things going inside them, mapping all of the weirdness of a V into a container is hard use containers. A container has one interface, a container has one storage with a bunch of bind mounts, I get why putting a VM in a container at production scale, and it will not scale. But complexity is hard. I'm just wondering, like when did that switch happen? Or hasn't happened? Are we still are people still driving Cooper? John Osborne 50:23 I can tell you at Red Hat like there's definitely some traction behind it, you know, running virtualization in general, I feel like you have to own that is layer. So if you look at the public clouds, you have to, you know, support nested virtualization, right? Like I can't just go run a VM inside of an easy to instance, right? Double nested virtualization is blocked in most public cloud providers. So if I want to run couverte you know, it's really an on premise bare metal thing. And you there were some technical challenges and when it comes here, you know, maybe in the future it'll be awesome. But, you know, we talked about like meeting customers where they are today, right? Like most customers are not on premise. on bare metal, their most customers are in public cloud. And if they're on premise, they're probably using VMware. Right. So, you know, when we talk about meeting customers where they are, there's not like a huge demand and the type of the type of customer that would be running on premise on bare metal, has been running Kubernetes probably for a few years and is a pretty advanced customer. And we see them now and then but now they're, that's not where most customers are at, like in their journey for Kubernetes, let's say, Jamie Duncan 51:23 the fargate announcement at reinvent, that was really when I got to thinking about it. We were spending all this time to put VMs in pods, and now we're putting pods in VMs. Again, I wonder if there is a solid comparison there. By the way you do it, you know, it's really around, you know, what's the management plane, are you controlling those using the current API or some sort of declarative fashion. So the last little news bit that I want to pull up and this is a little bit of a plug for a co worker of mine, Steven Augustus, who is in the Kubernetes community is one of those guys that does a lot of the chopping wood, carrying water. To make Kubernetes happen, he chairs CIG release. So he's the guy that actually gets the releases out the door for every point release. And then he does a mountain of other good work. Like he's just an awesome dude. On top of that he's really funny and really good fun to hang out with. He put an announcement out the other day that they have an I don't know if this is true, and maybe y'all have seen it more than I have on the Kubernetes release team for so for SIG release. They have official shadows, where you can go learn what they do, and learn how to use the tooling. And it's an official observed thing where you're part of the team as this sort of shadow roll. And you're not I guess, if it breaks, you're not on the hook, but you're like you can come in and there's a codified way to go learn how to do this process. I tell customers all the time. Maybe the only thing right now more impressive than Kubernetes in our industry is the tooling and the test rig for Kubernetes because to me, it's just stunning. The way integration testing works, the CI CD workflows, the release lifecycle, the release processing, it's amazing to me, that mechanism works as cleanly as it does. And you can actually go be a shadow, like you can go learn how all that stuff happens. And that team is dedicated enough to teaching other people how to do it, that they have an official process. So you actually have to go apply to be a shadow. I don't know if you John Osborne 53:20 saw the notes that came out from the K native announcement that they were trying to get some more contributors and open up some some more contributors that came out just last night. But they had a similar thing where they I think they said they were going to if I remember correctly from reading it, Turkey might know better than me, but they're actually going to have mentors in the K native community now that were willing to mentor and shadow people that were that were new, and were like looking for ways to contribute. Jamie Duncan 53:46 I just think it's awesome. And I was talking earlier about making sure that those barriers stay low. And this is one of those days and I'm not a hippie dippie guy I like to go solve the problem in front of me, but I love the fact that this community does that. Like they think forward enough, and they think enough of what they're doing, to build in ways to teach other people how to do it. Because that's how that's how good ideas happened to me like that, that that type of practical inclusivity to me is really important. And I really I commend that team for doing going back to Tariq Islam 54:18 the the earlier part of our discussion. I think that might be something that that that could be pushed, pushed to coop con, as a way for folks to potentially learn more or get involved at least teasing out, you know, a small subset of that crowd. I think if they invest in that model instructor, I think Kubernetes could go even further as a community and as a technology. John Osborne 54:39 Yeah, I agree. I mean, you know, especially over the years, of course, like whenever you have multi vendors, you know, trying to sell stuff, there's always gonna be like quarreling and stuff like that. But, you know, whenever there's been some sort of issue or disagreement on something, I think if you actually go look at what's actually happening in the Kubernetes SIG, that's just kind of noise from the outside right like the engineering and the actual community that's helped building a product is separate from like the sales teams in the in the marketing and so forth. So and that's pretty cool. Jamie Duncan 55:08 All the vendors are still represented in all those groups the way they're able to filter that garbage out. And we're all kind of like we're we've all done pre sales, like we're all on the front end of that world, sort of the mud slingers, and watching that not get into the into the groups that actually built this tool to me is pretty amazing.