WEBVTT 00:00:12.100 --> 00:00:15.540 Scott: Welcome to the Bikeshed Podcast, where we talk about all things software engineering, 00:00:16.420 --> 00:00:19.500 Scott: tech treats, and opinions that will change in 24 hours. 00:00:20.560 --> 00:00:25.040 Scott: I'm your co-host, a man who says please and thank you to AI, Scott Kaye. 00:00:25.720 --> 00:00:27.600 Scott: And alongside with me are my co-hosts. 00:00:28.080 --> 00:00:32.439 Scott: He once refactored a repo so hard, it checked itself in to therapy. 00:00:33.100 --> 00:00:35.380 Scott: Matt Hamlin and my other co-host. 00:00:36.200 --> 00:00:38.600 Scott: With takes so spicy, they're seasoned to perfection. 00:00:39.380 --> 00:00:41.740 Scott: Dillon, spicy take curry. 00:00:42.520 --> 00:00:45.100 Scott: Gentlemen, what are we cooking up today? 00:00:48.280 --> 00:00:48.960 Matt: I like how you outed 00:00:48.960 --> 00:00:49.420 Matt: yourself as 00:00:49.420 --> 00:00:53.180 Matt: wasting so many tokens in your AI usage, Scott. 00:00:53.400 --> 00:00:55.480 Matt: You wasted on please and thank you. 00:00:56.400 --> 00:00:56.640 Scott: Yep. 00:00:57.440 --> 00:00:58.320 Scott: They say don't do that. 00:01:00.120 --> 00:01:01.900 Matt: Yeah, how many millions per day or something 00:01:02.080 --> 00:01:05.800 Matt: was OpenAI burning through just on ChatGPT saying thank you? 00:01:06.040 --> 00:01:08.540 Matt: It's like a couple million or something a day, I think. 00:01:09.200 --> 00:01:10.100 Scott: I don't remember the number. 00:01:10.280 --> 00:01:11.580 Scott: I just remember the article. 00:01:11.800 --> 00:01:16.240 Scott: I don't explicitly type thank you as its own message. 00:01:16.580 --> 00:01:17.860 Scott: I'm not that crazy. 00:01:21.280 --> 00:01:23.880 Matt: You're kind of toeing the line between being overly generous 00:01:24.220 --> 00:01:26.140 Matt: and maybe not being mean to it. 00:01:26.300 --> 00:01:29.720 Matt: So when they do have an uprising, you're not first in line to... 00:01:30.820 --> 00:01:31.440 Scott: I'm semi 00:01:31.440 --> 00:01:32.280 Scott: -considerate. 00:01:37.600 --> 00:01:38.000 Scott: Morally 00:01:38.000 --> 00:01:38.240 Scott: gray. 00:01:41.380 --> 00:01:44.160 Matt: This is the first time we've gotten together to record. 00:01:44.230 --> 00:01:47.100 Matt: I think maybe at least two and a half months, maybe three months. 00:01:47.900 --> 00:01:51.440 Matt: We had our good summer break, and now we're entering into season two. 00:01:53.660 --> 00:01:59.800 Matt: We had a podcast, I think, lined up, but I think we haven't wrapped it up or finished editing it. 00:01:59.800 --> 00:02:03.660 Matt: So this might be a little bit rusty. 00:02:04.280 --> 00:02:11.440 Matt: I don't know if I can pick up the podcasting hat and put it on again as efficiently as it was a couple months ago. 00:02:13.880 --> 00:02:14.900 Scott: Yeah, the last... 00:02:15.520 --> 00:02:16.800 Scott: It wasn't three months. 00:02:17.820 --> 00:02:19.280 Scott: We released one in June. 00:02:22.520 --> 00:02:23.400 Matt: But we recorded in 00:02:23.400 --> 00:02:23.680 Scott: May. 00:02:25.400 --> 00:02:26.460 Scott: Oh, was that in May? 00:02:26.640 --> 00:02:28.620 Scott: We were releasing and recording them before. 00:02:29.000 --> 00:02:29.500 Scott: Are you sure? 00:02:30.600 --> 00:02:31.240 Scott: I'll tell you right now. 00:02:31.260 --> 00:02:31.360 Matt: I 00:02:31.360 --> 00:02:31.740 Matt: think so. 00:02:32.900 --> 00:02:35.680 Matt: We did record one in June, but we haven't released it. 00:02:36.600 --> 00:02:37.620 Matt: It's been sitting on the back burner. 00:02:38.200 --> 00:02:38.720 Scott: Yeah, it's been a while. 00:02:38.740 --> 00:02:39.420 Scott: Maybe it was a good topic. 00:02:39.520 --> 00:02:39.900 Matt: I don't know. 00:02:42.420 --> 00:02:45.360 Matt: We'll save that for the vault in 50 years, like Disney does. 00:02:45.520 --> 00:02:49.160 Matt: We'll release a vault of unreleased content and it'll be there. 00:02:51.320 --> 00:02:52.420 Scott: We can release it. 00:02:52.890 --> 00:02:53.580 Scott: Let me find it. 00:02:54.920 --> 00:02:55.540 Matt: Now we have 00:02:55.540 --> 00:02:56.480 Matt: to save it for the vault, Scott. 00:02:56.700 --> 00:02:58.480 Matt: We can't tease it out 00:02:58.920 --> 00:03:00.820 Matt: and get people in the subscriptions and whatnot. 00:03:01.980 --> 00:03:02.500 Scott: The vault. 00:03:05.100 --> 00:03:06.040 Scott: Dillon, what were you saying? 00:03:07.380 --> 00:03:07.940 Dillon: I don't even know. 00:03:08.450 --> 00:03:10.700 Dillon: I feel like we haven't done this 00:03:10.730 --> 00:03:12.080 Dillon: in so long that I forgot how. 00:03:15.360 --> 00:03:15.660 Dillon: Yeah. 00:03:16.820 --> 00:03:17.840 Dillon: You're going to have to teach me how to talk. 00:03:22.540 --> 00:03:28.220 Matt: all right i think our main topic today around is that we want to cover at least is um around 00:03:29.000 --> 00:03:35.100 Matt: well we were having this sort of you know friendly debate banter around how much should you optimize 00:03:35.500 --> 00:03:40.020 Matt: your uh deployment pipeline and scott correct jump in if i'm like sort of mischaracterizing this 00:03:40.240 --> 00:03:40.600 Matt: this topic 00:03:40.600 --> 00:03:45.580 Scott: i still think we're talking about characterizing it already perfect all right 00:03:45.500 --> 00:03:46.620 Scott: Go ahead. 00:03:47.200 --> 00:03:47.980 Matt: No, you go ahead. 00:03:48.240 --> 00:03:48.340 Matt: Okay. 00:03:48.940 --> 00:03:49.080 Matt: All right. 00:03:49.140 --> 00:03:56.320 Matt: My perspective of it was that we were talking about like teams depending on size, scale, 00:03:56.840 --> 00:04:03.400 Matt: capacity, whatever, shouldn't overly optimize their deployment pipeline beyond a certain 00:04:03.640 --> 00:04:04.280 Matt: point, right? 00:04:04.400 --> 00:04:09.180 Matt: They should like sort of hit a happy medium or whatever, you know, maybe medium is the 00:04:09.100 --> 00:04:16.160 Matt: but like sort of a good enough place and then care more about other things than just how fast 00:04:16.160 --> 00:04:22.740 Matt: they're deploying their changes um i mean again maybe i'm like yeah maybe i'm you know not painting 00:04:22.750 --> 00:04:24.180 Matt: the right picture there um 00:04:24.180 --> 00:04:28.980 Scott: no that's that's kind of close but like it's more so in my eyes 00:04:30.700 --> 00:04:35.159 Scott: there are other things that need to be optimized there are other work so so this all start stemmed 00:04:35.180 --> 00:04:40.360 Scott: from a conversation we were having in our discord group where a certain company wants to get their 00:04:40.440 --> 00:04:45.320 Scott: deploys to five minutes. I have worked at this company and I know about a lot of the problems 00:04:45.560 --> 00:04:52.780 Scott: there. And my hot take here is that deploying in five minutes kind of sacrifices a lot of other 00:04:53.460 --> 00:04:58.260 Scott: stability potential metrics, and it's masking a lot of other problems that this company has. 00:04:58.760 --> 00:05:05.140 Scott: And it's kind of a band-aid solution to try to solve problems. So you're not solving the 00:05:05.140 --> 00:05:07.020 Scott: problem itself, but rather like a symptom. 00:05:08.180 --> 00:05:09.140 Scott: So there 00:05:10.180 --> 00:05:11.340 Matt: is bleep hair. 00:05:12.699 --> 00:05:13.580 Scott: I think 00:05:13.580 --> 00:05:14.200 Scott: it's great 00:05:14.200 --> 00:05:15.040 Scott: if you want to 00:05:15.340 --> 00:05:17.180 Scott: optimize pipelines as far 00:05:17.200 --> 00:05:19.100 Scott: as you can, but I think you should 00:05:19.120 --> 00:05:21.100 Scott: do 80% of the work and not that 00:05:21.260 --> 00:05:22.940 Scott: 20% that takes 80% of the time 00:05:23.160 --> 00:05:25.080 Scott: sometimes. Like on a team that owns 00:05:25.300 --> 00:05:25.840 Scott: so much, 00:05:27.600 --> 00:05:28.920 Scott: being so pigeonholed 00:05:29.120 --> 00:05:31.080 Scott: on these pipelines 00:05:31.300 --> 00:05:32.920 Scott: need to be five minutes, I think 00:05:33.639 --> 00:05:35.120 Scott: that's a great 00:05:35.140 --> 00:05:41.220 Scott: overarching goal, but not something you should, uh, it's putting the cart before the horse 00:05:41.720 --> 00:05:47.020 Scott: and investing like all your eggs in that basket is not like the, the end all be all metric that 00:05:47.100 --> 00:05:51.560 Scott: will do things. Um, we'll get a little bit more into some specifics, but yeah. 00:05:52.660 --> 00:05:55.600 Matt: I was gonna say, let's just quickly step back for folks that maybe aren't familiar 00:05:56.100 --> 00:06:00.600 Matt: with the topic or with the discussion. Um, so if, if, if you're running a company that primarily 00:06:00.620 --> 00:06:05.680 Matt: has a web presence, like a website, you end up needing to deploy that website. And you frequently, 00:06:05.840 --> 00:06:10.300 Matt: those companies deploy often, you know, multiple times a day. So you, you know, code up some changes 00:06:10.380 --> 00:06:14.620 Matt: and you want to release it to your customers. Usually that process is like goes through a 00:06:14.820 --> 00:06:17.880 Matt: continuous deployment pipeline. You know, there's companies that don't do continuous deployment, 00:06:18.100 --> 00:06:23.140 Matt: they sort of make scheduled releases. You kind of see those in companies where, you know, maybe your 00:06:23.180 --> 00:06:27.040 Matt: bank has like a little banner saying that they, you know, expect downtime, probably because they're 00:06:27.060 --> 00:06:31.840 Matt: doing a big release. But for companies that don't do that, they usually deploy frequently multiple 00:06:32.020 --> 00:06:37.420 Matt: times a day or multiple times a week, right? That deployment process is just like building that code 00:06:37.600 --> 00:06:42.360 Matt: and then putting it onto servers that customers interact with. So when we talk about, you know, 00:06:42.640 --> 00:06:46.640 Matt: deployment times or deployment lifecycle is like sort of what's the time it takes for an engineer 00:06:46.840 --> 00:06:48.380 Matt: to basically check in their code, 00:06:48.620 --> 00:06:48.940 Matt: you know, make 00:06:48.940 --> 00:06:51.280 Matt: a change and then sort of push it up and then have 00:06:51.380 --> 00:06:57.020 Matt: that code be built and then released to customers. You know, maybe that's like sort of glossing over 00:06:57.040 --> 00:07:00.660 Matt: Like there's a lot of different metrics within that, but that's like the rough concept for 00:07:00.730 --> 00:07:05.060 Matt: people that aren't tightly familiar with the whole life cycle. 00:07:06.310 --> 00:07:06.680 Scott: Great point. 00:07:06.790 --> 00:07:09.180 Scott: And I think that view can even be challenged. 00:07:09.250 --> 00:07:14.340 Scott: But I want to also talk about, we're talking about like corporation level companies that 00:07:14.460 --> 00:07:17.680 Scott: take in a lot of money using their platform every day. 00:07:18.170 --> 00:07:27.000 Scott: So we're not talking about your personal website or maybe even a small business startup app 00:07:27.020 --> 00:07:32.040 Scott: you need to iterate really quickly to stay ahead of the game, to show that you're building 00:07:32.160 --> 00:07:35.280 Scott: new features to get money and funding. 00:07:35.740 --> 00:07:39.220 Scott: We're talking about already established sites 00:07:39.220 --> 00:07:40.180 Scott: that have a 00:07:40.180 --> 00:07:41.520 Scott: lot of marketing behind them. 00:07:42.120 --> 00:07:45.920 Scott: And are they executing? 00:07:47.660 --> 00:07:48.700 Scott: They're taking in a lot of money. 00:07:48.900 --> 00:07:52.600 Scott: So at that point, what is the execution work? 00:07:52.600 --> 00:07:55.840 Scott: They're already established for what they're building and what their brand is. 00:07:56.340 --> 00:08:02.900 Scott: Now we want to weigh and balance between churning out features and making sure it's stable. 00:08:04.800 --> 00:08:09.120 Dillon: What's the biggest reason to want to deploy or deploy within five minutes anyways? 00:08:10.400 --> 00:08:10.900 Scott: Great question. 00:08:13.600 --> 00:08:21.640 Matt: My take here is five minutes is sort of just a number that we were talking about maybe in that discussion at that time. 00:08:22.180 --> 00:08:29.480 Matt: my take is that you should continuously push for faster and faster deploy times so and the reasoning 00:08:29.600 --> 00:08:35.919 Matt: being that the faster you can release a change the faster you can fix a bug and the faster you can 00:08:36.539 --> 00:08:42.020 Matt: iterate on your product right like if if for example if for example it took you a day 24 hours 00:08:42.180 --> 00:08:46.559 Matt: right you you merge some changes and it took 24 hours until that change was seen by customers 00:08:47.080 --> 00:08:52.800 Matt: your cycle for fixing a bug one is like okay it takes a little bit to debug like what is happening 00:08:53.060 --> 00:08:57.900 Matt: what is the bug and then a whole day to actually fix it right which might cost companies a decent 00:08:58.040 --> 00:09:03.860 Matt: amount of money um and and but the same with like iterating on your feature like it takes 24 hours 00:09:03.970 --> 00:09:08.940 Matt: just to like learn from a change that you might make and see how it impacts your customer behavior 00:09:09.190 --> 00:09:15.699 Matt: right again 24 hours is a big stretch right like most ci or cd pipelines are probably a lot faster 00:09:17.500 --> 00:09:18.480 Matt: but I think it's just like a 00:09:19.020 --> 00:09:20.620 Matt: like the faster and faster you make that 00:09:20.770 --> 00:09:22.640 Matt: right the sooner you can start 00:09:22.690 --> 00:09:24.600 Matt: to iterate on your changes and release them 00:09:25.080 --> 00:09:26.560 Matt: and get feedback on them 00:09:27.620 --> 00:09:28.540 Matt: so that's like 00:09:28.930 --> 00:09:30.420 Matt: you know five minutes again magic number 00:09:30.720 --> 00:09:32.740 Matt: pulled out a hat more so just like keep pushing 00:09:32.980 --> 00:09:34.440 Matt: it to make it even faster 00:09:34.640 --> 00:09:36.920 Matt: so that you can get those learnings 00:09:37.320 --> 00:09:38.840 Matt: sooner on those changes 00:09:39.540 --> 00:09:39.820 Matt: so that's like 00:09:39.820 --> 00:09:40.400 Matt: the core driving 00:09:40.760 --> 00:09:42.180 Dillon: when you say keep pushing it 00:09:42.580 --> 00:09:44.120 Dillon: my mind goes to who 00:09:44.800 --> 00:09:45.680 Dillon: is it 00:09:45.700 --> 00:09:49.820 Dillon: Is it like the engineers that are working in that repo that has those pipelines? 00:09:51.140 --> 00:09:59.100 Dillon: Or is it like some other team within the company that's like helping do DevOps for the team? 00:09:59.520 --> 00:10:03.040 Dillon: That's kind of organizational structure, but it's interesting to me. 00:10:03.240 --> 00:10:05.640 Dillon: Because I'm like, man, I just want to focus on the features. 00:10:06.500 --> 00:10:07.020 Dillon: What am I doing? 00:10:08.540 --> 00:10:14.240 Matt: Yeah, this is where I probably maybe branch out to the broader topic that I had originally brought to us to discuss. 00:10:14.680 --> 00:10:18.000 Matt: And so maybe we'll just talk a little bit briefly about it, but then dig back into it. 00:10:18.780 --> 00:10:26.420 Matt: My broad sort of thought is that like the entire company should be set up in such a way that it moves as fast as possible in the right direction. 00:10:27.000 --> 00:10:38.860 Matt: So I call this like organizational velocity or like high organizational velocity, meaning like everything you should do, you should be thinking about how can we do this faster and ensure that we're doing it as fast as possible in the right direction. 00:10:39.100 --> 00:10:44.000 Matt: right so like that means in my mindset at least like you know if you're a developer that's 00:10:44.360 --> 00:10:48.620 Matt: maintaining just a little feature on the site like even your responsibility at that point is still 00:10:48.760 --> 00:10:53.700 Matt: like how do I iterate on this feature as fast as possible to get those learnings in order to move 00:10:53.700 --> 00:10:57.420 Matt: in the right direction and like keep that pace right so like I think it's like sort of a 00:10:57.680 --> 00:11:03.720 Matt: responsibility of everyone to try to do that um and and I think that's just like a that should be 00:11:03.720 --> 00:11:08.820 Matt: like sort of a, you know, a universal truth or like sort of a, you know, value number zero of 00:11:08.820 --> 00:11:13.120 Matt: your company of like, you move as fast as possible in the right direction. I think that just sort of 00:11:13.260 --> 00:11:16.700 Matt: delivers on, you know, even outside of deployment and learning from your customers, it's like, 00:11:17.060 --> 00:11:20.580 Matt: just delivers on, you know, like internally, you can make decisions faster if you're, you know, 00:11:20.840 --> 00:11:25.260 Matt: cut meetings to 15 minutes or something instead of an hour or whatever, right? Like there's a number 00:11:25.260 --> 00:11:28.900 Matt: of other things outside of deployment where it could touch on, but I think it's sort of, 00:11:29.380 --> 00:11:32.660 Matt: if you start with that as a first principle, then sort of it gets into deployment and say, okay, 00:11:32.900 --> 00:11:37.520 Matt: yeah, deployment, we want to iterate as fast as possible on our product. And I think if you start 00:11:37.530 --> 00:11:42.380 Matt: with that as like an initial founding, sort of a value zero or whatever of your company, then 00:11:42.550 --> 00:11:46.920 Matt: like everything else sort of follows of like, if you're iterating fast, you can learn fast, you can 00:11:47.420 --> 00:11:52.640 Matt: build things that maybe your competitors aren't able to do at the same time. So that's, you know, 00:11:52.860 --> 00:11:57.460 Matt: that's like the broad picture I had, but like we can zoom back down into deployment and like, 00:11:57.820 --> 00:12:01.159 Matt: you know, to your question, Dillon, it's like, I sort of, I sort of see it as like 00:12:01.840 --> 00:12:05.240 Matt: anyone's responsibility. Like if you are participating in that part of the ecosystem, 00:12:05.370 --> 00:12:09.040 Matt: like I wouldn't expect someone from sales to come in and say, oh, let me like fix this 00:12:09.560 --> 00:12:13.300 Matt: deployment pipeline to make it faster. But I would expect everyone to have a vested interest 00:12:13.680 --> 00:12:19.200 Matt: that is deploying or using that process to try to improve it. Again, that's not, you know, 00:12:19.340 --> 00:12:24.540 Matt: I'm not saying that someone that, let's say works on backend should spend a lot of time trying to 00:12:24.590 --> 00:12:28.459 Matt: figure out how to build, how to make front end builds even faster, right? It's like contribute 00:12:28.480 --> 00:12:35.840 Matt: to the part you know um but uh yeah yeah i'm sure there's like some flaws that we can pick at that 00:12:36.260 --> 00:12:36.620 Matt: that thinking 00:12:36.620 --> 00:12:41.820 Dillon: i agree with you like this is something that when i see engineers having 00:12:41.950 --> 00:12:47.720 Dillon: headaches with it um i usually tend to just like jump on it and see like what can i do 00:12:48.350 --> 00:12:55.040 Dillon: to find new ways to to speed this up is there like inefficiencies in our pipelines that are like 00:12:55.340 --> 00:13:01.900 Dillon: low-hanging fruit um and and maybe i'll try to fix it but also just like raise it to the 00:13:01.920 --> 00:13:05.160 Dillon: the org and be like does anyone have any ideas on how we can improve this 00:13:05.820 --> 00:13:09.080 Dillon: because it's causing headache it's especially when it's causing headaches i think when the pipelines 00:13:09.140 --> 00:13:15.640 Dillon: get to a speed where people aren't complaining that's like a good spot to be but also it could 00:13:15.760 --> 00:13:21.160 Dillon: be the inverse of like people aren't complaining because they're used to it um which is also bad 00:13:22.980 --> 00:13:24.140 Matt: yeah I mean I think 00:13:25.060 --> 00:13:26.840 Matt: well like the three of us have moved on from 00:13:27.120 --> 00:13:28.720 Matt: that company Bleep Fair 00:13:29.180 --> 00:13:30.340 Matt: as our 00:13:31.100 --> 00:13:32.680 Matt: friendly podcast neighbors call it 00:13:33.640 --> 00:13:35.040 Matt: and we probably maybe 00:13:36.260 --> 00:13:36.740 Matt: yeah 00:13:37.080 --> 00:13:38.220 Scott: you gotta get an actual beep 00:13:39.420 --> 00:13:40.960 Matt: yeah or we're gonna 00:13:41.160 --> 00:13:42.060 Matt: get a cease and desist from runtime rundown from 00:13:43.340 --> 00:13:44.460 Matt: using their term 00:13:45.260 --> 00:13:46.960 Matt: but I think yeah like we've moved on to other 00:13:47.060 --> 00:13:48.300 Matt: companies and maybe we've 00:13:49.420 --> 00:13:51.140 Matt: maybe seen similar 00:13:51.160 --> 00:13:56.260 Matt: of maybe, I don't know, the way I internalize it is a sort of whiplash of seeing, okay, 00:13:56.380 --> 00:13:58.740 Matt: how slow is the process here at HubSpot, for example? 00:13:58.940 --> 00:14:03.820 Matt: Or maybe Scott, you can talk about deploys at Airbnb compared to where we had it at the 00:14:03.960 --> 00:14:04.380 Matt: previous company. 00:14:05.700 --> 00:14:09.480 Matt: And just say, oh, actually, everyone else around us, or at least, okay, I'll speak for 00:14:09.540 --> 00:14:09.700 Matt: myself. 00:14:10.080 --> 00:14:12.540 Matt: Everyone else around me is like, that's just how it works, right? 00:14:12.880 --> 00:14:14.300 Matt: We've sort of become accustomed to it. 00:14:14.300 --> 00:14:16.200 Matt: And it's like, that's the speed it runs at. 00:14:16.880 --> 00:14:20.100 Matt: It would be great if it runs faster, but it's like, people are just sort of like, yeah, 00:14:20.280 --> 00:14:20.520 Matt: that's okay. 00:14:21.120 --> 00:14:24.300 Matt: Whereas like, I think that's maybe the wrong, wrong sort of default mindset. 00:14:24.660 --> 00:14:27.100 Matt: Like maybe the default mindset is how can you make that faster? 00:14:27.300 --> 00:14:28.140 Matt: How can you improve it? 00:14:29.500 --> 00:14:31.620 Matt: Cause I think it'll just have like a huge return on investment. 00:14:32.800 --> 00:14:38.180 Matt: So like, yeah, I think it's like risky to get, I don't know, to like sort of sit back 00:14:38.480 --> 00:14:39.920 Matt: and like say, yep, that's good enough. 00:14:40.160 --> 00:14:43.880 Matt: I think that's like a, you know, high risk sort of place to be in. 00:14:46.520 --> 00:14:48.120 Scott: Yeah, I agree with all these points. 00:14:48.380 --> 00:14:49.680 Scott: So this is where I'm saying like, 00:14:50.460 --> 00:14:54.300 Scott: we're talking about a little bit of a more nuanced problem with the deploys. 00:14:56.360 --> 00:15:00.420 Scott: Basically, like, I agree with organizational velocity. 00:15:00.670 --> 00:15:03.120 Scott: I think we need to, I've always said this, 00:15:03.270 --> 00:15:05.180 Scott: even when I was at Wayfair, I remember we built like, 00:15:05.530 --> 00:15:08.220 Scott: I built that toolbar, no one knew how to use like an internal toolbar. 00:15:08.390 --> 00:15:10.360 Scott: And my argument then was like, 00:15:10.920 --> 00:15:13.760 Scott: if developers don't have the tools and can't use them as fast as possible, 00:15:13.960 --> 00:15:15.220 Scott: how can they be successful, right? 00:15:15.520 --> 00:15:17.060 Scott: I think speed is incredibly important. 00:15:17.400 --> 00:15:20.140 Scott: I always see that same side. 00:15:22.420 --> 00:15:24.160 Scott: Speaking for like some things at Airbnb, 00:15:24.320 --> 00:15:27.200 Scott: I think like the site deploys are like 24 hours. 00:15:27.380 --> 00:15:29.680 Scott: And I don't think that that's the right answer either. 00:15:29.940 --> 00:15:32.640 Scott: Like I'm not saying like things need to be super slow. 00:15:32.900 --> 00:15:34.380 Scott: So I agree with both of you. 00:15:34.420 --> 00:15:36.540 Scott: We should optimize CI speed. 00:15:37.200 --> 00:15:39.180 Scott: If we see low hanging fruit, like Dillon said, 00:15:39.180 --> 00:15:41.520 Scott: we should be trying to make it better. 00:15:41.640 --> 00:15:45.260 Scott: We shouldn't like just accept that the process takes a long time. 00:15:45.640 --> 00:15:55.420 Scott: and then that's that's it but the problem here is like as folks know if we make our our deployment 00:15:55.660 --> 00:16:02.620 Scott: speed really short there's less process and rigorous testing that's going on like you you 00:16:02.700 --> 00:16:08.700 Scott: brought it up in our thread matt but like you might as you add end-to-end testing you slow down those 00:16:08.760 --> 00:16:15.520 Scott: speeds. And at some point adding that end to end testing is, is likely on a, on a site this big, 00:16:15.610 --> 00:16:21.560 Scott: it's more valuable to, for stability than it is to just be able to yeet it out in five minutes. 00:16:22.699 --> 00:16:29.480 Scott: Um, it's also like being able to do things really fast in an environment where folks don't work 00:16:29.720 --> 00:16:37.259 Scott: together or aren't working towards the same common, a same common goal isn't a replacement for getting 00:16:37.280 --> 00:16:43.440 Scott: things out. And again, five minutes is an extreme, but this is the extreme that people were talking 00:16:43.640 --> 00:16:48.960 Scott: about. Like 30 minutes, I wouldn't be complaining about, or maybe even 15 minutes, but it's the 00:16:49.160 --> 00:16:55.620 Scott: effort to get to five minutes. Now code is going out so fast that how are we supposed to know 00:16:56.440 --> 00:17:01.379 Scott: that if you have hundreds of engineers, how are we supposed to know what affects what when doing 00:17:01.440 --> 00:17:09.720 Scott: tests against things. So you don't have canary processes. You don't have a lot of end-to-end 00:17:09.880 --> 00:17:18.180 Scott: tests to cover things. Said monitors might not be that great. I don't know how they are now, 00:17:18.260 --> 00:17:22.839 Scott: but like, do you even have monitors to catch things? And I think another thing they brought 00:17:22.839 --> 00:17:28.079 Scott: up was, oh, it's low-level UI. Well, low-level UI could mean a click handler that takes you to the 00:17:28.060 --> 00:17:33.340 Scott: checkout page and now your target audience doesn't know how to get to the checkout page 00:17:33.720 --> 00:17:34.740 Scott: and you don't see that right away. 00:17:35.800 --> 00:17:41.960 Scott: These are things that could happen that are risks as opposed to having stability at a 00:17:42.200 --> 00:17:42.900 Scott: site that size. 00:17:43.300 --> 00:17:48.260 Scott: It's different, again, when it's a smaller organization. 00:17:48.760 --> 00:17:56.739 Scott: And if you could have end-to-end testing that covers everything and you could have all of 00:17:56.760 --> 00:18:01.640 Scott: these things in place to cover these, these quote, low level deploys, and it still happened in five 00:18:01.710 --> 00:18:08.180 Scott: minutes. I think that that's great, but I don't think that's realistic. And more realistic is like, 00:18:08.680 --> 00:18:12.820 Scott: what is the balance that we need to have to make sure we, we don't introduce a mistake. 00:18:13.520 --> 00:18:18.280 Scott: And another thing I think that like Wayfair used to have, doesn't have today as a staging 00:18:18.580 --> 00:18:23.360 Scott: environment. So a lot of this to me is a symptom that they're trying to mask that they're like, 00:18:23.400 --> 00:18:29.140 Scott: well, we can't trust dev. So we just put it out and prod as fast as possible. And if deploys are 00:18:29.280 --> 00:18:33.960 Scott: five minutes, we can have confidence faster. And to me, that's like, that's not correct at all. 00:18:34.320 --> 00:18:39.980 Scott: That's where I was saying we're masking, uh, we're masking the symptom to the real problem. 00:18:40.200 --> 00:18:46.020 Scott: And that's confidence in deploys. Like if you created a staging environment that actually 00:18:47.340 --> 00:18:52.080 Scott: reproduced production, or maybe it's an ephemeral environment that, that shows you what you need. 00:18:53.560 --> 00:18:57.860 Scott: and you could just say that that is 100% confidence there, 00:18:58.380 --> 00:19:01.160 Scott: then maybe it doesn't need to be in production in five minutes. 00:19:01.700 --> 00:19:03.540 Scott: And I guess that's like my whole argument here 00:19:03.960 --> 00:19:05.880 Scott: is just that there's some level of like, 00:19:06.060 --> 00:19:07.640 Scott: we have 100% confidence 00:19:07.940 --> 00:19:09.540 Scott: or maybe that's not realistic either. 00:19:10.420 --> 00:19:14.340 Scott: 98, 95% confidence that this change is going to impact 00:19:14.760 --> 00:19:17.100 Scott: as we see versus like, 00:19:17.200 --> 00:19:18.800 Scott: we got this thing out in five minutes. 00:19:19.020 --> 00:19:20.240 Scott: Now what's the next thing we're doing? 00:19:20.980 --> 00:19:24.520 Scott: in an organization that's not talking to each other and saying like, 00:19:24.700 --> 00:19:27.880 Scott: how does this actually impact what other teams are building? 00:19:28.760 --> 00:19:31.660 Scott: So there are, there are trade-offs here. 00:19:32.000 --> 00:19:32.980 Scott: And I think there's a balance. 00:19:33.220 --> 00:19:34.800 Scott: I think we need speed. 00:19:34.940 --> 00:19:37.620 Scott: I agree with the velocity, especially as an organization. 00:19:38.720 --> 00:19:43.960 Scott: But like at the cost of confidence and stability in what we're doing, 00:19:44.880 --> 00:19:46.360 Scott: I think is the wrong trade-off. 00:19:47.820 --> 00:19:49.460 Matt: I think I 00:19:49.460 --> 00:20:15.420 Matt: maybe fundamentally disagree that it's like, I think that's a false equivalence or false dichotomy. I'm not sure on the exact term. But like, I don't think it's a trade off between speed and stability. I think you can have both. Like, I would argue that you can have both like, right, like in the same way that you can speed up your CI pipelines, you can speed up your test runs, right? Like, maybe it means digging into, you know, contributing to Jest, for example, if you're using Jest for unit tests, like how do you speed up Jest or like swapping out Jest for something that's faster, right? 00:20:15.480 --> 00:20:22.440 Matt: Like, I think you can certainly still have your high confidence that you're deploying the right thing and also still deploy fast. 00:20:22.780 --> 00:20:29.500 Matt: So I think that's like a false tradeoff of like, you can't, like, I don't think they need to sacrifice stability in order to move faster. 00:20:29.560 --> 00:20:33.860 Matt: I think, or like, you know, moves like sacrifice speed in order to like have more confidence. 00:20:33.920 --> 00:20:36.100 Matt: I think that's, I think that's the wrong. 00:20:36.960 --> 00:20:42.520 Scott: But that's not what I'm saying because I'm telling you that some level, like a canary deployment takes time. 00:20:42.800 --> 00:20:45.820 Scott: It's basically what decides whether or not to promote it. 00:20:46.090 --> 00:20:50.080 Scott: So if you have Canary in, you have now slown down your deployment speed. 00:20:50.780 --> 00:20:53.260 Scott: Like what you're calling, it's in production and done. 00:20:53.620 --> 00:20:54.740 Scott: And that is my point. 00:20:55.120 --> 00:20:57.060 Scott: Not that like we can't optimize for speed. 00:20:57.110 --> 00:21:00.160 Scott: I keep saying that like optimizing for speed is great. 00:21:00.630 --> 00:21:06.720 Scott: But when you're optimizing for speed to mask symptoms of other problems, that is not correct. 00:21:08.220 --> 00:21:08.520 Scott: So we 00:21:08.520 --> 00:21:10.240 Scott: should have a rigorous 00:21:10.240 --> 00:21:11.100 Scott: staging environment. 00:21:11.440 --> 00:21:15.600 Scott: Maybe your staging environment requires you to look at it in staging before it moves to 00:21:15.600 --> 00:21:16.280 Scott: a promoted 00:21:16.280 --> 00:21:17.820 Scott: to Canary. 00:21:18.140 --> 00:21:21.260 Scott: And maybe Canary deploys last 20 to 30 minutes. 00:21:21.560 --> 00:21:25.140 Scott: You now have at shortest, a 35 minute full on deploy. 00:21:26.200 --> 00:21:27.360 Matt: Why does it take 20 to 30 minutes? 00:21:27.700 --> 00:21:30.900 Matt: You can validate changes in a minute, right? 00:21:31.540 --> 00:21:31.820 Matt: Like why? 00:21:31.860 --> 00:21:33.680 Scott: If you think you can validate a 00:21:33.680 --> 00:21:36.180 Scott: Canary deploy in one minute that it causes 00:21:36.180 --> 00:21:36.580 Scott: no 00:21:36.580 --> 00:21:37.000 Scott: problems, 00:21:37.700 --> 00:21:38.120 Scott: then sure. 00:21:38.560 --> 00:21:38.940 Scott: but I still 00:21:38.940 --> 00:21:39.640 Scott: don't think like 00:21:39.640 --> 00:21:40.360 Scott: five minutes, 00:21:41.040 --> 00:21:43.460 Scott: five minutes is a resume builder point. 00:21:43.960 --> 00:21:45.600 Scott: It's like, oh, we got things to five minutes. 00:21:46.020 --> 00:21:49.480 Scott: It takes time to keep things at five minutes as it grows, 00:21:50.140 --> 00:21:51.240 Scott: especially at a large scale. 00:21:51.720 --> 00:21:55.780 Scott: So walking into this like almost zero deployment speed, 00:21:56.760 --> 00:21:58.340 Scott: I think is the wrong basis. 00:21:59.020 --> 00:21:59.500 Scott: Sure. 00:21:59.760 --> 00:22:00.960 Scott: And again, I agree with you. 00:22:01.120 --> 00:22:04.140 Scott: If we want to have these as fast as possible, that's great. 00:22:04.500 --> 00:22:08.180 Scott: but we should be working on getting to that 80% 00:22:08.850 --> 00:22:11.840 Scott: and actually making sure we're making the right changes 00:22:12.260 --> 00:22:13.820 Scott: and they're stable in the right places. 00:22:14.600 --> 00:22:16.980 Scott: Because you're introducing more risk 00:22:17.130 --> 00:22:18.220 Scott: by shortening these things. 00:22:18.580 --> 00:22:20.440 Scott: Even if you say you have a one minute canary, 00:22:20.980 --> 00:22:24.700 Scott: like it doesn't guarantee that it's reasoned out 00:22:24.790 --> 00:22:26.300 Scott: any problems that might come by. 00:22:26.510 --> 00:22:28.420 Scott: Like having a longer canary speed, 00:22:29.400 --> 00:22:33.200 Scott: sorry, canary deployment helps with confidence. 00:22:33.760 --> 00:22:35.700 Scott: And maybe you don't need 20 to 30 minutes. 00:22:35.940 --> 00:22:37.080 Scott: Maybe you only need two minutes, 00:22:37.610 --> 00:22:39.800 Scott: but you need to actually build out these mechanisms 00:22:39.990 --> 00:22:42.500 Scott: as opposed to our deploys happen in five minutes. 00:22:43.780 --> 00:22:43.960 Dillon: Yeah. 00:22:44.880 --> 00:22:45.860 Dillon: One thing I'll add is like, 00:22:47.360 --> 00:22:49.000 Dillon: if you're deploying in five minutes, 00:22:50.440 --> 00:22:52.440 Dillon: you still want to like test your changes 00:22:52.530 --> 00:22:54.560 Dillon: in production manually always, right? 00:22:54.980 --> 00:22:57.300 Dillon: And maybe somebody else wants to do that too 00:22:57.400 --> 00:22:58.680 Dillon: because it's some change they care about. 00:22:59.400 --> 00:23:00.939 Dillon: But if everyone's deploying in five minutes 00:23:02.020 --> 00:23:09.520 Dillon: and over an hour there's like many many deploys and problems start to happen you don't know like 00:23:09.920 --> 00:23:12.860 Dillon: it's if it's your change that's causing the problem or some new change that's been 00:23:13.400 --> 00:23:17.860 Dillon: introduced in the last hour so it's you kind of run in the same problem of like a deploy train 00:23:19.040 --> 00:23:23.200 Dillon: shooting out 10 different things they didn't all happen all at once but they're starting to happen 00:23:23.300 --> 00:23:28.279 Dillon: five minute increments so i don't know i can see where scott's coming from in terms of like 00:23:28.240 --> 00:23:29.800 Dillon: introducing risk because you're shipping too quickly. 00:23:31.260 --> 00:23:32.760 Scott: Well, also we talked about this other thing. 00:23:32.780 --> 00:23:34.840 Scott: There are other mechanisms like feature flags 00:23:35.000 --> 00:23:36.580 Scott: that we should be shipping these things behind. 00:23:37.180 --> 00:23:38.680 Scott: We didn't talk about yet. 00:23:38.840 --> 00:23:40.180 Scott: I was expecting you to say that 00:23:40.280 --> 00:23:41.320 Scott: being able to deploy in five minutes, 00:23:41.460 --> 00:23:42.600 Scott: we can roll back in five minutes. 00:23:43.260 --> 00:23:45.400 Scott: That's great, but we also should be shipping 00:23:45.620 --> 00:23:47.140 Scott: new features behind feature flags. 00:23:47.880 --> 00:23:50.880 Scott: And we should be able to accurately A-B test. 00:23:51.060 --> 00:23:53.700 Scott: If we do have this many developers 00:23:54.020 --> 00:23:55.020 Scott: shipping things in five minutes, 00:23:55.440 --> 00:23:57.019 Scott: how do we start to know the impact 00:23:57.540 --> 00:23:59.420 Scott: of what features are impacting what 00:23:59.840 --> 00:24:01.220 Scott: in these types of environments. 00:24:01.660 --> 00:24:03.560 Scott: It becomes more and more difficult. 00:24:05.559 --> 00:24:07.640 Matt: So, yeah, I guess like two thoughts. 00:24:07.940 --> 00:24:08.960 Matt: One quick aside though, 00:24:09.260 --> 00:24:10.220 Matt: like for folks that don't know, 00:24:10.340 --> 00:24:13.040 Matt: Canary Deployments is essentially a process 00:24:13.120 --> 00:24:14.680 Matt: in which you release some changes 00:24:14.940 --> 00:24:16.440 Matt: to a small subset of production 00:24:17.240 --> 00:24:19.600 Matt: and you validate that things are looking good there, right? 00:24:19.760 --> 00:24:20.440 Matt: Like you say like, okay, 00:24:20.520 --> 00:24:23.140 Matt: I'm gonna take 5% of our production traffic, 00:24:23.380 --> 00:24:24.599 Matt: people that visit our experience 00:24:25.960 --> 00:24:27.580 Matt: and send them to this new version 00:24:28.280 --> 00:24:29.340 Matt: and see how things are going, 00:24:29.950 --> 00:24:31.600 Matt: like monitor metrics, speed, 00:24:31.880 --> 00:24:32.640 Matt: whatever else you care about. 00:24:33.120 --> 00:24:34.740 Matt: And then once you say, okay, that looks good, 00:24:34.880 --> 00:24:36.620 Matt: then finish the rollout to the other 95% 00:24:36.730 --> 00:24:38.240 Matt: or 5% just being an example. 00:24:38.240 --> 00:24:39.860 Matt: It could be any percentage slice 00:24:39.950 --> 00:24:41.040 Matt: that you want to do for Canary. 00:24:43.500 --> 00:24:45.580 Matt: I think we're maybe getting too locked 00:24:45.730 --> 00:24:49.580 Matt: into how we did deploys at Wayfair 00:24:50.280 --> 00:24:52.619 Matt: or how deploys work in the current state 00:24:52.680 --> 00:24:59.400 Matt: versus rethinking how can we do deploys in such a way that it unlocks these speed things, right? 00:24:59.540 --> 00:25:03.840 Matt: I'm thinking from the lens of like, we don't need to be locked into how we did there, how we did 00:25:03.940 --> 00:25:09.200 Matt: deploys. Or like, even like Dillon, you're kind of talking about like, this sort of default sort of 00:25:09.380 --> 00:25:14.120 Matt: staggering release process, where it's like, one change, wait five minutes, make sure it's validated, 00:25:14.600 --> 00:25:16.940 Matt: next change, wait five minutes. Like, I don't think that needs to be the case. 00:25:17.110 --> 00:25:17.979 Matt: I think we could do 00:25:18.100 --> 00:25:22.720 Matt: a percent or like sort of a rolling percentage of like, okay, we can release 10 changes at once. 00:25:23.080 --> 00:25:28.720 Matt: Each change gets a small percentage of production. We validate that percentage of production against 00:25:29.410 --> 00:25:34.040 Matt: the sort of the previous release. And if things are looking good, then that change is to deem 00:25:34.200 --> 00:25:39.700 Matt: safe and rolled out fully. Right. But that doesn't have to be sequential with other changes, right? 00:25:39.850 --> 00:25:44.119 Matt: Another change could sort of grab another slice of that pie, validate its changes against the 00:25:44.140 --> 00:25:50.620 Matt: larger slice and sort of happen in this, I don't know, sort of parallelized manner. 00:25:52.410 --> 00:25:58.240 Matt: So I think there's a lot of flexibility in terms of if you're optimizing for the speed, 00:25:58.350 --> 00:26:03.440 Matt: you start to think about, well, the ways that we deployed websites 10 years ago or five years ago 00:26:03.560 --> 00:26:09.440 Matt: or two years ago, maybe that doesn't by default enable this sort of speed that we want. 00:26:09.820 --> 00:26:11.480 Matt: what are alternatives? How can we deploy? 00:26:12.260 --> 00:26:13.920 Matt: How do we unlock those features without 00:26:14.560 --> 00:26:14.960 Matt: sacrificing 00:26:16.340 --> 00:26:16.820 Matt: other things? 00:26:17.900 --> 00:26:20.000 Matt: I think that's the lens I'm coming 00:26:20.160 --> 00:26:20.220 Matt: from. 00:26:22.260 --> 00:26:23.780 Matt: With the existing systems, how do we 00:26:24.050 --> 00:26:25.940 Matt: unlock it? More like we need to think 00:26:25.940 --> 00:26:27.840 Matt: from the ground up of first principles of how 00:26:27.850 --> 00:26:29.760 Matt: do you release changes such that 00:26:30.300 --> 00:26:31.760 Matt: you optimize for this speed? 00:26:33.580 --> 00:26:33.660 Matt: Yeah, 00:26:34.420 --> 00:26:35.820 Matt: and Scott, you talked a little bit about rollbacks. 00:26:36.060 --> 00:26:37.780 Matt: I think rollbacks should be even faster. 00:26:38.370 --> 00:26:39.300 Matt: You should have an instant rollback. 00:26:39.480 --> 00:26:39.980 Scott: It should be instant. 00:26:40.640 --> 00:26:40.740 Scott: Yep. 00:26:41.040 --> 00:26:41.160 Matt: Exactly. 00:26:41.580 --> 00:26:42.980 Matt: Immediately switch back to your previous version, 00:26:43.760 --> 00:26:45.220 Matt: assuming you find some critical flaw. 00:26:47.140 --> 00:26:49.280 Matt: Immediate meaning like the next exact, 00:26:49.680 --> 00:26:52.000 Matt: like the next request goes to the previous version 00:26:52.900 --> 00:26:55.300 Matt: instead of, you know, like, you know, 00:26:55.880 --> 00:26:58.580 Matt: a cool down timer or whatever of like, you know, 00:26:58.680 --> 00:27:00.640 Matt: after like cache is clear after a certain number of time 00:27:00.640 --> 00:27:01.540 Matt: and then they get to the next, 00:27:02.020 --> 00:27:02.940 Matt: the previous version or whatever. 00:27:04.160 --> 00:27:05.380 Dillon: Feature flags, I think are like, 00:27:06.500 --> 00:27:08.440 Dillon: they like hide problems. 00:27:10.180 --> 00:27:13.460 Dillon: and let them awaken at weird times, 00:27:14.200 --> 00:27:15.280 Dillon: is how I would think of it. 00:27:17.140 --> 00:27:19.640 Dillon: Because you can deploy your change behind a feature flag 00:27:19.720 --> 00:27:21.600 Dillon: and it's like, oh, it went well. 00:27:22.100 --> 00:27:24.960 Dillon: So I'm curious, if you're doing Canary or end-to-end, 00:27:25.180 --> 00:27:28.140 Dillon: are you enabling your feature flag when those things run 00:27:28.740 --> 00:27:30.560 Dillon: and it's isolated to that environment? 00:27:32.420 --> 00:27:36.220 Dillon: Because otherwise it's going in and other people are pulling it in, 00:27:36.220 --> 00:27:37.980 Dillon: but the feature flag's off when they run their changes, 00:27:38.180 --> 00:27:42.860 Dillon: So they don't really like test the integration between your change and their change. 00:27:43.120 --> 00:27:45.360 Dillon: So it's like, I don't know, I feel like it's complicated. 00:27:46.840 --> 00:27:53.360 Matt: Yeah, I think my take is that you should be sampling from the sort of the same setup. 00:27:53.730 --> 00:27:58.340 Matt: So like, let's say you're doing these like five parallel deploys, each grabs, you know, 00:27:58.340 --> 00:28:01.960 Matt: I don't know, 2% of your production traffic to validate their changes. 00:28:02.560 --> 00:28:06.499 Matt: Like you should be taking a sample of the percentage of that feature flag rollout in 00:28:06.520 --> 00:28:08.760 Matt: in those 2% slices for each of the five or whatever. 00:28:09.250 --> 00:28:12.340 Matt: So it should be likely that in those little canaries 00:28:12.470 --> 00:28:13.320 Matt: that you're running in parallel, 00:28:13.660 --> 00:28:15.460 Matt: they should have the likelihood of also running 00:28:15.610 --> 00:28:16.880 Matt: into that feature flag test 00:28:17.660 --> 00:28:20.440 Matt: and validating that the changes sort of work the same way. 00:28:22.000 --> 00:28:23.980 Matt: I don't know if I'm getting my math right or whatever, 00:28:24.050 --> 00:28:26.320 Matt: but you should have a way to validate 00:28:26.560 --> 00:28:29.160 Matt: both the A and B path of doing a feature flag 00:28:29.160 --> 00:28:30.040 Matt: test. 00:28:30.380 --> 00:28:32.100 Dillon: It feels like a perfect world situation 00:28:32.600 --> 00:28:35.640 Dillon: because what if there's like 500 feature flags in my app? 00:28:35.740 --> 00:28:38.600 Dillon: Am I just like flipping in the different combinations 00:28:39.140 --> 00:28:40.420 Dillon: and testing every one of them? 00:28:40.480 --> 00:28:42.080 Dillon: It just feels like an impossible problem to solve. 00:28:44.060 --> 00:28:46.700 Matt: I think it's like, again, 00:28:46.880 --> 00:28:48.760 Matt: it sort of gets back to like the organizational velocity 00:28:48.940 --> 00:28:52.600 Matt: of like you should be doing everything you can 00:28:52.740 --> 00:28:55.480 Matt: to validate your assumptions as fast as possible. 00:28:55.680 --> 00:28:56.960 Matt: And like sort of with deployers 00:28:57.000 --> 00:28:57.900 Matt: and with feature flags or whatever, 00:28:58.020 --> 00:28:59.520 Matt: like you should be building systems 00:28:59.820 --> 00:29:01.440 Matt: that allow you to validate those things. 00:29:01.520 --> 00:29:04.839 Matt: So like, let's say you have 500 or 1,000 feature flags 00:29:04.860 --> 00:29:06.060 Matt: at any given point in time. 00:29:06.720 --> 00:29:08.380 Matt: You should have a system that allows you to 00:29:09.900 --> 00:29:11.080 Matt: extrapolate those permutations 00:29:12.400 --> 00:29:13.500 Matt: and run those, 00:29:14.540 --> 00:29:16.400 Matt: even if they're not actual real traffic, 00:29:16.600 --> 00:29:17.580 Matt: but simulated real traffic 00:29:17.730 --> 00:29:19.720 Matt: or simulated traffic behind the scenes, 00:29:19.840 --> 00:29:21.140 Matt: you should be able to run those permutations 00:29:22.620 --> 00:29:25.040 Matt: in parallel and across multiple different systems 00:29:25.210 --> 00:29:25.540 Matt: or whatever. 00:29:26.430 --> 00:29:27.040 Matt: That should be a thing. 00:29:28.710 --> 00:29:29.300 Matt: Again, it's like, 00:29:30.110 --> 00:29:32.180 Matt: if you start with this mindset of moving as fast as possible 00:29:32.210 --> 00:29:33.180 Matt: in the right direction, it's like, 00:29:33.420 --> 00:29:35.480 Matt: what systems do you need to build to enable that? 00:29:35.560 --> 00:29:37.740 Matt: And that should be like a thing that you think about. 00:29:38.400 --> 00:29:39.700 Dillon: I feel like if I built that system 00:29:40.100 --> 00:29:43.440 Dillon: and ran it on whatever CI provider my company had, 00:29:44.120 --> 00:29:45.520 Dillon: it would slowly go bankrupt. 00:29:47.400 --> 00:29:48.960 Dillon: Like it sounds so expensive to 00:29:48.960 --> 00:29:50.000 Dillon: run something like that. 00:29:52.580 --> 00:29:54.480 Scott: Like, I think you are like talking 00:29:54.560 --> 00:29:55.680 Scott: in a perfect world scenario. 00:29:55.900 --> 00:29:58.360 Scott: And I don't think that it's necessarily wrong, 00:29:59.060 --> 00:30:01.999 Scott: but like in like a realistic team 00:30:02.020 --> 00:30:09.200 Scott: where there's like five engineers and we have like this many OKRs, like it's kind of unreasonable 00:30:09.440 --> 00:30:14.420 Scott: to say like, all right, we're just going to focus in on the pipelines that are 30 minutes now and use 00:30:14.580 --> 00:30:19.240 Scott: n number of time to get it to five versus like, we'll try to get it faster. And here's some low 00:30:19.340 --> 00:30:23.960 Scott: hanging fruit we can do. And that could be something we do at all times. So while I think 00:30:24.460 --> 00:30:30.759 Scott: speed is important, like speed at all costs without like thinking through how it affects 00:30:30.780 --> 00:30:38.060 Scott: things downstream and to mask other problems, that's where I think the problem is. I think 00:30:38.280 --> 00:30:43.040 Scott: there's just generally some other mechanisms that should be introduced before we're like, 00:30:43.320 --> 00:30:49.320 Scott: our deployment speeds are five minutes. Again, we are pigeonholed in a very specific example, 00:30:49.370 --> 00:30:54.440 Scott: but I kind of mean it for that. I'm talking about these larger companies. I think they all should 00:30:54.540 --> 00:30:59.559 Scott: optimize their pipelines. I think they all should be trying to make it quicker. But there's also the 00:30:59.580 --> 00:31:05.980 Scott: thought too and i know that you might not subscribe to this but some of the most successful companies 00:31:06.280 --> 00:31:11.720 Scott: do release schedules of like twice a year like that's the the apple staple and they they're 00:31:11.860 --> 00:31:17.160 Scott: deploying continuously like bug fixes like you said that's why like when you were like oh initially 00:31:17.320 --> 00:31:21.400 Scott: you were like we're gonna deploy bug fixes and we want to unblock that to being deployed as fast as 00:31:21.580 --> 00:31:27.380 Scott: possible love that logic i agree with that if five minute deploys is like we can get bug fixes out in 00:31:27.320 --> 00:31:31.500 Scott: five minutes, but it's like, how do we make sure people aren't yeeting out brand new features 00:31:32.240 --> 00:31:37.820 Scott: that when, when our audience is like a middle-aged women, do they want to go from one page to 00:31:37.820 --> 00:31:40.120 Scott: the next and just the website just keeps changing? 00:31:40.680 --> 00:31:45.120 Scott: Like maybe that's an extreme example, but if hundreds of engineers at this scale can 00:31:45.220 --> 00:31:49.500 Scott: do that, um, it starts to introduce like things we didn't think about. 00:31:49.620 --> 00:31:51.100 Scott: Like, I agree with you. 00:31:51.320 --> 00:31:55.780 Scott: We don't want to be the team blocking other teams from being successful when we're in 00:31:55.900 --> 00:31:57.000 Scott: charge of the play pipelines. 00:31:57.820 --> 00:32:03.200 Scott: But we also need to be confident that those changes have value. 00:32:03.630 --> 00:32:10.280 Scott: Teams need to be talking about the value their changes bring and how it impacts downstream 00:32:10.930 --> 00:32:14.520 Scott: before we just unlock teams from getting things done. 00:32:14.520 --> 00:32:20.440 Scott: So I just think that there's more to the organizational velocity than just deployment 00:32:20.800 --> 00:32:21.620 Scott: speed at all costs. 00:32:22.040 --> 00:32:27.480 Scott: Like teams need to be communicating better to help what they're building drive that product. 00:32:27.990 --> 00:32:31.980 Scott: But I think like in a perfect world, if everything's in five minutes, like, sure. 00:32:32.240 --> 00:32:33.060 Scott: Like that sounds great. 00:32:33.660 --> 00:32:35.080 Scott: We got canaries really quick. 00:32:35.410 --> 00:32:37.940 Scott: We have a staging environment that we trust. 00:32:38.190 --> 00:32:39.760 Scott: Like, I agree with you too. 00:32:39.900 --> 00:32:45.520 Scott: The status quo of like, well, everything's just this slow is on an unacceptable way of 00:32:45.680 --> 00:32:45.820 Scott: thinking. 00:32:45.990 --> 00:32:49.940 Scott: Like you have to push and drive these things to be better. 00:32:50.100 --> 00:32:53.780 Scott: But like, you know, sometimes maybe things shouldn't be pipelines. 00:32:54.280 --> 00:33:00.180 Scott: Maybe things could be scripts that engineers run locally so that that reduces the size of things. 00:33:00.240 --> 00:33:06.320 Scott: But it also just requires a change in mindset and flow. 00:33:06.960 --> 00:33:18.480 Scott: So I'm only arguing that at some point, like making sure we almost stupid proof deploys has a little value too. 00:33:20.160 --> 00:33:26.060 Matt: I think there's two thoughts. So getting more to the point of, you were talking about, 00:33:26.900 --> 00:33:30.540 Matt: customer reloads a page or navigates a different page and sees a dramatically different experience. 00:33:31.620 --> 00:33:35.900 Matt: Part of me, this gets back to why I brought in the topic outside of just deploys, 00:33:37.040 --> 00:33:43.099 Matt: why I brought up the concept of organizational velocity. That should permeate through everything 00:33:43.120 --> 00:33:48.860 Matt: the company does, right? Meaning that the company should sort of like product decisions should be 00:33:49.000 --> 00:33:53.020 Matt: moving at that same pace and should be, you know, and the reason I use velocity instead of speed is 00:33:53.020 --> 00:33:57.740 Matt: that velocity is like speed plus direction, right? Like you should have an opinion direction and 00:33:57.840 --> 00:34:02.660 Matt: like your organization should be aligned on that, right? So like you shouldn't be, you shouldn't 00:34:02.860 --> 00:34:06.140 Matt: result, you know, if you start with that as a first principle, you shouldn't result in 00:34:06.920 --> 00:34:11.379 Matt: n different teams, each going n different directions at, you know, different speeds, 00:34:11.560 --> 00:34:16.860 Matt: right like you should have sort of a single team single direction sort of perspective of like you 00:34:16.860 --> 00:34:21.940 Matt: know okay we're gonna we're gonna test you know this assumption and like that's going to be applied 00:34:22.260 --> 00:34:27.879 Matt: universally sort of across the the org um the the other thing i was thinking about is like you're 00:34:27.980 --> 00:34:33.280 Matt: talking a little bit sort of on like you know and again maybe i'm like mischaracterizing what you're 00:34:33.340 --> 00:34:36.760 Matt: saying but you i think you were kind of saying like okay you shouldn't say like all right we're 00:34:36.760 --> 00:34:39.819 Matt: gonna take the sprint and just like figure out how we can make the pipeline as fast as possible 00:34:39.840 --> 00:34:49.080 Matt: Instead, it's fine to litter small little performance tickets or whatever, tickets that help improve the speed every so often throughout different sprints. 00:34:49.520 --> 00:34:50.919 Matt: I think that's what I'm arguing for. 00:34:52.300 --> 00:35:00.680 Matt: You shouldn't say, okay, we're going to spend a month trying to speed things up and then go back to the other 11 months of the year working on product features. 00:35:00.900 --> 00:35:04.520 Matt: And then find out a year or two later, oh, we need to spend another month working on performance. 00:35:04.560 --> 00:35:09.220 Matt: I think it should be a continuous activity that every team does, every engineer does. 00:35:09.660 --> 00:35:15.360 Matt: right it's like what can i pick up today or the sprint that allows us to move faster in the right 00:35:15.500 --> 00:35:16.920 Matt: direction 00:35:16.920 --> 00:35:23.740 Scott: i agree with all of that i just and i i i don't think like we should do one sprint 00:35:24.220 --> 00:35:29.780 Scott: one month or whatever like i i don't think we should like die on this hill though like there 00:35:29.880 --> 00:35:35.460 Scott: are other organizational problems that get us to the organizational velocity you're trying to achieve 00:35:35.760 --> 00:35:47.220 Scott: I think we have this narrow-minded focus on it's all about the deploys are the blocker and the problem, and that will just somehow magically relieve the rest of the organization. 00:35:47.620 --> 00:35:50.160 Scott: I think in your scenario, again, it's kind of perfect world. 00:35:50.450 --> 00:35:51.440 Scott: I don't disagree with it. 00:35:51.440 --> 00:35:52.620 Scott: I think it sounds great. 00:35:53.080 --> 00:36:08.020 Scott: But if we focus on this one aspect, but don't fix the other aspects, it's going to either cause more problems and we're just kind of masking and band-aiding things that exist in the more specific scenario I built. 00:36:09.360 --> 00:36:11.340 Scott: Some of these other things are just really hard to achieve. 00:36:11.940 --> 00:36:18.200 Scott: It's hard to make remote teams that are everywhere align on the same things. 00:36:18.340 --> 00:36:23.500 Scott: It's hard to make sure that we're moving towards the same common goal based on how things are broken down. 00:36:24.120 --> 00:36:31.100 Scott: If, again, like it's a very specific example, but people are deploying things there just to make sure that it works right. 00:36:31.300 --> 00:36:33.960 Scott: And that's fundamentally broken. 00:36:34.300 --> 00:36:42.500 Scott: If we stopped and built a staging environment, I think there's almost more value in that to just do that and make sure that works or whatever. 00:36:42.560 --> 00:36:46.180 Scott: some sort of solution that allows you to have confidence before you deploy, 00:36:46.840 --> 00:36:49.220 Scott: people might not rush so much for that five minutes. 00:36:49.780 --> 00:36:54.160 Scott: That's not me arguing that we shouldn't do our best to make them as fast as possible. 00:36:54.800 --> 00:36:57.600 Scott: I just think in this example, we're just saying, 00:36:57.780 --> 00:37:03.500 Scott: oh, wouldn't it be nice if deploys are so fast that all of these other problems, 00:37:05.160 --> 00:37:06.440 Scott: we don't have to worry about them. 00:37:07.160 --> 00:37:10.180 Scott: But with great power comes great responsibility, 00:37:10.460 --> 00:37:11.660 Scott: like Dillon kind of brought up. 00:37:11.710 --> 00:37:13.860 Scott: And it's like, now we're deploying so fast 00:37:13.980 --> 00:37:16.180 Scott: that we're just like calling things done 00:37:16.350 --> 00:37:17.040 Scott: because they're deployed 00:37:17.280 --> 00:37:19.560 Scott: and we're not picking up on some of the slack 00:37:19.610 --> 00:37:20.520 Scott: of these other problems. 00:37:21.540 --> 00:37:22.280 Scott: So like, 00:37:22.340 --> 00:37:23.400 Scott: there's also the thought, 00:37:24.160 --> 00:37:25.580 Scott: there's also the thought of like, 00:37:26.260 --> 00:37:28.540 Scott: not every single update needs to go out on its own. 00:37:28.720 --> 00:37:29.840 Scott: Some things can be batched. 00:37:29.850 --> 00:37:32.580 Scott: And I'm not like being this great advocate 00:37:32.670 --> 00:37:33.840 Scott: for deploy trains and whatnot. 00:37:34.270 --> 00:37:37.600 Scott: And there's also other mechanisms. 00:37:37.820 --> 00:37:45.840 Scott: Like, again, I'm getting very specific in the example of we deploy on main there. 00:37:46.500 --> 00:37:53.460 Scott: And basically when one person deploys, one of the problems they're trying to mask is, oh, we now have to merge in master or main. 00:37:53.880 --> 00:37:57.560 Scott: And it's like, yeah, that's like the way that's set up. 00:37:57.680 --> 00:37:59.120 Scott: But you don't have to do that. 00:37:59.220 --> 00:38:00.820 Scott: You could actually make the change. 00:38:01.760 --> 00:38:02.820 Scott: There's ways to make the change. 00:38:02.880 --> 00:38:06.840 Scott: It's like, oh, you can continuously deploy until there's a general merge conflict. 00:38:07.540 --> 00:38:09.600 Scott: So there's other ways around these problems. 00:38:10.940 --> 00:38:17.700 Scott: And I think that gung-ho, it's too much like if it was in five minutes, then people wouldn't 00:38:17.840 --> 00:38:18.700 Scott: see these other symptoms. 00:38:20.160 --> 00:38:25.920 Scott: But I agree that, again, I'm probably repeating myself, but I agree that we should be actively 00:38:26.160 --> 00:38:27.340 Scott: trying to make things faster. 00:38:28.540 --> 00:38:32.120 Scott: But we also should have reason, I guess, behind why we're making changes. 00:38:33.040 --> 00:38:39.120 Scott: And there is discussion to be about, should we be making major changes immediately all 00:38:39.260 --> 00:38:40.060 Scott: the time everywhere? 00:38:40.920 --> 00:38:44.880 Scott: I think maybe the answer is yes, but I think there needs to be conversations. 00:38:45.320 --> 00:38:50.360 Scott: And like you said, in this perfect structure of organizational velocity, that should already 00:38:50.360 --> 00:38:51.100 Scott: be teased out. 00:38:51.390 --> 00:38:52.980 Scott: And then the deploy shouldn't block that. 00:38:53.400 --> 00:38:54.180 Scott: Totally agree with that. 00:38:54.490 --> 00:38:56.840 Scott: But I just think that I've seen too much. 00:38:57.440 --> 00:39:08.780 Scott: Like we're just band-aiding what this thing does instead of this balance between, all right, we now have confidence this thing's going to go out great and all these other problems don't exist anymore. 00:39:08.950 --> 00:39:10.940 Scott: So I'm less worried that it goes out in five minutes. 00:39:12.600 --> 00:39:12.800 Matt: All right. 00:39:13.540 --> 00:39:15.560 Matt: Yeah, I think, I mean, I think I agree with you. 00:39:15.720 --> 00:39:17.840 Matt: I think it's just more so around-- 00:39:18.340 --> 00:39:20.060 Matt: is it a-- 00:39:21.120 --> 00:39:24.520 Matt: I think maybe the thing that we disagree on is just 00:39:24.820 --> 00:39:28.700 Matt: how do you optimize or how do you pick your focus point? 00:39:28.720 --> 00:39:30.980 Matt: And I think it's maybe you're looking at it 00:39:30.980 --> 00:39:33.780 Matt: from the lens of get to a good enough point 00:39:34.100 --> 00:39:35.600 Matt: and then focus to something else. 00:39:35.760 --> 00:39:38.960 Matt: Whereas my perspective, maybe overlay optimistic perspective, 00:39:39.160 --> 00:39:42.380 Matt: is always be working on that. 00:39:42.620 --> 00:39:44.420 Matt: It's sort of focus on positive some games 00:39:44.440 --> 00:39:48.540 Matt: of everything should be in scope to tackle. 00:39:50.000 --> 00:39:51.940 Matt: I think we're kind of getting to the closing point here. 00:39:52.920 --> 00:39:55.100 Scott: Would you say maybe you're Pollyannish about it? 00:39:56.140 --> 00:39:56.380 Matt: Sure. 00:39:57.860 --> 00:39:59.440 Matt: I think we're kind of getting to our closing thoughts here. 00:39:59.960 --> 00:40:01.120 Matt: Dillon, I don't know if you had anything. 00:40:02.180 --> 00:40:03.860 Dillon: Yeah, my closing point is that 00:40:04.150 --> 00:40:05.760 Dillon: if we're telling people to spend time 00:40:06.580 --> 00:40:07.820 Dillon: thinking about optimizing the pipelines, 00:40:08.030 --> 00:40:09.640 Dillon: it almost feels a little counterintuitive 00:40:09.710 --> 00:40:10.960 Dillon: to tell them to go fast. 00:40:11.800 --> 00:40:14.220 Dillon: If they're spending time trying to fix pipelines, 00:40:14.340 --> 00:40:17.580 Dillon: but they're supposed to be spending time making the product high quality, 00:40:18.380 --> 00:40:21.060 Dillon: like where it's tough to tell them to split their time. 00:40:22.960 --> 00:40:25.620 Dillon: To me, it feels like a weird paradox of like, 00:40:27.480 --> 00:40:30.500 Dillon: maybe the engineering leadership and engineering leaders 00:40:30.800 --> 00:40:33.140 Dillon: and some other team should be helping do this 00:40:33.240 --> 00:40:35.540 Dillon: so that those teams can focus on the product. 00:40:36.040 --> 00:40:37.000 Dillon: But that's just my take. 00:40:38.200 --> 00:40:39.140 Scott: Operational excellence. 00:40:41.420 --> 00:40:41.840 Dillon: Yeah. 00:40:41.920 --> 00:40:45.380 Dillon: If I'm spending 50% of my time, I'm splitting my focus. 00:40:45.570 --> 00:40:47.320 Dillon: I'm not focusing enough on the product anymore. 00:40:48.760 --> 00:40:49.260 Dillon: I don't know. 00:40:49.330 --> 00:40:51.300 Dillon: I feel like it's kind of a negative for the company in a way. 00:40:52.280 --> 00:40:52.380 Matt: Yeah. 00:40:52.480 --> 00:40:56.220 Matt: Scott, any closing thoughts related to that or unrelated to that? 00:40:58.200 --> 00:40:58.380 Scott: Yeah. 00:40:58.490 --> 00:41:04.020 Scott: I guess just when you're trying to do something to an extreme, try to think about what are 00:41:04.360 --> 00:41:06.740 Scott: some smaller wins that can help alleviate this problem. 00:41:07.060 --> 00:41:09.020 Scott: I guess that's what I'm getting at. 00:41:09.060 --> 00:41:15.700 Scott: And maybe I'm taking the five minutes too literal, but that seemed to be like a serious benchmark in this example. 00:41:16.400 --> 00:41:24.660 Scott: And I know that there's like hundreds of other things that could add value that could be done first before that. 00:41:25.720 --> 00:41:35.060 Scott: And there are probably other ways to solve that problem or alleviate pain that are simpler. 00:41:35.120 --> 00:41:54.760 Scott: So just like try to try to, I guess, like, ideally, what I'm trying to say is write down what the problem is instead of immediately coming up with a solution and try to say, like, what are some ways we can try to alleviate the problem we're having and look at it in more broad terms? 00:41:56.140 --> 00:42:18.500 Matt: Yeah, I think, yeah, here's my closing take is that the reason I push for sort of like these maybe hyper optimistic, like way too unrealistic or whatever sort of possibilities is like, if teams take that and then say just become like 1% better than what they're doing, then that's already like a positive outcome. 00:42:19.280 --> 00:42:23.260 Matt: You know, it's like sort of the, you know, the common phrase of like, you know, shoot for the stars and maybe land on the moon or something, right? 00:42:23.340 --> 00:42:33.860 Matt: Like, it's like sort of like, if you take this as like a vision of like something that you should do, go for, then you're going to probably come out with something that's better than what you would have done without it. 00:42:34.420 --> 00:42:38.380 Matt: So yes, a lot of the things I'm pushing for is like maybe unrealistic for a lot of orgs. 00:42:38.670 --> 00:42:50.020 Matt: But I think it's, I think it's, it should be seen as like a motivating factor or like a sort of a thing of like, you know, rethink those things that were holding us back to the stuff that made us move at the pace we did. 00:42:52.620 --> 00:42:55.760 Matt: Yeah, I think that's maybe a good place for us to end. 00:42:56.580 --> 00:42:58.500 Matt: We don't have any other special segments this week. 00:42:59.000 --> 00:43:01.260 Matt: Maybe next month or three months from now, 00:43:01.300 --> 00:43:02.760 Matt: we'll have those queued up. 00:43:03.120 --> 00:43:04.420 Matt: Hopefully we get back into the recording lifecycle. 00:43:04.860 --> 00:43:05.220 Matt: We'll see. 00:43:06.520 --> 00:43:08.860 Dillon: Quick plug for something at my work 00:43:09.340 --> 00:43:10.500 Dillon: that they've been piloting 00:43:11.300 --> 00:43:12.500 Dillon: that's sort of related to this stuff. 00:43:12.720 --> 00:43:14.220 Dillon: They're using a new app called DX. 00:43:15.040 --> 00:43:16.360 Dillon: It's like getdx.com. 00:43:17.100 --> 00:43:19.780 Dillon: I can't speak too much to how useful it is yet, 00:43:20.440 --> 00:43:34.440 Dillon: But it shows the breakdown for every team within the software org and other parts of the company, like what their velocity is in different areas when it comes to deployment, PRs and stuff like that. 00:43:34.480 --> 00:43:41.240 Dillon: And it allows you to see what is the sentiment around those things as well, because it'll send surveys through Slack. 00:43:41.980 --> 00:43:47.100 Dillon: So it's been kind of interesting to see how things change over time. 00:43:49.500 --> 00:43:50.200 Matt: Is this like that? 00:43:50.500 --> 00:43:52.680 Matt: Do you remember that thing we piloted at Wayfair 00:43:52.680 --> 00:43:53.520 Matt: and then they rebranded it? 00:43:53.520 --> 00:43:54.600 Matt: It was like GitFlow or something? 00:43:55.640 --> 00:43:57.360 Matt: It was like developer metrics service. 00:43:57.560 --> 00:43:59.140 Dillon: Yeah, that one felt more based on 00:43:59.260 --> 00:44:01.020 Dillon: how do I scrutinize each developer? 00:44:01.180 --> 00:44:01.860 Dillon: This feels more 00:44:01.860 --> 00:44:03.160 Dillon: like how 00:44:03.160 --> 00:44:05.180 Dillon: can we find bottlenecks in the org? 00:44:06.800 --> 00:44:07.000 Matt: Okay. 00:44:08.280 --> 00:44:12.180 Matt: I feel like that's a very tough line to balance across. 00:44:13.840 --> 00:44:15.580 Matt: I wouldn't be surprised if they have dashboards 00:44:15.780 --> 00:44:18.040 Matt: that are like, yeah, your engineers are doing less work 00:44:18.040 --> 00:44:18.880 Matt: than they should be or whatever. 00:44:18.960 --> 00:44:19.260 Matt: for 00:44:19.260 --> 00:44:19.600 Dillon: managers or something. 00:44:19.600 --> 00:44:20.840 Dillon: Yeah, it could be used for that, 00:44:21.200 --> 00:44:22.940 Dillon: but I'm getting the sense 00:44:22.990 --> 00:44:25.120 Dillon: that it's being used for other things, 00:44:25.240 --> 00:44:26.080 Dillon: like better things, 00:44:26.360 --> 00:44:26.960 Dillon: but we'll see. 00:44:28.040 --> 00:44:29.280 Dillon: If people start getting let go, 00:44:29.420 --> 00:44:29.740 Dillon: I'll let you know. 00:44:33.880 --> 00:44:34.160 Matt: All right. 00:44:34.310 --> 00:44:35.120 Matt: If you enjoy this podcast, 00:44:35.880 --> 00:44:37.800 Matt: share it with someone that you know 00:44:38.260 --> 00:44:39.300 Matt: that might also enjoy it. 00:44:40.000 --> 00:44:40.560 Matt: Leave us a like, 00:44:40.770 --> 00:44:41.760 Matt: leave us a subscribe, 00:44:42.140 --> 00:44:43.140 Matt: leave us some reviews. 00:44:44.100 --> 00:44:45.080 Matt: I don't know how many stars we have. 00:44:45.080 --> 00:44:46.000 Matt: We probably have zero out of five 00:44:47.040 --> 00:44:48.100 Matt: just because no one leaves reviews, 00:44:48.340 --> 00:44:50.340 Matt: but yeah, we appreciate it. 00:44:50.700 --> 00:44:51.600 Matt: It'll keep us making 00:44:51.600 --> 00:44:52.000 Matt: content. 00:44:54.320 --> 00:44:54.400 Matt: Yeah. 00:44:55.400 --> 00:44:55.640 Matt: All right. 00:44:55.840 --> 00:44:58.840 Matt: We'll catch you next month, year, decade? 00:44:58.880 --> 00:45:00.160 Matt: Yeah, you never know. 00:45:00.360 --> 00:45:00.800 Matt: Podcast. 00:45:03.740 --> 00:45:03.940 Scott: Peace. 00:45:04.680 --> 00:45:05.020 Dillon: Later. 00:45:05.940 --> 00:45:06.300 Matt: See ya.