CI/CD with Robert Erez

**Gergely Orosz** (0:00)
CICD remains one of the hardest things to get right in software engineering.

**Robert Erez** (0:04)
But why?

**Gergely Orosz** (0:05)
Rob Erez is a CICD expert having worked in this field for more than a decade. In the early 2010s, we were teammates on the Skype for Web team, and then Rob joined Octopus Deploy as one of the first engineers 10 years ago.
In today's episode we cover Progressive delivery and practice, Canary deployments, blue-green, and why feature toggles are often still better, what is GitOps and why it's not about Git, and where that everything and Git mindset breaks down. Why you should prioritize rollbacks less and focus on roll-forwards. And many more. If you want harder-earned lessons about CI-CD, Progressive delivery, and what's coming as AI changes how much code we ship to production, then this episode is for you. This episode is presented by Intesis. Verify your system's correctness without human review or traditional integration tests and avoid bugs or outages. Today's episode will be about CI-CD. CI-CD at scale is one of the hardest infrastructure problems to get right and the teams who nail it know that the details very much matter.
This is where I need to mention our season sponsor WorkOS.
WorkOS brings the same rigor as many of us use with CI-CD at scale to Enterprise Auth. SSO skim, RBAC, production-ready, battle-tested and built to handle real load and real compliance requirements. To add Enterprise Auth without the infrastructure project, visit workos.com. Rob, it's awesome to have you here on the podcast.

**Robert Erez** (1:24)
Hello, go-go, it's good to be here. Yeah, I'm loving Amsterdam.

**Gergely Orosz** (1:27)
Yeah, it's been like what, 11, 12 years since we walked together, worked together.

**Robert Erez** (1:31)
Yeah, I think 2014, 2015, I think I left UK. Yeah, it's a while.

**Gergely Orosz** (1:38)
And Skype, when there was still Skype. Our team somehow inherited the outlog.com plugin, which had like 400 million users per month or something like that.

**Robert Erez** (1:47)
It was crazy. So this was an interesting job. Deployments were very much a case of, you ship once a week and you have to go to a cab board, a change advisory board, you have to get sign off and approval. I always found that really weird, like we're building this piece of software. It runs on the web, we can ship it whenever we want. It was running on Azure at the time, and so we've got full access to push whenever we want. We make these changes through the week, but we'd have to hold them back. I guess to Adela, our manager, both of our managers at the time, we worked around the system. When the code was ready, we'd build it and ship it through the week.
I was really impressed and proud at this process that the whole team had put together, where we'd commit the code, the test would run, several layers of testing, it would go to staging, etc. Then it would get shipped to production.
So we were kind of, I guess, executing a form of, I guess, continuous delivery at the time, and we would then ship ourselves once a week. I kind of always like to tell this story that at the time, when we'd have a build ready to go, we'd do a form of canary deployments, and so this is where you kind of roll out to a small percentage of your customer base. And we always found that the customer base that would be our test subject was New Zealand. So New Zealand was always our canary. A bunch of reasons for that.
The first country kind of reached this new date. So it was the first ones to kind of roll out into a time.

**Gergely Orosz** (3:21)
When it comes like midnight passes, it's like 1 AM, first country is New Zealand.

**Robert Erez** (3:27)
Bang, exactly. So first country that's of significant size. They speak English, so if there's any bugs or issues or reports, it's kind of easy to understand. But to be honest, New Zealand is small enough that no one really cared if we shipped a bug and had to fix it quickly. So sorry to all the New Zealanders listening.
I think that's kind of this good example of using a continuous delivery technique to ship the code faster than what we otherwise could have if we had these kind of big bang releases. This whole process, I guess, opened my eyes to what progressive delivery, what good CI, CD could be. Yeah, I guess from there, I spent a few years there at Skype and eventually, my wife and I decided it was time to come home to Australia.