Securing Developer Environments with Pill Club’s Cameron Seebach

What do DevOps, contraception availability, and a composer named Ham Sandwich Quantum Party have in common? These are all pieces that make up the unique and wonderful first guest in our DevOps Fireside Chat series, Cameron Seebach. Cameron is the DevOps Engineer and Tech Lead at The Pill Club, a company whose mission is to make healthcare more accessible by building digital primary care that closes gender, socioeconomic, and racial gaps in our current health system.

During the chat, Twingate Co-Founder and Chief Product Officer Alex Marshall dug into Cameron’s journey to DevOps, how he is securing developer environments today, and why Zero Trust is the next step in security maturity.

DevOps Fireside Chat: Pill Clubs Cameron Seebach

Fireside Chat Transcript

Elliot Volkman: Hello, good morning and good afternoon. Thank you for joining us for today’s presentation. Today we’re going to be chatting with Cameron over at Pill Club. We’re going to be focusing on two different elements, for the start of this presentation we’re going to be looking at a couple of use cases that Cameron is putting into play over at Pill Club on how he and his team are securing their dev environment. And then we’re going to shift over to a fireside chat, which will be co-hosted by our CPO and co-founder Alex.

Before we get started, let’s introduce our guest speaker today, Cameron Seebach. He is the senior DevOps engineer and team lead over at Pill Club. As the DevOps engineer, Cameron was the first to identify opportunities for expanded use of cloud computing services across the Pill Club.

Elliot Volkman: Cameron has led the implementation of systems that handled millions of SMS messages monthly, and leveraged Elasticsearch to optimize operation workflows, passionate about all things AWS, Cameron builds solutions that meet the scaling and challenges, strict security needs and compliance goals of a hyper growth healthcare business. He composes electronic music under the name of Ham Sandwich Quantum Party, and holds a BA in computer science from Cornell College in Mountford, Iowa.

Alex Marshall: If you’re just joining us right now I’m talking to Cameron Seebach who’s a senior DevOps engineer tech lead at Pill Club. I really appreciate you guys over the presentation. I’ve learned some things, but I definitely have some more questions. I’d love to go into that, but just a quick introduction of myself. I’m one of the co-founders at Twingate, currently the chief product officer, and we’ve seen this through from the very beginning.

One of the things I think we’ve been very passionate about is, if you look at the Zero Trust space, it makes sense as this direction that things have been going in, and makes sense as a model that’s a lot easier for companies to use and adopt.

Alex Marshall: We’ve been really passionate about making things easy to adopt. We’ll have some questions about that for you in a second. But just to jump into a little bit more background and depth on some of your experience with the Pill Club. Could you give us a little bit of background? I think there’s some sequencing things here, there’s sort of the background of what you were doing previously to gain secure access into some of your more privileged environments. Then I think that, that’s sort of a parallel effort to some of the changes you were just describing in terms of the dev environments were more stable and easier to get them running.

Could you talk a little bit more like what that decision making was like, did those two things come together? Was one independent of the other? I’d love to hear a bit more about that then we can dig in.

Cameron Seebach: Sure. Specifically this kind of access into a dev environment wasn’t really possible before we adopted Twingate, so there’s that piece right there that we should definitely dig into, but to go back a couple of steps we were using, across the company, a traditional VPN.

We tried working with a split tunnel setup that many of you are familiar with and that has its own headaches and challenges. The VPN went down all the time because it was trying to serve too much traffic. Whenever we had a user come in that needed to be added to the VPN it was always a hassle that involved sometimes restarting the whole service and taking down access for more folks than we would like.

Cameron Seebach: We made the switch to Twingate, I think maybe it’s been a year and a half or something like that ago, and the experience is way different. We were using a really hefty machine to power our VPN solution before, because it was serving everybody’s traffic, now we spend maybe $40 on our machines to serve over 400 users. That kind of came parallel to the stuff that we were doing with these dev environments, where we started solving the access challenges for folks across the company with Twingate, and then we took a look at this engineering environment that we had built and said,

“Well, this is okay, but how do we enable really good collaboration without having to install a heavyweight VPN solution on every developer’s machine?‚

Because Twingate works essentially on top of existing DNS standards we can just drop the Twingate connectors inside these machines, they’re lightweight, they only take up resources when they’re being used. And it’s been such a huge difference in what we can do.

Alex Marshall: When you think about what this Zero Trust approach lets you achieve, how do you think you would’ve actually deployed these dev environments without something like this? Like, what would the approach have been?

Cameron Seebach: Gosh, it would’ve been something probably like we would’ve had a Secure Shell forwarding system that was inside every developer environment that would’ve talked to some kind of a central service that we would had to spin up ourselves. There would’ve been that central point of failure for this that I would’ve been really worried about, and there would’ve been a lot of access and administration hurdles that we don’t have today. There would’ve been things like making sure that developer machines had proper SSH keys. There would’ve been things like making sure that we had scoped the security and firewall groups on that central resource properly. And these are challenges that I don’t think I would’ve been willing to take on, it would’ve been a lot of additional work on the DevOps plate.

Alex Marshall: Before I go to the next question, on that point, you mentioned earlier that one of the advantages is that you can have things administered centrally. How do you think that shifted the responsibilities between configuring the environments, make sure they’re upper running versus maintaining access, and maybe some things actually don’t work as you’d like right now, but curious how you’d look at that split right now.

Cameron Seebach: One of the great things that Twingate enables us to do is, to link in our existing single sign-on provider for managing access. When people enter or exit the company those settings are applied immediately on the day of, whereas if it were up to the DevOps team to administer that access, we have seen unfortunately that sometimes we miss a departure or an entrance of employees, so that’s one part that is made really easy via the deployment of Twingate. Then, to answer the other part of your question, you’re asking specifically about the ways in which the DevOps team has shifted its responsibility sort of?

Alex Marshall: Yes, because I guess you’ve answered it a little bit, so it sounds like before you might’ve been responsible for defining, building, defining processes around the environment, and then also managing access. It sounds like where you’ve ended up as this place where you can be more focused on the environment itself and then access becomes a responsibility that you can either share or move on to another team, like what you’re saying with the IDP integration.

Cameron Seebach: Yes, that’s exactly right. I don’t spend any time managing, I don’t spend any time pushing people in and out of groups anymore. I spend my time when I’m working on developer productivity concerns like this, I spend my time just on making that experience of installing the software and configuring it easier, which is a huge reduction in the amount of left effort required.

Alex Marshall: Great. For anyone listening or watching, I think that’s a really important point actually, that you can split these responsibilities where I think previously you had… I think with past or existing methods, you sort of link your network infrastructure the way you’re configuring your environment to access, now you can actually split those things out. We’ve talked about some of the good things, tell us a little bit about what the process was like in terms of how difficult it was, did you run into limitations or challenges along the way?

Cameron Seebach:There were a few challenges. The one that is most prominent in my mind is that we had to come up with an environment that a user in the engineering side could check in and out as needed, and that basically means that over the course of a day an engineering user might check out like let’s say environment dev one or something like that. We had to find a way to say, “Okay, a user could check out one of these environments, but we don’t necessarily know when they’ve stopped using it.‚ We ended up solving that basically with an API integration with Twingate, where we checked to see if the resource that is attached to that particular developer environment is up and running or not. And if that developer environment is not running, we give it 30 minutes and see, and we check a couple times. If it’s still not running at the end of that time, we check that developer out of that environment and leave it for use by other developers. So, that was one of the challenges that we had.

Cameron Seebach: Another challenge was making sure that while we were developing these developer environments, that we were keeping in mind our audience. Because there are many different kinds of developers, even in our team of 40 or 50 engineers. There are some developers who want things to be up and running and don’t really care how it’s done, and then there are developers who want to really get into the details of how that MySQL password is configured, or which database tables are loaded at app startups, stuff like that.

We’ve done our best to strike a balance between, “Hey, these are reasonable defaults that we think work for this environment.‚ We run our end-to-end tests on that environment to make sure that it’s running and building okay. But we also offer a lot of knobs and buttons for engineers who want to tweak things to their satisfaction and striking that balance was challenging at first.

Cameron Seebach: It was challenging also for engineers to adopt the idea that they were not necessarily in control over every detail of their environment. I think as we’ve gone on through this implementation, they’re starting to discover that it is actually better to have a set of defaults than to configure everything from scratch and just in terms of the time savings. But that has been a shift in the way that our engineers think about their development environments.

Alex Marshall: I think, a really closely related area in all this, which is that you can make the technical, the configuration, or the engineering changes, but there’s actually a lot of internal change management and communication that has to happen. How did you guys approach that and did you run into any resistance, and how did you overcome some of those things?

Cameron Seebach:We did our rollout in a couple of different ways. We started first with the front end focused engineers, who typically are a lot more interested in just getting something up and running. Once we were able to make some of those engineers into champions for this approach, the rollout across the rest of the team was a lot easier.

On top of that, we started saying things like, “You don’t have to use this environment, it’s totally optional. I can tell you, it will save you a couple of days every time you need to rebuild something, but don’t need to use it.‚ And that kind of a non-aggressive approach towards rolling this out really helped us. Because when people feel that they have a choice about whether or not to do something like this and they can evaluate the benefits and risks a lot more easily than if it’s a forced thing.

Alex Marshall: That makes a ton of sense. All right, I got one more question here actually from one of the attendees, and then want to move on to a little bit more about your background and what your journey through DevOps has been. The question here from Eran is really around whether you had any security compliance or risk hurdles that you had to overcome now that your remote access solution is tied to a third party, or Twingate versus self-managed VPN infrastructure.

Cameron Seebach: That’s a great question, there are a number of ways to think about these kinds of trade offs, and the way that we think about them at the Pill Club is most of the traffic that’s flowing through Twingate is encrypted already because a large portion of it is HDPS traffic. So, that is one thing that we really like. The other thing is, there are always going to be security and compliance trade offs to make when you are considering any kind of a VPN solution. The way we think about them is that I’d much rather have a team like Twingate that is dedicated to making sure that the environment is secure and compliant than trusting my team of three people to do their best, to keep up with security patches, ongoing cyber attacks, and risks like that.

Cameron Seebach: We feel that’s a great trade off to make, as we were talking earlier, have a separation of those concerns, have my team be focused on providing a lot of value to the business and also to auditing how we’re doing with the Twingate configuration, but then have the Twingate team be primarily focused on their side of the equation.

Alex Marshall: Great. I wanted to switch gears a little bit just for anyone else listening in the DevOps arena or something, or people that are maybe interested in getting into DevOps. Could you talk a little bit more about how you got into the role. I know you talked about this a little bit earlier, but if you talk a little about your journey and maybe along the way, explain what advice you’d have for somebody that’s looking to do the same.

Cameron Seebach: Sure. Gosh, where should I start? There’s a lot there. When I was 20, I started out as an IT administrator at a law firm, and I was the most junior of the most junior folks there. I remember doing things like foam cleaning keyboards, and if a printer went offline I’d go over to that person’s computer and reinstall the drivers. And although it looked like magic to the folks that I was helping and it put a smile on their face, it always felt a little bit to me like I could do more.

I worked there for a couple of years and then decided to go back into school and actually learned some programming, so I got a computer science degree and I loved it. It was amazing. Just the idea that you can type these magic incantations into the computer and something pops up on the screen or sends a request somewhere, that was really cool. I moved to the Bay Area, I started work at a FinTech company as an application developer, it was a Ruby on Rail stack. Ruby, it’s a fantastic language, but it is very slow for those of you who know.

I worked there for a couple of years and I realized that I was always interested in the infrastructure pieces of something, how the database connections worked, what kind of proxy was in place to send requests from the front end server to the backend server? Things like that. And when I got the opportunity to switch jobs I started thinking more seriously about this, and I raised it with my VP of engineering one day. And I said, “Hey, I have had a lot of experience working with infrastructure at our company. I’d like to eventually become a DevOps engineer.‚ And I remember that conversation and he basically looked at me for a second and he said, “Okay, let’s start training you on CI processes and get you into our AWS environment so you can start building some features there that we really need.‚

I made the switch from being a pure Ruby on Rails, Java application developer, to building things that were kind of in the middle there. Like, I worked on a lot of continuous integration environments. I worked on a lot of spinning up Docker files for new applications and containers, and that was all really fruitful for me. What I’d recommend to folks who are interested in becoming DevOps engineers, I’d recommend that you get a wide swath of experience.

You don’t necessarily need to be an expert at everything you do, but you need to have some familiarity with the whole stack. And the easiest way to get that in my opinion is just to ask if you can help out on a project like that. You probably are in an environment where there are existing DevOps engineers that you can work with closely on a project. You may be lucky enough like me to have a VP who can see that you want to do this stuff and is willing to take a risk. And there are so many good ways to get into the field now.

Alex Marshall: Absolutely. I think there’s some really great advice embedded in all that, which is that I think a lot of times, especially talking to people earlier in their career, trying to figure out what they should do. I think that the consequences of early career decisions seem much greater than they really are, and I think I’m paraphrasing a little bit. So, the advice would be, just start somewhere where you’re interested, so maybe in your case it’s on the application side, but then continue to explore what else might be interesting until you find something that you’re really passionate about. And then, I think, the big one as well is just other pieces of advice where you just ask. A lot of times it’s just a question of asking for something and often you’ll often get it. I think that’s really great advice. Anything…? Yes, go ahead.

Cameron Seebach: I was going to say, especially in the DevOps space where it is so hard to hire DevOps engineers. If I were a VP and I saw that somebody had some infrastructure talent and they asked me if I could have them do an infrastructure project, I would be all over that.

Alex Marshall: Yes. From what you’re saying, your story is the story you found out about the role, because you saw that someone else was doing it, that there was a team, but there isn’t obviously a degree in DevOps as far as I know, it’s computer science. How would you suggest that people find out about these things or maybe develop skills and make them good candidates for these different roles?

Cameron Seebach: One of the ways that’s really powerful for people that I feel is often overlooked is that pretty much in any domain that you’re interested in, all the way from art history to sports, there is some way that you can build either a computer application or a web application to improve, or further your understanding of some field like that. I had a friend in college who built a baseball game simulator, because it was something he was interested in and he ended up showing that to a potential employer who gave him a job basically because of the cool work that he had done on that project. Personal projects play a huge role in furthering our understanding of what’s involved in building a web application.

Cameron Seebach: That’s one thing I would highly recommend is, try building a small application completely from scratch, pick a language that you love, pick a topic that you’re interested in and don’t feel like there’s a pressure to produce anything, the goal is to explore. And if your exploration leads you to someplace interesting, great. If it leads you to someplace where you decide you don’t want to do any more of it, that’s fine too.

Alex Marshall: I think that’s wonderful. I think it’s really just being curious. I think the other advice that I think I’ve gotten myself along my career too, is that a lot of times if you want to learn something new or explore, it’s actually good to set yourself a goal. If your goal is, I’m going to produce this app in this particular domain, I just learn a bunch of things to get there and a lot of things that I might not even know about. Maybe just like a few more questions here, then I think we’ll start wrapping up. Moving over to current events, sort of like what’s happening in the world today? What do you feel more broadly on the business side, like some of the biggest threats to this in general at the moment, do you have any high level thoughts on that?

Cameron Seebach:High-level thoughts on that, it’s easy, especially for companies that have grown their cloud setups organically, to find that you have a whole bunch of resources that you don’t think are open to the internet, but actually are for one reason or another. And it doesn’t have to be a huge exercise to go and find these, there are great automated tools today that will either scan your existing cloud configuration, if you’re working out of a configuration language like Terraform, you can scan the Terraform directly. And that’s one thing that is very top of mind for me is, whenever we develop a new service application, making sure that it’s behind either a tightly IP whitelisted firewall or that we know it’s going to be public and take appropriate precautions with a web application firewall or some other product that sits in front of it.

Cameron Seebach:As with the Log4J disaster that happened last year, it’s not always the big applications or details that make something a problem. It can be something as small as a logging library. Top of mind for me also is keeping dependencies up to date. There are great automated tools that can do this for you today as well. We use AWS inspector for a lot of that and GuardDuty, but there are products from Microsoft and Google that do the same thing.

Also, top of mind for me is the individual permissions inside your system. Again, a lot of companies that have grown their AWS setup organically, there are some folks who’ve been with the company for a long time that probably have broader permissions than they need. Coming back to this automated tools thing that I keep harping on, there are great tools that exist that can tell you exactly which resources a user recently accessed in a 90 day period or something like that. And generally you can scope the permissions pretty tightly to make sure they need no more than they actually need. Those are kind of the top of mind things for me.

Alex Marshall: I think it is really a fact of modern software that there’s a massive number of dependencies, and so I think a lot of the automation is definitely very important. We hear that quite a bit. I think one of these often comes up on this topic, and this probably be a good topic to wrap things up on is, whose responsibility is that? I think a lot of times there’s diffuse responsibility around who should really be in charge of checking dependencies, managing security, things like that. And I think a lot of organizations would say that it’s everybody’s job, but then at the same time it sort of becomes no one’s job. don’t know what you’ve seen work well or not work well around that.

Cameron Seebach: One of the things that has worked really well for us, the Pill Club is we actually set an OKR on the DevOps team around security risks like that. We said, “Based on the severity of the issue scored on the typical CVSS scale, if it is a critical issue, we need to have it 95% of the time we turn it around within two days or less.‚

We are working on getting there, it’s a pretty big ask, but that helps give a sense of urgency to fixing these things when they are tracked in a company level objective. And what that has ended up looking like for us is that we partner very closely with backend engineers who need to make the updates and test the functionality to make sure it’s still working, but we will typically set up the base change in a poll request.

That’s basically like, “Hey, bump this library version to this level and see what breaks, and then it’s your job to fix it‚. Whoever has been assigned to this particular issue. And that’s worked pretty well for us so far. Intentionally we set a 95% target on these things, because we know there are going to be some issues that require very large changes that we may not be able to get to within that timeframe, but it sets a pretty high bar, and we have seen engineers on the backend teams make time to fix these things because they also know how important it is. And they also know that I’ll keep bugging them if they don’t get it done.

Alex Marshall: That’s great. I think there’s actually a lot of companies that are learning around actually incorporating some of these objectives into OKRs. That sounds like a pretty fantastic processor guys aside. Any closing thoughts, like we’ve covered a lot of things here, we covered some of the specifics of what you’ve done internally, some of the thoughts on your trust, filling things out internally, your background, anything else you’d like to say to the folks listening?

Cameron Seebach: Gosh, that’s a big responsibility Alex. If you are in this webinar, chances are that you have some interest in the technologies at play here. First of all, I commend you for that. There are a lot of ways that you can improve and enhance the security posture of your organization. And whether you decide to adopt a Zero Trusts technology or not just the fact that you’re here listening is really good. The things I would close out on would be to say, one of the central challenges in software is how do I make use of the work of other people?

Cameron Seebach: Because you’re always running out of time or have limited resources to accomplish a task. And most of the time there is someone out there who has solved that problem already more or less. There’s a saying, “There are no new problems in software engineering.‚ When I think about adopting any new technology, be it Zero Trust, be it Docker containers, be it changing IDEs, the number one thing I think about is does this make use of a lot of work that someone else has already done? And that’s one indicator of how much use it can be to you. That might be my parting thought actually.

Alex Marshall: I really like that. Well, great. Well, Cameron, it’s been really enjoyable chatting with you. Thank you for presenting and sharing all your thoughts. We really appreciate you guys as a customer as well, but thank you very much. I think we’ll wrap things up here.