EPISODE 1802

[INTRODUCTION]

[0:00:00] ANNOUNCER: Heroku is a cloud platform as a service that enables developers to build, deploy, and scale applications. It was founded in 2007 and was acquired by Salesforce in 2010. The platform supports multiple programming languages including .NET, Ruby, Python, Node.js, and Java, and has features such as automated scaling, database monitoring tools, and a streamlined deployment workflow.

Vish Abrams is the Chief Architect at Heroku and previously worked at Oracle and NASA among other organizations. He joins the show to talk about the history of Heroku, its role
within Salesforce, open sourcing the 12-factor manifesto, the long-standing challenge
of credentials management, and much more. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

[INTERVIEW]

[0:01:02] SF: Vish, welcome to the show.

[0:01:03] VA: Thank you. 

[0:01:04] SF: Awesome. Well, nice to meet you. I'm glad you're here. So, you're the Chief Architect at Heroku. How did you end up at Salesforce? 

[0:01:11] AB: I joined Salesforce/Heroku about six years ago. Formerly, I was at Oracle where I was working on cloud computing and AI, helping them build their public cloud. I joined because I've always had a deep love for Heroku. In fact, my early days of software engineering, I was working at NASA where we were trying to replicate the Heroku platform for an internal project at NASA.

[0:01:36] SF: I would suspect, because Heroku, when I think about it, it was such a pioneer when it came to cloud platform as a service, like the whole get pushed to Heroku deployment. There's a whole generation of engineers that went through that. That was like a really simple idea, but super transformative in terms of how I think it's changed. the iPaaS, and also expectations about deployment. Now that Heroku has been part of Salesforce for the past 14 years or so, what are some of the big changes that have happened to Heroku during that time? 

Well, it's been quite an adventure. I mean, I feel like when Salesforce first bought Heroku, they didn't really know what they wanted to do with it. It was kind of an understanding that developers were really important. force.com was relatively recent and they were really trying to figure out how to appeal to developers. They saw Heroku and said, "Well, they're doing really cool stuff." 

If you think about when Heroku was founded, it was sort of in the time frame where in order to be a developer, you had to also be an operator, you had to be a Linux admin, you had to understand how to spin up a VPS, and do a lot of work. The whole pitch of Heroku was, "Hey, what if you didn't have to worry about any of that stuff?" And so, Heroku went through a period where internally, it sort of just grew fairly independently for a long time, massively grew in revenue, and users, and was very successful, but it sat independently from Salesforce for a lot of its life.

A few years back, around the time I joined was part of a big effort to bring Heroku closer into Salesforce and to start using Heroku internally a lot more. Over time, what's happened is, Heroku has become - Salesforce has become the largest customer of Heroku, in fact. Were used all across Salesforce for internal projects from business technology, we're behind Trailhead, which is the Salesforce education platform. So, it's really become sort of a linchpin of all of the internal systems at Salesforce, and been able to grow in terms of how it's appealed to customers as well.

[0:03:36] SF: Are there certain types of projects that aren't the right fit for Heroku?

[0:03:41] VA: Yes, I think basically projects where companies don't want to spend a lot of investment in building out a platform or operations team, where they want to leverage the advantage that Heroku gives you in terms of being able to just focus on business logic and development. So, we find that application teams that don't have a lot of, say, platform expertise, cloud expertise, or even Kubernetes expertise, for example, derive a lot of benefit out of Heroku.

I had a pretty interesting story talking to a customer who said, "You know, I built up a Kubernetes platform, and my best developer spends all his time maintaining that platform." So, he doesn't really get any value out of the most senior developer on his team because that developer is tasked with supporting the operations of the system. I think that's where you really start to see the value of Heroku, is when you don't have to give your best engineers over to the platform side, and you can have them working on the business logic and the application itself.

[0:04:39] SF: Then, I think, I believe at QCon this past March, it was announced that Heroku would be replatforming onto Kubernetes. What's the status of that project? 

[0:04:48] VA: Yes. In fact, we just took that project to pilot. So, we call it our Fir platform. So, Heroku traditionally has named their platforms after trees. So, the first platform long, long ago was called Aspen, and then we went to Bamboo, Cedar. We've been on the Cedar platform for a long time. There was some D& E in there with various starts and stops, that we don't need to get into the details of. But the new platform is called Fir. So, leaning back into that, the F name for our next generation platform.

That is a replatforming of the Heroku experience on top of Kubernetes. So, deriving a lot of the value of the cloud standards, cloud native standards, specifically like open container initiative images, cloud native buildpacks, open telemetry, all the real underlying pieces of value that Kubernetes offers, but providing the same simple user experience that Heroku is known for.

[0:05:39] SF: Basically, as a customer of that, I'm getting the value of, essentially, containerization, scalability of Kubernetes, but I don't have to manage it myself. 

[0:05:49] VA: Exactly, yes. So, I mean, one way to think about it is sort of like an easy button for Kubernetes.

[0:05:53] SF: Okay. In terms of where Salesforce is in the world, what is their continued interest, I guess, in something like Heroku? In some ways, at least as an outsider, they seem not that related, I guess, as products.

[0:06:07] VA: Yes. Well, you may have seen this big shift towards agents and AI in Salesforce recently.

[0:06:14] SF: Kind of massive.

[0:06:15] VA: The big, new, hot thing. The place that Heroku has sat in the Salesforce ecosystem has been, essentially, when you need to get a little bit lower level, and extend the capabilities of the platform. That's where we've had the most success. We have a product called Salesforce Connect, which allows you to sync your Salesforce data out into an external data store to serve it at sort of high volume for customer-facing applications, et cetera.

We tend to fit really well as ways to extend the platform when you need to get down to a little bit lower level. I think that's going to continue to be the case, especially as we get into agents and AI development. There's edges where the out of the box solution isn't going to solve all the problems, and you need to start extending it. That's where Heroku has been super valuable in the Salesforce ecosystem. Now, we also have a lot of other customers that aren't necessarily customers of the rest of Salesforce that just use this as a really great platform as a service as well.

When we look at our customer landscape, we kind of divide it up into different segments of, here's more customers that are using the other Salesforce and are trying to extend Salesforce products, and customers that are just using us as more of a platform as a service and building all of their systems on Heroku itself.

[0:07:24] SF: Yes. I mean, back to your first point or in terms of certain use cases where you need to extend the sort of default functionality. I think that's certainly very, very true and relevant in the early days of abstraction, which we are very much in when it comes to like AI. I don't think we know for sure, like what the proper abstractions are for building agents, or doing even things like RAG or whatever you're doing with your sort of gen AI project. It's very early days. So, you're going to probably end up for doing something at production, at scale for enterprise. A lot of bespoke work that's going to be very customized to whatever specific problem you're trying to solve. 

[0:08:03] VA: Yes, I agree. I mean, I look at it when I think back about the days of Heroku, we're in a transition period very similarly to where we were when Heroku was founded. We were just starting into web development, building stuff, rapid development, agile development, we're all very new, and people didn't really know how to build things. There was a lot of experimentation, and I think we're in the same place with AI right now, where there's going to be a lot of trying stuff out, gluing things together, and having an easy place to do that is really what Heroku is about. So, we fit in really well in these kind of transition periods.

[0:08:36] SF: Yes, absolutely. Cool. So, I want to talk about the 12-factor app methodology, kind of taking this in a different direction. But for those that are unfamiliar familiar with this framework, can you provide a bit of an overview?

[0:08:49] VA: Yes, sure. The 12th-factor app manifesto methodology was written about 2010, 2011 by Adam Wiggins, who was one of the co-founders of Heroku. It was really his attempt to distill down the essential principles that an application developer needs to think about in order to build an application that is easy to scale and grow as needs change. It has become sort of a go-to manifesto for everyone who's building web applications, cloud-native applications, to kind of remember the principles that help make those applications easy to run.

[0:09:23] SF: Given that it was proposed in 2011, has it changed much since then?

[0:09:28] VA: Surprisingly, not a lot has changed. If you go back to the original principle, so it's 12 factors or principles. So, for things like dev/prod parity, having a single code base, separating your config from code. So, those types of principles, a lot of them are still super relevant. We've done some pretty extensive research and thinking about which principles we want to change, which principles need to be updated. And the essential essence of the principles are still pretty much the same.

There are a few places where the world has moved on a little bit, where either the technology that was in use or is no longer relevant, but a lot of the changes are just in terms of the examples. It's referencing - the manifesto is referencing application or libraries that don't really exist anymore. A lot of the principles stay the same. There are a few areas that we're looking to actually improve things or change things, but they're pretty few and far between, actually.

[0:10:21] SF: What are a few of the challenges, I guess, that organizations run into with running applications and scale, and how do those things kind of map to these principles?

[0:10:32] VA: Yes. So, looking at the journey of people in building applications, there's kind of different tiers that you go through as you start out. When you're dealing with a single application, the main thing that you're trying to worry about is, how does that application grow as demand increases. So, I might start originally just running one copy of the application, and then eventually, I need to horizontally scale and run multiple copies.

So, the first part of the manifesto is really about how you build your application in a way that prepares you for that sort of stateless scale out type of deployment. So, things like separating your stateful services into backing services that can be scaled independently, so that your application tier can scale out. Being fast to start up, and shut down. And then, it's about development principles as the application changes over time, and you need to make updates. How are you going to do that in a way that makes it easy to deploy new versions, to be part of a continuous integration, and delivery pipeline, and things like that? That's the early phase.

Then, as you start to push up against the boundaries of the 12-factor application is when you start getting into things like, okay, multiple applications, microservices, dealing with secrets at scale, having to deal with security incidents, and rotating those credentials, and things like that. So, your needs and concerns start to change as you grow bigger. I think, 12-factor really nailed the first part of that story, how you go from zero to one. Then, when it goes from one to 10, it needs some extension.

[0:11:57] SF: What about things like consistency, reproducibility across like dev environments? Or is that part of the framework as well? Yes, absolutely.

[0:12:06] VA: So, one of the principles is dev/prod parity, which is essentially that you should be using the same services in development and staging as you are in production. That is kind of an obvious one these days, I feel. But in the time when it was written, people were doing all sorts of very different versions of things when you're running locally versus running in production. And that, as we've all experienced, leads to challenges. It's funny how many of that principles have kind of become common knowledge over time, since they were written, and sort of the obvious thing to do.

[0:12:06] SF: Yes, that can get complicated, though too, of mirroring your prod environment, especially if your dev environment's on like a local machine from resourcing and compute perspective. Or even if there's specific data you need and things like that. So, how do people typically try to deal with those issues?

[0:12:53] VA: Yes, it can be. Actually, that's one of the areas where we're looking to extend the factors a little bit. So, it especially gets difficult when you're talking about sets of microservices and you're developing one of those microservices and you want to be able to run copies of the rest of them as they would run in production. This is something that much of Heroku is built on Heroku, and so, this is something we run into when we're, for example, developing our API server.

Our API server has a component that's reserved on an identity server, an API server, or proxy. So, there's a bunch of different pieces. Replicating an environment for development that has some of those pieces that you're working on, that you can iterate on quickly, whereas, other pieces are running sort of in a shared environment that you might be working with other developers on. Those types of things can be challenging. Some of the principles that you start with are making sure that, for example, you're using the same version of a database locally as you are when you're running in production. So, rather than an old practice might be something like, "Okay, well, I'm just going to use an in-memory SQL database for my local development. Then, when I push to production, I'm going to be using Postgres, or MySQL, or something like that." 

One of the issues you run into is, that works up to a point, and then you run into weird edge cases where things start to break. So, it's much better if you can run locally the same version of the database that you're running remotely and things like that. One of the ways that we simplify this in the 12-factor manifesto is to talk about stateful applications being backing services that are exposed through config.

The idea here is that you have a production environment, you have a staging environment, you have a local environment. They're all talking to the database by referring to a connection that's in the environment like database URL that can be changed per environment independently. So, they're all talking to the same version and type of database, but they're not all talking to the same database because you don't want to be doing development against your production database. Having those proper interfaces between stateful and stateless services was one of the ways that it helps you sort of prepare for that kind of development versus production.

[0:14:54] SF: Okay, got it. I mean, how is 12-factor influence the design of cloud applications, even outside of the Heroku ecosystem?

[0:15:03] VA: Well, a lot of the principles have actually - let me back up a little bit. Heroku was created before things like Docker existed. So, the containerization that we used at the time was custom containerization, but it pioneered the idea of what we called Dyno, which was essentially a primordial container, you can think of it. A lot of the way that configuration and config maps work in Kubernetes were sort of modeled after Heroku. The way that there was an open service broker project that came out of the way that Heroku models add-ons and backing services. So, a lot of the way that applications are built, and the way that they're separated, and defined as containers, different containers, having a build process, actually came from the principles of the 12-factor app.

In fact, when I've started talking about 12-factor app in public at KubeCon recently, a lot of people are surprised to even find out that it was written by Heroku. It's become such a well -understood way of building applications that's influenced the landscape that people don't even realize that it was originally a Heroku-led document.

[0:16:03] SF: We've talked a little bit about how containerization has influenced the direction Heroku is going. What about these types of trends like containerization and infrastructure as a service, serverless, things that didn't exist when a lot of the 12 factor was written. It was essentially - have they impacted the original manifesto at all?

[0:16:24] VA: Well, the original manifesto has not been changed since it was written, which is one of the reasons that we're updating it now. But it has definitely impacted how we're thinking about the rewrite without a doubt. One of the things that I think has become clear is that, the thing that manifesto got right is it really focused on the application developer and how an application developer has to think about their code and their deployment of
their application.

When you look at a lot of what's come afterwards, so the Kubernetes landscape, it really is focusing on how to build a platform and not how to talk to an application developer. When you go to KubeCon and experience the tracks, a lot of times, the developer experience for an application developer is not the main focus. So, many times, developers that end up using the platforms that are built by the platform teams have a very difficult time because the level of abstraction is incorrect. Constantly, you hear stories about platform team built this great platform, and then, gave the developers a Kubernetes API and said, "Here, go for it."

What the developer was looking for is more of a Heroku-like experience where I have my code, and I push it, and then it all just happens like magic. Even the design of Docker and Docker files really focused on, okay, what if we took this containerization thing and did a DevOps/platform lens on, what can you do with a container? It's a very different place that you end up when you go in with that view versus when you go in looking at how a developer would think about, these things, how an application developer would think about it.

When we look at the updates, what we're trying to do is go back to that original view of like, "Okay. What if you had a lens on Kubernetes and platforms from an application developer's perspective, what would that look like?" Then, defining sort of the interface on both sides between the platform and the application so that it serves both users much better than many of these platforms do today.

[0:18:20] SF: In terms of going back, and refreshing this, and making these updates, what is ultimately the goal with putting time and effort into prioritizing some of this stuff?

[0:18:28] VA: I think there's a couple of things. One of the goals is just to remind people that Heroku is still around and thinking about making their lives easier. I think there's a memory of Heroku being a - people have fond memories of what it was like to run stuff on Heroku, and they think back, "Oh, back in the early 2010s, I was running stuff on Heroku and it was so easy. But now, I have all of these complex things I have to think about. Now, I have to think about my container orchestration, and terraform, and Helm charts, and there's all of these new concerns." There's this fond memory of Heroku.

One of the goals of updating is, just reminding people, "Hey, we're in a new world, there's new technology, but we can still have that same simplicity." If we go back to the fundamental principles, we can still find that same simplicity and provide an experience that really is like the magic that people remembered. So, it's bringing in that modern development approach, but then, layering back on with the simplicity of the original manifesto. I think we can find that balance.

In addition, I think there's other goals, which is, there are certain areas around the edges of application development that are still really hard. The only way for them to get easier is for a bunch of application, and framework, and platform developers to work together on creating common solutions. So, one example of this that we've run into in Heroku is the complexity of dealing with credentials at scale, especially long-lived credentials.

I think, there's a best practice that's evolved around, okay, well, you put your credentials in something like Vault from HashiCorp that'll help you manage sets of credentials. But the real solution here is to move away from long-lived credentials, and that really requires application developers to understand something like workload identity, where you can have a short-lived credential to authenticate to things. And then, application developers, and frameworks and platforms collaborating on making it super easy to use those things.

So, if there are some areas where we can really move the industry forward, but it's going to require some excitement and collaboration from a bunch of different groups. Twelve-factor was a way for us to do that in the early days, and I think it's a way for us to do that again.

[0:20:36] SF: Yes. I mean, that's interesting because I think, one of the things that's happened over the last 14 years as well, has been since the framework originally came out was, it's also been the rise of the API. Everything essentially has an API, and we're essentially, we're building, we're really like interconnecting a lot of times between APIs, either internally or externally. We live in very much more in a world of essentially credentialing all the time, essentially, in order to just stitch these things together. 

[0:21:04] VA: Yes. I think what you end up with, especially at scale, is hundreds or even thousands of unique credentials that you have to deal with. As soon as you have a security incident, that becomes a huge pain. So, it's much nicer if you can minimize the number of credentials. Another area is around telemetry. I mean, we kind of have standards that are coming around about how to do telemetry, but the implementation of those standards across platforms from different application developers are all bespoke. We would all benefit from having more consistent ways of doing that.

The industry is moving forward. OpenTelemetry is a big new project that has garnered a lot of attention. And it's really connecting those pieces together into the platform and application developers, so that there's sort of one consistent story for how you do things like application metrics and traces. The original manifesto focused on, well, you throw your logs to standard out, and then it goes to some backend system, and you don't have to worry about it. I think we need something a little bit more robust than that for the current world.

[0:22:02] SF: As part of this update, another thing that you've done recently is open source the 12-factor. I guess like, what is the significance of open sourcing it? You want to essentially update and continue to build this as a community-led project?

[0:22:18] VA: Yes. The intention here is, everybody loves 12-factor. There's a lot of appreciation for it in the community, but we don't want it to be something that's just Heroku telling people what to do. It should be something that community is saying, these are the things that we agree on are the right practices. Using that as a platform to sort of influence other development communities, and framework communities, platform communities to sort of build things collaboratively, we think that's the right way to go.

The only way for us to do that is to expand the control of it to a community-led project. When we say open source, usually, that refers to source code. As it stands right now, it's more like documentation. So, we use the CC by license, but we do expect bits of code, and software to exist in the fringes around the manifesto. So, for example, we might have a local CLI that helps you implement the principles of a 12-factor platform locally, so that you are working in a similar environment to how you would be if you're on a platform. Or we might have defined different standards around 12-factor where some don't exist in the community already, just to make it easier for us to work together on things. There may be code at some point in the future. Right now, it's essentially just documentation at the moment.

[0:23:35] SF: Were there any challenges with or concerns, I guess, with making the framework more public and more of a community-led thing?

[0:23:44] VA: Yes, I think there were. I mean, primarily, we spent months and months investigating it ourselves and developing an opinion about what does need to be updated and where should it go. As always, when you take something to the community, not everybody in the community is going to agree with you. So, the concern of course is, like we have these great ideas. and we bring it to the community, and they take it in a totally different direction that doesn't really align with where Heroku is going. That would be a big miss if that happened. But we felt pretty confident that the ideas and the direction we wanted to go are solid, and what we did is we started out by inviting some trusted community members to come in, and review it, and see if they had the same opinions, and develop a collaborative start. Which resulted in an initial maintainer list that's broader than just Heroku that has people from different companies, and the community as a whole that are interested in it.

We've been actually quite surprised with the interest, and the intention that we've gotten so far. We didn't realize how much people cared about 12-factor and how much it affected their lives. We've had a bunch of people jump in and say, "Wow, I've been using this for 10 years. It's great to be able to participate in the next version of it."

[0:24:54] SF: Given the original 12-factor, I mean, it's even the name, you have these, essentially 12 principles. Now that you open source it, and you begin to expand it, how do you not end up with 28 of these or something like that to become an unreasonable amount for someone to like really keep in their head? 

[0:25:11] VA: Yes, that's a good question. I mean, so let's go back to why 12 originally. I think, 12 was somewhat of an arbitrary choice. I mean, there's a lot of reasons that 12 is a good number. I think you want somewhere more than five and less than 15. So, you could have picked nine, you could have picked 12. In terms of the number in the future, I think what we really want to lean into the name recognition of 12-factor.

So, our intention going in is if we want to add a new factor, we're going to delete or combine another one. But the great news is, there's so many ways to slice an onion. You can, you can take the principles and you can massage them into different forms and still achieve the same goal. So, I can't imagine a place where we couldn't find a way to convey what we needed to convey in the number that we choose. If we choose 12, we'll find a way to convey the principles as 12.

What we've discussed so far, for example, we're talking about adding an identity factor, like a workload identity factor. To make room for that, the discussion has been, there's a few of the other factors specifically around processes which have a lot of overlap. So, we're probably just going to combine a couple of them into one and cover the information from those two as a single point, which would give us room to make a new factor as necessary. There will be trade-offs like that that we'll discuss over time.

[0:26:28] SF: In terms of tying us back to some of the things that we were talking about at the beginning, in terms of the rise of agentic AI, and everything that's happening with agents and Salesforce is making a big investment there. How do you see, if anything, that impacting, I guess, how the 12-factor gets updated? Are there new things that need to be considered because we're sort of moving in this direction of agentic AI?

[0:26:54] VA: I think there will be. Ultimately, I think we have to realize that AI applications are just applications. Principles that apply to applications in general will likely apply to AI applications as well. That said, what we've discussed is, after going through the principles of 12-factor, potentially taking different lenses on 12-factor. So, for example, there are 12-factor focuses on stateless web applications primarily running on Linux, right? But a lot of the principles would apply to other types of execution environments. For example, you could imagine WebAssembly being an execution environment that you'd want to talk about. A lot of the principles that apply to 12 -factor would also apply to that environment as well.

We might have sort of a different lens on 12-factor. Twelve-factor for WebAssembly, for example, or we might have 12-factor for AI applications, which would maybe emphasize some of the factors over others or say, "Here's how you interpret this principle in this context." I think that's the route that we're likely to take with some of this future thinking stuff. I will say that one area that I've seen a lot of challenge happening in the AI agentic space is, what I think is an agent interop.

The definition for how one agent would call or talk to another agent. There's been some stuff released recently around this, but it's still kind of an open question. One of the challenges is how you share data and authentication between the different agents that need to talk to each other, especially when you're considering, I have an agent over here that's maybe doing RAG with this particular data store. I'd like to go make a query to another agent that's better at this particular type of workload, but I'd like to send the data store over there for them to use, so they can also do RAG against that data store. Those types of interoperability are authentication and backing service-like questions.

When we start thinking about updating how we talk about workload identity and how we talk about backing services to apply to modern development practices, we'd like to include a lens towards doing that kind of thing in the future. I think if we get the pieces right, they will also apply to the agentic collaboration workflows as well.

[0:29:01] SF: Yes. I mean, it kind of goes back to what we were saying in terms of all the interop challenges that now all engineering organizations essentially face with the rise of the of the API. Agents only accelerate those problems, essentially, because they'll need access to data, tools, sharing across agents, and all these types of things. So, it's a gigantic, essentially, interop challenge.

[0:29:23] VA: Yes. The good news is, we need to solve it for applications and 12-factor apps. If we solve for 12-factor apps, hopefully, we can extend that to working for agents as well.

[0:29:33] SF: How does all this influence the product investments that you make for Heroku.

[0:29:40] VA: Yes. It's very interesting. This type of thing, the type of things that we're thinking about making easier in 12-factor, we've run into directly in product development in Heroku, both internally usage of Heroku, and also, our costumers use Heroku. The lens that I'm taking to the 12-factor work as we work in the community is like, these are real problems that we've seen and we like to solve. But we like to solve it in a way that's broader than just Heroku, and just take those solutions and offer them as product features on Heroku itself.

The process is more like, let's bring stuff to the community, get everyone aligned on the right way of doing things, so that when we release features on the product side, that they're very aligned with the sort of community effort of 12-factor as well. I think it happened a little bit in reverse in the early days of 12 factor, which is we've released product features, and then built 12-factor to sort of represent the product features that we already had. We're attempting this time to do it in the community first, and then, release features according to what we find. But we do end up prototyping a lot of what we're bringing to the community internally before we do that. 

[0:30:46] SF: Then, going back to where we very first started talking about this magical experience that I think developers came to know like Heroku as, in having this very developer friendly experience essentially for building, and deploying, and managing code. Now, given that it's within a large organization, it's been around for a long time. How do you continue to keep sort of the developer first mindset in the way that you're, you're building the product while also living within this larger company. I'm sure there's competing priorities and things like that, that come along. I guess, how does your role tie into that?

[0:31:25] VA: Yes, it's definitely a challenge. I mean, I think the primary way we stay in touch is by using Heroku ourselves. So much of our product is built on Heroku that we are dealing with the same challenges that developers on our platform deal with on a daily basis. That keeps us in touch with really serving customer needs. So, the classic phrase is, "Eat your own dog food." I think, we like to say, "Drink your own champagne," because maybe it sounds a little better to drink champagne than eat dog food. But it's the same story, it's like, that's how we know that we're building the right thing because we feel the pain ourselves.

[0:32:01] SF: Well, given that Salesforce is the largest customer for Heroku, and you're drinking your own champagne. What were some of the - I got to assume that you must have run into some significant scale challenges with just the kind of load that you're probably going to see Salesforce as a customer. What were some of those issues that came up, and how did you essentially manage those, and make changes essentially in terms of the way that Heroku operates to meet the demand of Salesforce?

[0:32:25] VA: Yes, that's a good question. Let me think back a second to see if I can come up with a specific example. One of the internal users we had of Salesforce was a company that was doing quote to cash, basically taking complicated orders and figuring out the pricing for those orders. That ended up running on Heroku. I think that that's the one that's pushed our scale the most. Funnily enough, the places that you would expect the scale to be a problem are not always where you think they're going to be. We actually were totally fine scaling up to match their execution demand because we had our multi-tenant platform that has a bunch of extra capacity because it's a lot of small users, and we were able to scale up to meet their demand fairly well.

The places where we started running into issues was actually around generating billing and invoicing. They were running so many short, what we call one-off dynos. So, short execution dynos, that the billing system failed to build the bill. I think this is a really good lesson, which is, the scaling issues never hit you where you think they're going to hit you.

Another example was a customer that just had a huge amount of applications and teams. The thing that broke for them was the display in the WebUI. They couldn't actually see all their applications in one view because it would time out. It's really fascinating to see that you never really know where you're going to hit the bottleneck until you run into it. It's hard to prepare for all the places that those bottlenecks are going to hit. So, a lot of our scaling issues actually happen around the edges, in the small - the extra systems. You're building the main system, and you're building it for scale, and you're really thinking about and testing all the edge cases. Then, all the auxiliary systems, which you're not really thinking about, oh, that part needs to scale, that's where you tend to run into the bottlenecks.

[0:34:12] SF: Heroku originally was focused on Ruby on Rails. How has the language support changed over time?

[0:34:19] VA: So, we introduced multiple languages. I see the release was actually the release where we went beyond Ruby. That was back around the time of the acquisition. So, number of years ago now, 14 years ago now. We've seen language adoption actually be kind of gradually improving over time or improving, growing over time. We just released our .NET support. .NET had been supported through community buildpacks for a long time, but we have released an official buildpack. We're seeing actually a lot of excitement around that one, which is a little bit surprising to me. I didn't realize that that was going to be a big hit, but yes, there are a lot of enterprise customers out there that are building on .NET as their main backend.

[0:35:01] SF: I noticed there's certain pockets of tech sectors based on geography, where certain languages or certain frameworks are really popular. Like .NET might be a really hot thing in, I don't know, I'm just throwing out - Brazil or something like that. It might be different in another state, or country, or something like that. What's it take to essentially support another language?

[0:35:22] VA: The main thing that we like to do is we have a language owner when we have a new language. That's kind of the line between whether or not it's going to be an officially supported buildpack or it's going to be a community supported buildpack. So, you can run any language on Heroku, and people do. But, at the point when we decide that we have a language owner, or we want someone in the community who's familiar, who's keeping the buildpack up to date, making sure that it's following sort of the best principles and practices of the language and the frameworks in that language. 

That's the main thing that keeps us from having a lot more supported buildpacks. It just takes a significant amount of time to have an engineer focused just on a particular language. You'll notice that we don't have Rust as an officially supported one yet, even though there's a ton of excitement about Rust. I love building stuff in Rust, but have to use the community supported buildpacks for that one at the moment. As we see demand for a particular language coming up, we decide to invest some time or a person into that community, so that they can keep the buildpack up to date.

[0:36:21] SF: Well, so what's next for Heroku and also for 12-factor?

[0:36:25] VA: Yes. So, you mentioned earlier, you asked about our replatforming onto Fir or onto Kubernetes that we call our Fir platform. Really, that has been a ton of work over a number of years that we're now finally getting to pilot. But the big reason for making that move wasn't just to provide sort of cloud-native standards to our customers, but also, because it allows us to innovate much more quickly. So, we have a whole bunch of new features that actually can be seen. We have a public roadmap on GitHub, where we show the stuff that we're working on and people can make suggestions, but there's a lot of stuff lined up about behind or on top of the Fir support.

We introduced with Fir, ARM dynos, for example. Another thing that's coming out soon are GPU dynos. We better telemetry support, better add-on support. So, we have a lot of pieces that we've just been kind of getting the foundation done in order to be able to produce features more quickly and really provide some new stuff. Then, on the 12-factor side, it's really about enabling this easier development for sets of applications. So, one of the areas that I think people struggle with is, "Okay, I've got a single application, but I've grown. My team has grown. There's three teams now, and I need to divide this up into multiple applications and take a bit of a microservices architecture."

Managing things like multiple microservices as part of a single view, I think is a really important place where we need to push the industry forward in 12-factor. So, you'll see exploration of that space in 12-factor, and then, follow on in the future features helping support the commonly agreed upon ways of doing those kinds of things.

[0:38:01] SF: Well, Vish, thanks so much for being here.

[0:38:03] SF: Thank you. This has been great. 

[0:38:05] SF: All right, cheers.

[END]