Successful software always gets changed.
– Frederick P. Brooks
What is true for software in general, also holds for APIs: successful APIs get changed. The reason is simple: successful APIs are used by various API consumers who demand new features, extensions, bug fixes and optimizations. From this perspective, changes in APIs are inevitable.
But this is only half the story. Successful APIs are used by various API consumers, who rely on the API to remain stable to avoid breaking their existing integrations. Their software clients have dependencies on the API. Even a small change in the API is enough to break some of the clients consuming the API. From this perspective, changes in APIs are undesirable.
Obviously, there is a dilemma. And it is the task of the API provider to resolve this dilemma. In this article, I want to give you some tips for dealing with it. To deal with change and versioning effectively, we need to take a life cycle approach. From the perspective of the API life cycle, we look at the API from its inception and design all the way to its retirement. And I hope it does not come as a surprise to you – but dealing with change is also about being prepared. So you need to start thinking about changes to the API way before the concrete change proposal is appearing on the radar.
We can divide the life cycle of an API into two big phases: the design-time phase before publishing the API and the phase after publishing the API. In these two phases, you need to deal with changes in a very different manner; in fact, when transitioning from one phase to the next, your mindset about change needs to switch 180 degrees.
Before publishing the API
When you initially design an API, you might not even think about future changes. But they will come. So here are three tips on how to support the evolution and versioning of your API already during design time.
Before publishing the API: welcome feedback in iterative API design
Before the API is published, change is great - you need to welcome it and seek it out. Use an iterative development methodology, elicit feedback from beta customers, adapt the API spec based on the feedback you have received on API mocks and beta-versions of the API. This iterative development should ensure that the API you create is actually quite stable, and should reduce the likelihood of immediate future changes. But don’t stop with your API design here and publish the API.
Before publishing the API: prepare an API design pattern for versioning
Create an API design pattern for versioning right from the start when you publish the API for the first time. It might seem superfluous at the time of initial design, but it allows you to roll out consistent versions in the future and sets the right expectation for API consumers. Ideally, the API design pattern for versioning chosen for your API is documented in your API design style guide.
While the most commonly observed pattern is realizing API versioning as the URL path parameter, there are various alternative API design patterns for making major versions accessible:
- Realize API versioning in the Accept header: the client can use the HTTP Accept header to explicitly indicate the version of the API. The Accept header still contains the MIME-types as usual, but in addition, the version is appended. The URL of the API does not contain any version information. For new versions, the URL does not change, but the request header.
- Realize API versioning as URL path parameter: the most common technique for versioning uses a URL parameter with the version number
- Realize API versioning in a custom HTTP header: a custom HTTP header could be defined for the version
- Realize API versioning as a query parameter: a query parameter could be defined for the version
- Realize API versioning as a new subdomain: a subdomain could be defined for the new version
Before publishing the API: anticipate future change
Anticipate future changes to your API and accommodate for it in your API design. Everyone knows that this is difficult as there are no hard facts about the future. But it is important preparation for the evolution and versioning of your API. Base your assessment on experience and domain knowledge. Maybe you know there is an upcoming industry standard in the domain of the API that needs to be supported. Maybe you know that some scalar data fields will likely need to become an array in the future. The point is to prepare the API spec so that evolution in the anticipated direction is possible without breaking changes.
After publishing the API
After the API has been published, your attitude to change in the API needs to be much more conservative in the interest of your existing API consumers. Change needs to be managed much more rigorously. The basic rule is that the externally observable behavior of an API (from the perspective of the clients) cannot be changed, once the API has been published – and the exception to the rule is: creating a new version of the API.
Managing change – step 0 – is the change really necessary?
First, assess if the proposed change is really necessary. If you can avoid the change: great. One less problem.
Managing change – step 1 – is the change backward compatible?
If the change is really necessary, determine what the change means for the API contract — typically your OpenAPI spec — and for the API consumers relying on the API contract. Ideally, you have an OpenAPI spec for the published API and a new API spec that has all the proposed changes in it. Assess each of the changes and determine whether they are backward compatible or incompatible.
Backward compatible changes:
- Adding query parameters (they should always be optional)
- Adding header or form parameters as long as they are optional
- Adding new fields in JSON or XML data structures as long as they are optional
- Adding endpoints, e.g. a new REST resource
- Adding operations to an existing endpoint, e.g. when using SOAP
- Adding optional fields to the request interfaces
- Changing mandatory fields to optional fields in an existing API
Incompatible changes (a.k.a. breaking changes):
- Removing or changing data structures, i.e. by changing, removing or redefining fields in the data structure
- Removing fields from the request or response (as opposed to making it optional)
- Changing a previously optional request field in the body or parameter into a mandatory field
- Changing a previously required response field in the body or parameter into an optional field
- Changing the URI of the API, such as hostname, port or path
- Changing the structure or relationship between request or response fields, e.g. making an existing field a child of some other field
- Adding a new mandatory field to the data structure
Managing change – step 2 – dealing with backward compatible changes
If you have found no incompatible changes, only backward compatible ones, you should be able to implement them without breaking clients. Apply semantic versioning and give the API release that covers the changes a new minor version. This new minor version will replace the published API already available. Old clients should not be affected by the changes while new clients can take advantage of the new functionality.
There is one general issue with changes even if they are backward compatible. Backward compatibility is based on the API spec, the OpenAPI specification, which serves as the "official contract” with the API consumer. But Hyrums law tells us that, given enough API consumers, there will be an API consumer relying on any observable behavior of your API, not only on the behavior described in the Open API specification.
Managing change – step 3 – dealing with breaking changes
If you have found at least one incompatible change, it will break clients that rely on the changed feature. A new major version of this API is required, and this new version needs to be maintained in parallel with the previous version. Moreover, major versions need to be visible to the consumers, and they need to be accessible by the consumers. If you have followed this guide from the start, you already have a mechanism in place for picking different versions. See section (Prepare an API design pattern for versioning).
(TODO: alternative - remove: incompatible changes need to be implemented in such a way that existing clients are not affected. This can be accomplished by creating a new major version of the API and maintaining the unchanged API alongside the changed API. Creating and publishing a new version of the API is overhead and thus slows down any changes to the API.)
Managing change – step 4 – sunsetting
Operating and maintaining multiple versions of the API requires a lot of effort on the side of the API provider. Sooner or later, old versions of the API need to get turned off. But this should not be a sudden event, rather a gradual one – like a sunset. That’s why we call this phase the API sunset.
You need to set clear expectations for the API sunset and communicate it as early as possible to all affected API consumers – giving them as much time as possible to deal with the sunset and lift their API clients to the new version of the API. You need to know your API consumers and have a way to contact them, which is typically in place when API consumers need to be onboarded. Get API consumers on a mailing list – not for marketing purposes but for product changes. Send out a targeted message on the mailing list to all API consumers when a new version of the API is published, including a timeline for switching off the old API. Motivate your API consumers to switch to the new version in time and explain the benefits of using the new version.
Conclusion
We have seen that dealing with versioning in APIs starts way before the first breaking change is introduced in an API, and even before the API is initially published. A properly designed API will likely be able to live without breaking changes because it is iteratively designed, future changes have been anticipated and a versioning pattern is defined. And if those changes eventually come, you can categorize the changes in backward compatible and breaking change and react accordingly with minor and major versions and eventual sunsetting of API versions.
About the Author Dr. Matthias Biehl
Matthias empowers customers to discover their opportunities for innovation with APIs & ecosystems and turn them into actionable digital strategies. Based on his experience in leading large-scale API initiatives in both business and technology roles, he shares best practices and provides strategic guidance. Matthias works as an API strategist at Software AG, publishes a blog at API-University, is the author of several books on APIs, and regularly speaks at technology conferences.
