Principles

The goal is to sub-divide the application into multiple highly cohesive components which are weakly connascence with each other. To achieve the desired cohesion it will be necessary to align the component boundaries with natural fissure points in the application.

The strategy should allow for the production of a arbitrary number of components. A component that was of a manageable size yesterday could easily become too large tomorrow. In that situation the over-sized component will need to be sub-divided. Applying the same strategy repeated will result in a system that is more easily understood.

We want to minimize redundancy in the components. Redundancy results in more code with must be understood and maintained. More importantly redundancy usually introduces connascence of algorithm, making changes more error prone and expensive. In a perfect world, any particular behavior would be implemented in exactly one component.

We want to isolate changes to the system. When implementing a new feature it is desirable to change as few components as possible. Each additional component that must be changed raise the complexity of the change. The componentization strategy should minimize the number of components involved in the average change to the system.

With those metrics in mind lets explore the two most common approaches and see how they compare with each other. Those two patterns of componentization are horizontal slicing and vertical slicing.

Horizontal slicing

In this approach the component boundaries are derived from that implementation domain. The implementation is divided into a set of stacked layers in such a way that a layer initiates communication with the layers below it. This results in a standard layered architectures. By implementing each layer in a separate component you can achieve the horizontal slicing. This style of componentization strategy results in the very common n-tier architecture pattern.

For example, an application that has a business logic and a presentation layer the application would be divided into two components. A business logic component and a presentation component.

Vertical slicing

In this approach the component boundaries are derived from the application domain. Related domain concepts are grouped together into components. Individual components communicate with any other components as needed.

This approach is also quite common but is usually thought of a lot less formally. It is more common for this type of segmentation to develop incidentally. For example, because separate teams developed the parts independently, and then integrated them later. Any time you integrate separate applications you have vertical componentization.

The Score

Against the metrics we laid out earlier, vertical slicing does much better than horizontal.

Cohesion

Horizontal slicing has high cohesion. Each of the components can represent the a logically cohesive part of the implementation.

Vertical slicing also has high cohesion. Each component represents highly cohesive part of the application domain.

Repeatability

Vertical slicing provides a mechanism for reapply the subdivision pattern an arbitrary number of times. If any component gets too large to manage it can be divided into multiple components based on the application domain concepts. This same process can be repeated from the initial division of a monolithic application until components of the desired size have been achieved.

Horizontal slicing is less repeatable. The more tiers the harder it is to maintain cohesiveness. In practice it is very rare to see an tiered architecture with more than 4 tiers, and 3 tiers is much more common.

DRYness

Horizontal slicing tends to result in some repetition. Certain behaviors will have to be repeated a each layer. For example, data validation rules. You will need those in the presentation layer to provide good error messages and in the business logic layer to prevent bad data being persisted.

Vertical slicing allows you to reduce the connascence of algorithm because any single user activity is implemented in exactly one component. Components usually do end up communicating to each other, however, they do so in a way that does not require in the same algorithms be implemented in multiple components. For any one bit of data or behavior, one component will its authoritative source.

Change isolation

Vertical scaling tends to allow new features to be implemented by changing only one component. The component changed is the one which already contains features cohesive with the new one.

Horizontal slicing, on the other hand, tends to require changes in every layer. The new feature will require additions to the presentation layer, the business logic layer and the persistence layer. Having to work in every layer increase the cognitive load required to achieve the desired result.

Conclusion

Vertical slicing provides significant advantages. The high cohesion, dryness, and change isolation combine to drastically reduces the risks and cost of change. That is turn allow better/faster maintenance and evolution of the system. The repeatability allows you to retain these benefits even while adding functionality over time. Each time a component gets too large you can divide it until you have reach a application size that is human scaled.

Having a large number of components operate as a system does result in a good deal of communication between the components. It important to pay attention to the design of the APIs. Poor API design can introduce excessive coupling which will eat up most of the advantages described above. Hypermedia – or more precisely, following the REST architectural style – is the best way i know to reduce coupling between the components.

I have leaned in both directions at various times. I have never fully convinced myself either.

To make the issues a bit clearer let me use and example lifted from the article that got me thinking about this most recently.¹

<cart>
  <!-- some stuff here -->
  <link rel="http://ex.org/rel/abort"
        href="http://ex.org/cart/cancel;token=987654321"/>
  <link rel="http://ex.org/rel/add-more"
        href="http://ex.org/cart/add;token=987654321"/>
  <link rel="http://ex.org/rel/buy"
        href="http://ex.org/cart/buy;token=987654321"/>
</cart>

I place this example in the “links for every action” camp. Each of the links in the example describes exactly one action.

An alternate approach might look something like this.

<cart>
  <!-- some stuff here -->
  <link rel="http://ex.org/rel/line-items"
        href="http://ex.org/cart/line-items;token=987654321"/>
  <link rel="http://ex.org/rel/new-order"
        href="http://ex.org/orders?cart=987654321"/>
</cart>

From a client perceptive these are a bit different.

Abandoning cart

A client that wants to abandon a cart in the first example would make a DELETE or POST – it’s a bit hard to tell which from the example – request to the href of the http://ex.org/rel/abort link. In the second example a similar client would just DELETE the cart resource.

Adding item

When adding an item in the first example the client would post a www-form-urlencoded document containing the URI of the item to add and the quantity to the href of the http://ex.org/rel/add-more link. In the second example, the same document gets posted to href of the http://ex.org/rel/line-items link.

Placing order

In the first example the client would make a POST request to the href of the http://ex.org/rel/buy link. In the second example the client would POST a www-form-urlencoded document containing the cart URI and some payment information to the href of the http://ex.org/rel/order link.

Differences

Obviously the to approaches result in quite similar markup. The same behavior is encoded in both. In the first example the links are action oriented. All actions that can be taken on an item are explicitly stated using a link. In the second approach the links are data oriented rather than action oriented. Rather than having separate links to retrieve the current line items and to add a new line item the http://ex.org/rel/items link provide both actions using the GET and POST HTTP methods respectively.

The first approach it better at expressing what actions are allowable at any given point in time. For example, once the purchase process has been initiated it does not make sense to abort a cart. So if you GET a cart after POSTing to the http://ex.org/rel/buy link the representation would not have the http://ex.org/rel/abort link.

The second is more concise because it, at least potentially, provides access to more than one action per link based on the standard HTTP methods. You don’t need to provide a separate abort link because DELETEing the cart is sufficient. You don’t need to provide separate get line items and add line item links because a single link that can handle GET and POST requests will work.

The first approach is a bit more flexible with regard to implementation details. If you need for some reason to have different URIs for the retrieve line item request than the add line item request you could easily achieve it. The second example makes that impossible.

Conclusion

I am still not entirely convinced but i am leaning toward the more flexible, verbose and explicit approach of a link for every actions.² Having links represent actions rather than resources feels a bit odd, but i think it provides more of the benefits we hope to get from a RESTful architecture.

I really dislike this idea. It sets my teeth on edge because it treats links – which are possibly the most important bits of data in existence – as second class citizens. It is easiest to show what i mean with a bit of extrapolation:

<complexElement rel="entry">
  <string rel="id">234132</string>
  <string rel="displayName">Peter Williams</string>
  <complexElement rel="name">
    <string rel="familyName">Williams</string>
    <string rel="givenName">Peter</string>
  </complexElement>
  <complexElement rel="emails">
    <link rel="email" href="mailto:pezra@barleyenough.org"/>
    <string rel="type">personal</string>
  </complexElement>
</complexElement>

That is what a portable contact might look like if we treated all data the way link elements work. That example looks pretty ugly to me, as i suspect it does to most people. It is ugly because very important information regarding the role of elements is relegated to a subsidiarity position in favor of fairly unimportant information about its type. However, link elements do to links exactly what my example does to all the data. The effect is that properties whose values happen to be independently addressable resources are obfuscated.

The revealed preference of the world is against link elements. Just look at pretty much any format that embeds application specific semantics. As far as i know, there is not a single widely used format that actually represents its links as link elements. Even atom uses properly named elements for most its links. The link element it defines is largely relegated to the back water of extensibility.

One benefit that link elements have, or at least could have if they where more widely used, is the facilitation of standard link processing tools. Fortunately, we do not have to give up the expressiveness and clarity of intention revealing names to achieve this result. Rather than obscuring the links we could just treat them as normal data. The additional information needed to support standard tools could be added in a relatively unobtrusive way. Consider the following:

<entry xmlns:link="http://unobtrusive-generic-linking.org/">
  ...
  <emails>
    <value link:hrefDisposition="elementContent"
           link:rel="foo">
      mailto:pezra@barelyenough.org</value>
    <type>personal</type>
  <emails>
</entry>

This is idea is similar to xLink but more flexible and simpler to use.

You could expand the idea to JSON with relative ease. Consider the following expansion of the portable contacts json format tagged with some unobtrusive link info.

{"entry": [
  {"id": "42",
  "emails":
    [{"address"   : "mailto:pezra@barelyenough.org",
      "type"      : "personal",
      "_linkInfo" : {"hrefDisposition" : "address",
                     "rel" : "foo"}}]}]}

Unobtrusive link info makes links visible to and usable by generic link processing tools while protecting the use of intention revealing names that format designers, and users, want. This is important because it allows new formats to reuse “standard” link semantics more easily and uniformly.

The idea of an application as a set of states (read: representations) with transitions (read: links) to other states seems to go against the idea of storing links. Transitions from one application state to another are surely transient. Any change in the application state, either by this client or some completely unrelated client, could easily invalidate those transitions. In that context a client that stored links for later use would surely be doomed dereferencing dead links for the rest of it’s days.

Further, the idea that clients might store links is a frightening specter for maintainers of services. If clients store links, and you prefer not to break those clients, you must continue supporting any links you have ever included in any representation in perpetuity. Talk about limiting your design freedom. Such a strict requirement would surely raise the cost of maintaining the service over time.

Reality sets in

Those are scary thoughts. Some of these issues are even real. But end the end it doesn’t matter. Almost all non-trivial systems are going to require that URIs be stored in places other than the origin server. Sometimes these stored URIs will merely be caches. Other times they will be data that cannot be recalculated mechanically.

For example, say you have an order taking system and an inventory system. When placing an order the user goes to the web site, searches for “coffee”, selects the third item in the results and places an order 1 of that item. An order is a set of line items each of which references a product. Once payment is received the order system is going to need to be able to tell the shipping department which items from inventory to send to the customer.

The inventory system has, of course, a URI for every type of product that is for sale. So the simplest and most effective way for an order to reference a product is to use that the inventory URI for that product. URIs are called universal resource identifiers for a reason, we might as well use them as such.

In this example, we have a situation where the product references in the order are not merely caches of URIs. Many things may change the ordering of search results – a new product being added, an old one being discontinued, even a small change to a description of some product. So at any moment the the third item in the search results for “coffee” might be different. Once the user has made their selection no automata can reliably retrace those steps.

The implications of this are significant. The inventory must continue to support the product URIs used in orders until such time as the order system would never care to dereference those URIs again. If a month from now the user comes back and wants to see their order history, those product URIs had better still work.

Fortunately, HTTP provides us with a ready solution. Behold the awesomeness that is HTTP redirection. HTTP redirection is your best friend when it comes to gracefully changing REST/HTTP applications. Clients get what they need – URIs continue to work as identifiers indefinitely – and servers get what they need – a lot of freedom to change the names and dispositions of resources.

Transient links

We are still faced with this issue of the transient nature of links. Certainly, many links encode transitions which may be transient. The client has no general way of distinguishing between links which represent transiently available state transitions, and those that represent more permanent transitions.

In our example, immediately after creating an order, it probably provides some links to pay for the order. After the user has provided payment those transition would no longer be valid. However, the link to the inventory product is a more permanent part of the order resource.

The only tractable way i see to deal with this issue is to document the lifespan the various link found in a representation. Once the client implementer understand the semantics of the links they well often be able to infer the likely lifespan of the links without further input. However, guidance can be provided in situations where precision is required or the lifespan is ambiguous. A transient link is, by definition, an option part of the representation so documenting the conditions that cause it to be present is likely to be required anyway.

Best practices: Server

REST/HTTP application developers should assume that clients will store links and dereference them after indeterminate periods of time. When resources are relocated or renamed requests to the resource’s obsolete URI should be redirected to the canonical URI using a 301 Moved
Permanently response.

For links whose validity has a bounded lifespan the documentation of the representations (the media type) should explicitly layout that the link is transient and optional. If possible the documentation should also describe the conditions of the links existence.

Remind client developers early and often that client must follow any and all redirects from the server.

Best practices: Client

Clients should follow redirects. Fastidiously.

Clients should update it’s internal storage upon receiving a 301
Moved Permanently response by replace the URI it requested with newly provided location.

Client developers should be aware of transient links in the representations being dealt with. Either do not store these URIs or ensure that attempts so use these URIs handle failure in ways that make sense for the application.

Believe and follow the redirections the server sends to you. Seriously.

This idea is at the very core of the architecture of the web. However, achieving such compartmentalization has been difficult for business application development. The recent advancement of REST into the mainstream is bringing this mentality within the reach of many development teams.

The small pieces approach totally dominates the lower levels of software development in the form of object oriented programing. Each “object” is a small self-contained piece, and a large number of the small pieces are joined together to provide the functionality of the application.

At the application level, however, this approach does not enjoy the same ubiquity. It is much more common to see a monolithic approach. There is one giant application that does everything for everyone. The path of least resistance for any single new feature is to implement it in the existing application structure. There is some overhead in creating a new application, so in the short term multiple small applications seems more costly.

Unfortunately, while it is easier to add any particular feature to an existing application doing so means you give up all the advantages of small pieces loosely joined. And, the advantages of a small application approach are significant.

Perhaps the single largest advantage is that smaller applications are easier to understand. To effectively improve or maintain software it is necessary to understand it. The connascence of one bit of code and everything else is often unclear. This lack of clarity means that the risk of any particular change having unintentional impacts increases with the size of the code base. Building more smaller applications is an effective way to manage such risks.

Another advantage of this approach is that it makes using novel implementation techniques more workable. If you have a large monolithic application and you get a new requirement that might benefit from a different language, framework or runtime you are pretty much out of luck. In a compartmentalized architecture each application can have it’s own technology stack. If you new a set of features that that might benefit from the concurrency of Erlang you can use for that component without impacting the other components in any way.

Have several small applications often turns intractably large efforts in to several smaller tasks. For example, consider upgrading the framework on which your application(s) are built to a new version. The new version has features and improvements that would be highly beneficial but target version is incompatible with the version you are currently running in some minor ways.

Such an upgrade will necessarily touch most of the application. Its risk profile will be very broad. The benefits of the upgrade will rarely be directly visible to the business so the priority of such work is always rather low. The cost and risk of such work is often so large, and the perceived benefit so small, that such work is put off until support is being terminated for the version currently in use.

In a compartmentalized system the any single component can be upgraded much more quickly, and at lower risk. The benefits of upgrading match the effort required much more closely in such situation. Many times the effort required is often so low there is no discussion required, upgrades become normal refactoring tasks.

Experimentation is also quite a bit more manageable in a compartmentalized architecture. The smaller size of the components makes implementing new ideas faster and more approachable. This lowers the cost of implementing new ideas and recovering if the idea does not pan out.

When experimenting it is often not immediately clear if a new idea really is an improvement. Sometimes developers and users need to work with it a for a while to form a reasoned opinion. In compartmentalized systems experiments can be designed to impact a small portion of the total application. This allows small experiments to can soak for a while until the team is ready to call the results. If idea worked the practice can be expanded to the rest of the components, if not it is only a small portion of the code base that needs to be cleaned up.

It is worth noting that this approach will effectively true your large application into a distributed system of small applications. This is distributed applications are a little scary, and for good reason. Before you embark on this path you should have a plan for how to integrate the parts into a whole. For most business application REST/HTTP is a very good technology for integrating applications.

There are many other situations where the small pieces approach’s conversion of large tasks into small ones is an advantage. There are also situations where it causes more overall work. In my experience, though, the chunking of tasks is well worth the small additional overhead. It is much easier to manage many small semi-independent development efforts than a few large ones.

Resourceful is a sophisticated HTTP client library for Ruby. It will (when it is complete, at least) provide an simple API for fully utilizing the amazing goodness that is HTTP.

It is already tasty, though. The 0.2 release provides

• fully compliant HTTP caching
• a framework for implementing cache managers (memory based cache manager provided)
• fully compliant transparent redirection handling (with hooks for overriding the default behavior)
• plugable HTTP authentication handling (Basic provided)

Introduction

The API is strongly influenced by our successful experiences with REST. Each URI is represented by a Resource object. The Resource objects act as a proxy for the conceptual resource. Resources expose the basic set of HTTP verbs: get, put, post, delete. For example to get a representation of some resource you do this

require 'resourceful'
http = Resourceful::HttpAccessor.new
resp = http.resource('http://rubyforge.org').get
puts resp.body

If you want to post a form you do this

require 'resourceful'
http = Resourceful::HttpAccessor.new
resp = http.resource('http://rubyforge.org').post("name=Peter&hobbies=programming,diy", :content_type => "application/x-www-form-urlencoded")
puts resp.code

All non-2xx responses are either handled transparently (e.g. by following redirects) or the method will raise a UnsuccessfulHttpRequestError.

Conclusion

If you need a decent HTTP library in Ruby come on over and check it out. If you see something you want, or want fixed, feel free to branch the git repo and do it. Paul and I would love some more contributors and will welcome you with open arms.

Handling obsolescent service providers

Mr. Dubray asks how we deal with version skew between the client and server.

Backwards compatibility is when the consumer comes in with a “newer” request version than the service provider can provide. This is common when a consumer uses different providers for the same type of service. So ultimately, you need to provide some room to define the version of both the consumer and the version of the service provider that it is targeting. Your mechanism only supports “one version”.

Not true, the versioning mechanism I describe easily handles multiple versions. First, lets be clear, a service provider cannot provide capabilities that where not conceived of until after it was written. So Mr. Dubray must be interested in is the ability of a single consumer to successfully communicate with multiple versions of the service provider. I agree with him that this is an absolutely vital feature of any versioning mechanism.

Fortunately, content negotiation deals with this issue quite handily. I left this out of the original post for simplicities sake but it well worth talking about. HTTP allows user agents – or service consumers, if you prefer – to specify more than one acceptable response format. For example, the following is a perfectly legal HTTP conversation.

===>
GET /accounts/42
Accept: application/vnd.myapp-v2+xml, application/vnd.myapp-v1+xml;q=0.8

<===
200 OK
Content-Type: application/vnd.myapp-v1+xml

<account>
  <name>Inigo Montoya</name>
</account>

The Accept header field in the request indicates that the consumer can operate using either version 1 or 2 of the API but it prefers version 2. Accept headers can include any number of MIME media types along with preference indicators (the q=number part). This allows consumers to inform the server of all acceptable dialects of the API with which it can work. In the example, the server obviously did not support version 2 of the API and therefore responded using version 1.

Resource deprecation

Further along Mr. Dubray asks this question,

Another flaw of your versioning strategy is that URIs are by default part of the versioning strategy. I have often pointed out that “Query-by-examples” are encoded by members of the REST community (MORCs) in a URI syntax, for instance:

/customer/{id}/PurchaseOrders ...

Peter, how do you express that a particular QBE belongs to one version and not to the other?

I don’t. The set of purchase orders associated with a particular customer is not version specific. The customer has agreed to purchase the same things regardless of which version of the service you are talking to.

Perhaps the question Mr. Dubray is really trying to ask is, what happens if you want to deprecate such resource?

(One reason to do so might be that the purchase order collections become too big to reasonably render in a single response. There are other, better ways to solve that particular problem but it is a nice concrete use case for resource deprecation.)

Resource deprecations is easily handled in REST using media types to handle versioning. First some ground rules, user agents should never be constructing such a URI. Doing so should be a gross violation of the HATEOAS constraint of REST. Rather they would be extracting that URI from the representation of the customer provided by the server. In such a case, an HTTP conversation getting the purchase orders for a customer might look like this.

===>
GET /customer/42
Accept: application/vnd.myapp-v1+xml
<===
200 OK
Content-Type: application/vnd.myapp-v1+xml

<customer>
  <purchase-orders href="http://service.example/customer/42/purchase-orders"/>
</customer>

===>
GET /customer/42/purchase-orders
Accept: application/vnd.myapp-v1+xml
<===
200 OK
Content-Type: application/vnd.myapp-v1+xml

<purchase-orders>
  ...
</purchase-orders>

At version 2 of the API we deprecate the all-purchase-orders-for-customer resource – removing all references to it in the customer representations – and replace it with a purchases-order-by-month-by-customer resource. A similar HTTP conversation with a client capable of handling version 2 of the API would look like this.

===>
GET /customer/42
Accept: application/vnd.myapp-v2+xml
<===
200 OK
Content-Type: application/vnd.myapp-v2+xml

<customer>
  <purchase-orders-by-month href-template="http://service.example/customer/42/purchase-orders?in_month={xmlschema-gYearMonth}"/>
</customer>

===>
GET /customer/42/purchase-orders?in_month=2008-05
Accept: application/vnd.myapp-v2+xml
<===
200 OK
Content-Type: application/vnd.myapp-v2+xml

<purchase-orders>
  ...
</purchase-orders>

Notice that in version 2 of the API the all-purchase-orders-for-customer resource is no longer exposed in any way. As a human you might guess that it still exists, and indeed it would need to in order to handle requests to version 1 of the API. However, a version 2 consumer will never make a request to that resource because it is not mentioned in the version 2 representations. Indeed, any requests for the all-purchase-orders-for-customer by a version 2 consumer would be met with a 406 Not Acceptable response because it is not part of the version 2 API.

Wrap up

Toward the end Mr. Dubray gets into full rant mode with these bits,

You will soon start realizing that resources do have a state that is independent of the “application” since by definition a resource can participate in multiple “applications”. This is the essence of SOA, i.e. the essence of reuse.

Resources certainly may participate in multiple “applications”. There is nothing in the REST principles that prevent that. I don’t really claim to be an SOA expert. I just make systems work using REST principles. So far I have not found a problem reusing my resources in multiple applications. In fact, REST seems to excel at that very thing.

At least, some of the MORCs Member Of the REST Community could have the courtesy to acknowledge that they are indeed building a programming model on top of REST, that this programming model needs clear semantics and that these semantics are not intrinsically part of REST (nor always RESTful).

I, for one, will readily acknowledge that we have built, and are continuing to build, programming models on top of REST. REST is merely a set of principles, articulated as constraints, that facilitate the creation of useful network based architectures. I would be very surprised if many in the REST community would disagree with me. These programming models do, for the most part, adhere to the REST principles.

Building REST/HTTP web services is certainly not fully understood yet. That does not make it special, hardly any sort of system design or architecture is fully understood. However, REST seems, to me at least, to be a better fit for today’s applications and technologies than any of the alternatives.

Related Posts

If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

Nonstandard MIME media types

Using content negotiation to manage versions requires, by definition, the introduction of nonstandard media types. There is really no way around this. I personally don’t feel this is a Bad Thing. The new, nonstandard, media types do a much better job describing the sort of media the client is requesting. It does, however, mean that browsers – and perhaps some HTTP tools – will work less well with the web service.

The browser not working is a pretty big issue. They are almost certainly not the target consumer of the services, but having the browser not work raises the level of effort for exploring the API. If you have created a cool new service you want as few barriers to entry as possible. Personally, I always use curl when I am exploring but I know several people who would prefer to use a browser.

Unfortunately, I don’t really have a great general solution for browsers. That being said, in many situations a much can be done to make life better. For example, if the resources in question do not have HTML representations you could serve the current preferred format with a generic content type that browsers can render – e.g. text/plain or application/xml – to browsers.

Curl

One advantage of having the version token directly in the URL is that it makes it really easy to use curl against the service. By default curl makes requests with the Accept header field set to */*. For a reasonably designed service this would result in a response in the current preferred format. If you want to change to Accept header you need to invoke curl like this

curl --header 'Accept: application/vnd.foo.myformat-v1+xml' http://api.example/hello-world

That is not too horrible, really. It is a bit much to type all the time, but I have curl rc files for all the formats I deal with on a daily basis. If your service is implemented in Rails there is an even easier way. With Rails you give each format you support a short name that may be used as an “extension” for URLs. For example, if we define the short name for application/vnd.foo.myformat-v1+xml to be mf1 we can say this

curl http://api.example/hello-world.mf1

That is equivalent, from the point of view of a Rails based service, to the previous example. I imagine similar functionality could be implemented in most web frameworks. This effectively puts you back to having the version embedded in the URL, which is convenient for debugging and exploration. (It is still unsuitable for production use, though, for all the same reasons as other approaches to embedding the version in the URL.)

Nonobviousness

Another potential downside of using content negotiated versioning is that the various versions my be less discoverable, compared to a version-in-the-URL approach. I am not entirely sure this is true – after all there is a version token in the media type – but if it is true it would be a Good Thing.

Do you really want people “discovering” a version of the API that was deprecated a year ago? I think it might be better, in either approach, to use version tokens that are not readily guessable. Obviously, previous versions of and API will be documented and remain accessible, but raising some barriers to entry on depreciated parts of a system seems appropriate to me.

Unfamiliarity

This may be the biggest issue of all. People are just not very familiar, and therefore comfortable, with content negotiation. This in spite of the fact that it has been a fundamental part of HTTP since forever. I think this features obscurity is waning now, though, because it is such a powerful feature.

Two years ago Rails got content negotiation support. (That link seems to be broken at the moment. You can see part of the post I am talking about by going here and searching for “The Accept header”.) As frameworks like Rails keep adding and improving their support for this powerful feature the community of developers will become more familiar and comfortable with it. What is needed now is more education in the community on how best to utilize this feature.

Related Posts

If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

Let’s assume you have a REST/HTTP web service that has some account resources. Say you can make a request like this:

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp+xml

<account>
  <name>Inigo Montoya</name>
</account>

First, you probably noticed that my example uses a vendor MIME media type to describe the representation. Using a more generic MIME media type like application/xml is much more common, at least in my experience. Using generic media types is perfectly legal but a bit silly. You are not really asking for any old XML document, but rather an XML document that has a quite specific schema. Aside from my idealistic rantings, using a specific media type has some strong practical benefits which are at the core of this post.

Backwards compatible changes

Often changes will need to be made to expose new behavior of the system that do not negatively impact correctly implemented clients. Say, for example, you want to start tracking email address for accounts. If the application/vnd.mycompany.myapp+xml format documentation is clear that elements that are not recognized should be ignored you can simply add a email element to the account representation.

<account>
  <name>Inigo Montoya</name>
  <email-address>mailto:prepare-to-die@youkilledmyfather.example</email-address>
</account>

Any client that was created before the addition of the email element will simply ignore it’s presence. Problem solved.

Incompatible changes

Unfortunately, not all changes can be implemented in a way that is backwards compatible. For example, a couple of months after adding email to accounts the sales team sign a deal for 1 bazillion dollars. But the new customer demands that each account be allowed to have more than one email address. After thinking for a while, you decide that the best way to expose that is by changing the account representation as follows.

<account>
  <name>Inigo Montoya</name>
  <email-addresses>
    <email-address priority='1'>mailto:prepare-to-die@youkilledmyfather.example</email-address>
    <email-address priority='2'>mailto:vengeance@youkilledmyfather.example</email-address>
  <email-address>
</account>

That, of course, will break any clients that are expecting the old format – so pretty much all of them. This is a place where we can bring content negotiation to bare. You can simply define a new media type – say application/vnd.mycompany.myapp-v2+xml – and associate new multi-email format with it. Clients can then request whichever format they want. Older clients don’t know the new media type so they get served the older single email format.

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp+xml

<account>
  <name>Inigo Montoya</name>
  <email-address>mailto:prepare-to-die@youkilledmyfather.example</email-address>
</account>

Newer clients do know the new media type so they can have access to the new functionality.

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp-v2+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp-v2+xml

<account>
  <name>Inigo Montoya</name>
  <email-addresses>
    <email-address priority='1'>mailto:prepare-to-die@youkilledmyfather.example</email-address>
    <email-address priority='2'>mailto:vengeance@youkilledmyfather.example</email-address>
  <email-address>
</account>

Everyone gets what they need. Easy as pie.

Alternate approaches

The most commonly proposed approach for versioning REST/HTTP web service interfaces today seems to be to mutilate the URIs by inserting a version. For example,

http://foo.example/api/v1/accounts/3

I really hate this approach as it implies that an account in one version of the API is really a different resource than the account in a different version of the API.

It also forces clients into a nasty choice, either support multiple versions of the API simultaneously or break one of the core constrains of REST. For example, say a client exists for the v1 API that saves references (URIs that include the version indicator) to accounts. Some time later the client is updated to support the new version of the API. In this situation the The client can support both versions of the API simultaneously because all the previously stored URIs point at the old version of the API or it has to mung all the URIs it has stored to the point at the new API. Munging all the URIS breaks the HATEOAS constraint of REST and supporting multiple versions of the API is a maintenance nightmare.

Conclusion

Changes to REST/HTTP web service interfaces come it three basic flavors, changes to the properties associated with a type of resource, additions of new types of resources and deprecation of obsolete types of resources. If you are following the HATEOAS constraint of REST the approach described here can be used to safely handle all three scenarios.

This approach does lead to media types being created, but media types are cheap so we can – and should – have as many as we need. Used properly, content negotiation can be used to solve the problems related to versioning a REST/HTTP web service interface.

Related Posts

If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

never once — not even once — have I seen anyone develop a consuming application without relying on some form of human-oriented documentation for the service being consumed

When you start writing an application that makes use of some services you are not writing some sort of generic web services consumer. You are writing a consumer of one very specific web service and the semantics of a service, as with everything else, turn out to be a lot more complicated, subtle and interesting than the syntax.

Human-oriented documentation necessary because only human can understand the really interesting parts of a service description. Based on my experience, it also seems to be sufficient. Sure we could all jump on the full fledged service description language band wagon but I don’t think that service consumers would get much, if any, value out of it.¹

Discoverability

Discoverability is the most important capability that interface definition languages bring to the table. However, most service description languages provide discoverability almost as a side effect, rather than it being their primary purpose.

I think it would be better to promote discoverability by working on a more focused capabilities publishing mechanism. To that end, I want to describe what my team has done on this front. It is not entirely suitable general use, but useful standards often emerge from extracting the common parts of many bespoke solutions.

First I want to be clear about the terminology I am using just to make sure we all understand one another

service

A cohesive set of resources that are exposed via HTTP.

resource

An single entity which exposes a RESTful interface via HTTP.

service provider

A process or set of processes that implement one or more services.

container

Another name for a service provider.

Background

We needed the ability to discover the actual URIs of resources at runtime from very early in our project because of our basic architecture. Our system is composed of at least four services². The containers that provide these services may be deploy in various (and arbitrary) ways. Maintaining the list of top level resources of other services in configuration files became unmanageable long before we every actually deployed the system in production.

We need a way that any component in the system could discover the URIs of resources exposed by other components in the system. We handled this by providing a system wide registry of all the services that are available and a description resource for each service that provides link to the resources contained with that service.

Service Description

Containers that provide a service are responsible for exposing a “service description” for that service.

A service description is a resource that provides links to all the top level resources in service. Currently we support just one type of representation (format) for service description, a JSON format that looks like this

{
  "_type":        "ServiceDescriptor",
  "service_type": "http://mydomain.example/services/something-interesting",
  "resources": [
    {
      "_type": "ResourceDescriptor",
      "name":  "OpenIdProvider",
      "href":  "http://core.ssbe.example/openid"
    },
    {
      "_type": "ResourceDescriptor",
      "name":  "AllAccounts",
      "href":  "http://core.ssbe.example/accounts"
    }
  ]
}

service_type is the globally unique name for the type service that is being described. It should be a URI that is owned by the creator of the service. Each top level resource that is exposed as part of this service has a resource descriptor in the resources set.

If you wanted know about all the accounts of the system you would

1. GET the service descriptor resource 2. iterate over the resources collection until you found the AllAccounts resource descriptor 3. GET the URI found in the href pair of the resource descriptor (http://core.ssbe.example/accounts in this example)

One important thing to note is that each resource is really exactly one resource, and not a type of resource. If you are looking for a particular account you have to get the AllAccounts collection and find the account you are looking for in that set.

Capabilities

The Capabilities resource is the only well known entry point for our system. If a program wants to interact with our system it always starts with the capabilities service and the works it’s way down, using the links in documents, to the resource it actually cares about.

The JSON representation we support looks like

{
  "_type": "SystemCapabilities",
  "services": [
    {
      "_type":        "ServiceDescriptor",
      "href":         "http://alarm.ssbe.example/service_descriptors/escalations",
      "service_type": "http://mydomain.example/services/something-interesting"
    }
  ]
}

To discover the URI of a particular top level resource a consumer must

1. GET the capabilities document 2. iterate though the objects in services until if it finds the one of the correct service_type 3. GET the full service descriptor using the URI in the href pair 4. iterate of the resources until it finds the one with the correct name 5. extract the URI from it’s href pair

Services are registered with the capabilities resource, by POSTing a service description to it, when the containers that provide those services are started.

Issues

No supported methods or format information

This approach only provides a way to discover the URIs of top level resources. It makes no attempt to describe the representations (formats) or methods those resources support. That sort of thing would not be hard add but so far I have had absolutely not need for it. That information is provided by the human-oriented documentation and since it does not change in each deployment there is no need for it included in the dynamic resource discovery mechanism.

Non-top level resources are not represented

Resources that are not top level – by which I mean resources that not listed in a service description document – are not represented at all. This is a feature, really, but it makes extending this format to include method and data format information less compelling becayse only a relatively minor subset of the resources in the system are surfaced in the service descriptions.

Encourages large representations

The fact that only singleton resources are supported can lead to top level documents that are excessively large. In fact, we have already had to deal with this issue. We have basically punted on the issue but I think the correct approach would be to introduce a ResourceTypeDescriptor that would operate much like a ResourceDescriptor except that the link would be a URI template rather than a concrete URI.