Small pieces

The idea of creating a functioning application by loosely connecting many small pieces has been around for a long time. Certainly since early in the development of Unix, and probably even before. It has survived because it is such a powerful approach.

This idea is at the very core of the architecture of the web. However, achieving such compartmentalization has been difficult for business application development. The recent advancement of REST into the mainstream is bringing this mentality within the reach of many development teams.

The small pieces approach totally dominates the lower levels of software development in the form of object oriented programing. Each “object” is a small self-contained piece, and a large number of the small pieces are joined together to provide the functionality of the application.

At the application level, however, this approach does not enjoy the same ubiquity. It is much more common to see a monolithic approach. There is one giant application that does everything for everyone. The path of least resistance for any single new feature is to implement it in the existing application structure. There is some overhead in creating a new application, so in the short term multiple small applications seems more costly.

Unfortunately, while it is easier to add any particular feature to an existing application doing so means you give up all the advantages of small pieces loosely joined. And, the advantages of a small application approach are significant.

Perhaps the single largest advantage is that smaller applications are easier to understand. To effectively improve or maintain software it is necessary to understand it. The connascence of one bit of code and everything else is often unclear. This lack of clarity means that the risk of any particular change having unintentional impacts increases with the size of the code base. Building more smaller applications is an effective way to manage such risks.

Another advantage of this approach is that it makes using novel implementation techniques more workable. If you have a large monolithic application and you get a new requirement that might benefit from a different language, framework or runtime you are pretty much out of luck. In a compartmentalized architecture each application can have it’s own technology stack. If you new a set of features that that might benefit from the concurrency of Erlang you can use for that component without impacting the other components in any way.

Have several small applications often turns intractably large efforts in to several smaller tasks. For example, consider upgrading the framework on which your application(s) are built to a new version. The new version has features and improvements that would be highly beneficial but target version is incompatible with the version you are currently running in some minor ways.

Such an upgrade will necessarily touch most of the application. Its risk profile will be very broad. The benefits of the upgrade will rarely be directly visible to the business so the priority of such work is always rather low. The cost and risk of such work is often so large, and the perceived benefit so small, that such work is put off until support is being terminated for the version currently in use.

In a compartmentalized system the any single component can be upgraded much more quickly, and at lower risk. The benefits of upgrading match the effort required much more closely in such situation. Many times the effort required is often so low there is no discussion required, upgrades become normal refactoring tasks.

Experimentation is also quite a bit more manageable in a compartmentalized architecture. The smaller size of the components makes implementing new ideas faster and more approachable. This lowers the cost of implementing new ideas and recovering if the idea does not pan out.

When experimenting it is often not immediately clear if a new idea really is an improvement. Sometimes developers and users need to work with it a for a while to form a reasoned opinion. In compartmentalized systems experiments can be designed to impact a small portion of the total application. This allows small experiments to can soak for a while until the team is ready to call the results. If idea worked the practice can be expanded to the rest of the components, if not it is only a small portion of the code base that needs to be cleaned up.

It is worth noting that this approach will effectively true your large application into a distributed system of small applications. This is distributed applications are a little scary, and for good reason. Before you embark on this path you should have a plan for how to integrate the parts into a whole. For most business application REST/HTTP is a very good technology for integrating applications.

There are many other situations where the small pieces approach’s conversion of large tasks into small ones is an advantage. There are also situations where it causes more overall work. In my experience, though, the chunking of tasks is well worth the small additional overhead. It is much easier to manage many small semi-independent development efforts than a few large ones.

Announcing Resourceful

Resourceful has its initial (0.2) release today.

Resourceful is a sophisticated HTTP client library for Ruby. It will (when it is complete, at least) provide an simple API for fully utilizing the amazing goodness that is HTTP.

It is already tasty, though. The 0.2 release provides

  • fully compliant HTTP caching
  • a framework for implementing cache managers (memory based cache manager provided)
  • fully compliant transparent redirection handling (with hooks for overriding the default behavior)
  • plugable HTTP authentication handling (Basic provided)

Introduction

The API is strongly influenced by our successful experiences with REST. Each URI is represented by a Resource object. The Resource objects act as a proxy for the conceptual resource. Resources expose the basic set of HTTP verbs: get, put, post, delete. For example to get a representation of some resource you do this

require 'resourceful'
http = Resourceful::HttpAccessor.new
resp = http.resource('http://rubyforge.org').get
puts resp.body

If you want to post a form you do this

require 'resourceful'
http = Resourceful::HttpAccessor.new
resp = http.resource('http://rubyforge.org').post("name=Peter&hobbies=programming,diy", :content_type => "application/x-www-form-urlencoded")
puts resp.code

All non-2xx responses are either handled transparently (e.g. by following redirects) or the method will raise a UnsuccessfulHttpRequestError.

Conclusion

If you need a decent HTTP library in Ruby come on over and check it out. If you see something you want, or want fixed, feel free to branch the git repo and do it. Paul and I would love some more contributors and will welcome you with open arms.

REST/HTTP Service Versioning (Response to Jean-Jacques Dubray)

Jean-Jacques Dubray takes issue with my approach of using content negotiation to manage service versioning in HTTP. I actually hesitate to respond to Mr. Dubray because the overall tone of his piece is rather off putting. On the other hand, he raises a couple of interesting questions which I have been really looking for and excuse to talk about. So I will give it a go.

Handling obsolescent service providers

Mr. Dubray asks how we deal with version skew between the client and server.

Backwards compatibility is when the consumer comes in with a “newer” request version than the service provider can provide. This is common when a consumer uses different providers for the same type of service. So ultimately, you need to provide some room to define the version of both the consumer and the version of the service provider that it is targeting. Your mechanism only supports “one version”.

Not true, the versioning mechanism I describe easily handles multiple versions. First, lets be clear, a service provider cannot provide capabilities that where not conceived of until after it was written. So Mr. Dubray must be interested in is the ability of a single consumer to successfully communicate with multiple versions of the service provider. I agree with him that this is an absolutely vital feature of any versioning mechanism.

Fortunately, content negotiation deals with this issue quite handily. I left this out of the original post for simplicities sake but it well worth talking about. HTTP allows user agents – or service consumers, if you prefer – to specify more than one acceptable response format. For example, the following is a perfectly legal HTTP conversation.

===>
GET /accounts/42
Accept: application/vnd.myapp-v2+xml, application/vnd.myapp-v1+xml;q=0.8

<===
200 OK
Content-Type: application/vnd.myapp-v1+xml

<account>
  <name>Inigo Montoya</name>
</account>

The Accept header field in the request indicates that the consumer can operate using either version 1 or 2 of the API but it prefers version 2. Accept headers can include any number of MIME media types along with preference indicators (the q=number part). This allows consumers to inform the server of all acceptable dialects of the API with which it can work. In the example, the server obviously did not support version 2 of the API and therefore responded using version 1.

Resource deprecation

Further along Mr. Dubray asks this question,

Another flaw of your versioning strategy is that URIs are by default part of the versioning strategy. I have often pointed out that “Query-by-examples” are encoded by members of the REST community (MORCs) in a URI syntax, for instance:

/customer/{id}/PurchaseOrders ...

Peter, how do you express that a particular QBE belongs to one version and not to the other?

I don’t. The set of purchase orders associated with a particular customer is not version specific. The customer has agreed to purchase the same things regardless of which version of the service you are talking to.

Perhaps the question Mr. Dubray is really trying to ask is, what happens if you want to deprecate such resource?

(One reason to do so might be that the purchase order collections become too big to reasonably render in a single response. There are other, better ways to solve that particular problem but it is a nice concrete use case for resource deprecation.)

Resource deprecations is easily handled in REST using media types to handle versioning. First some ground rules, user agents should never be constructing such a URI. Doing so should be a gross violation of the HATEOAS constraint of REST. Rather they would be extracting that URI from the representation of the customer provided by the server. In such a case, an HTTP conversation getting the purchase orders for a customer might look like this.

===>
GET /customer/42
Accept: application/vnd.myapp-v1+xml
<===
200 OK
Content-Type: application/vnd.myapp-v1+xml

<customer>
  <purchase-orders href="http://service.example/customer/42/purchase-orders"/>
</customer>


===>
GET /customer/42/purchase-orders
Accept: application/vnd.myapp-v1+xml
<===
200 OK    
Content-Type: application/vnd.myapp-v1+xml

<purchase-orders>
  ...
</purchase-orders>

At version 2 of the API we deprecate the all-purchase-orders-for-customer resource – removing all references to it in the customer representations – and replace it with a purchases-order-by-month-by-customer resource. A similar HTTP conversation with a client capable of handling version 2 of the API would look like this.

===>
GET /customer/42
Accept: application/vnd.myapp-v2+xml
<===
200 OK
Content-Type: application/vnd.myapp-v2+xml

<customer>
  <purchase-orders-by-month href-template="http://service.example/customer/42/purchase-orders?in_month={xmlschema-gYearMonth}"/>
</customer>


===>
GET /customer/42/purchase-orders?in_month=2008-05
Accept: application/vnd.myapp-v2+xml
<===
200 OK    
Content-Type: application/vnd.myapp-v2+xml

<purchase-orders>
  ...
</purchase-orders>

Notice that in version 2 of the API the all-purchase-orders-for-customer resource is no longer exposed in any way. As a human you might guess that it still exists, and indeed it would need to in order to handle requests to version 1 of the API. However, a version 2 consumer will never make a request to that resource because it is not mentioned in the version 2 representations. Indeed, any requests for the all-purchase-orders-for-customer by a version 2 consumer would be met with a 406 Not Acceptable response because it is not part of the version 2 API.

Wrap up

Toward the end Mr. Dubray gets into full rant mode with these bits,

You will soon start realizing that resources do have a state that is independent of the “application” since by definition a resource can participate in multiple “applications”. This is the essence of SOA, i.e. the essence of reuse.

Resources certainly may participate in multiple “applications”. There is nothing in the REST principles that prevent that. I don’t really claim to be an SOA expert. I just make systems work using REST principles. So far I have not found a problem reusing my resources in multiple applications. In fact, REST seems to excel at that very thing.

At least, some of the MORCs Member Of the REST Community could have the courtesy to acknowledge that they are indeed building a programming model on top of REST, that this programming model needs clear semantics and that these semantics are not intrinsically part of REST (nor always RESTful).

I, for one, will readily acknowledge that we have built, and are continuing to build, programming models on top of REST. REST is merely a set of principles, articulated as constraints, that facilitate the creation of useful network based architectures. I would be very surprised if many in the REST community would disagree with me. These programming models do, for the most part, adhere to the REST principles.

Building REST/HTTP web services is certainly not fully understood yet. That does not make it special, hardly any sort of system design or architecture is fully understood. However, REST seems, to me at least, to be a better fit for today’s applications and technologies than any of the alternatives.


If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

Versioning REST Web Services (Tricks and Tips)

In my previous post on this subject I described an approach to versioning the API of a REST/HTTP web service. This approach has significant advantages over the approach that is currently most common (i.e. embedding a version token in the URL). However, it does have some downsides. This post is an attempt to outline those and to present some ways to mitigate the negative impacts.

Nonstandard MIME media types

Using content negotiation to manage versions requires, by definition, the introduction of nonstandard media types. There is really no way around this. I personally don’t feel this is a Bad Thing. The new, nonstandard, media types do a much better job describing the sort of media the client is requesting. It does, however, mean that browsers – and perhaps some HTTP tools – will work less well with the web service.

The browser not working is a pretty big issue. They are almost certainly not the target consumer of the services, but having the browser not work raises the level of effort for exploring the API. If you have created a cool new service you want as few barriers to entry as possible. Personally, I always use curl when I am exploring but I know several people who would prefer to use a browser.

Unfortunately, I don’t really have a great general solution for browsers. That being said, in many situations a much can be done to make life better. For example, if the resources in question do not have HTML representations you could serve the current preferred format with a generic content type that browsers can render – e.g. text/plain or application/xml – to browsers.

Curl

One advantage of having the version token directly in the URL is that it makes it really easy to use curl against the service. By default curl makes requests with the Accept header field set to */*. For a reasonably designed service this would result in a response in the current preferred format. If you want to change to Accept header you need to invoke curl like this

curl --header 'Accept: application/vnd.foo.myformat-v1+xml' http://api.example/hello-world

That is not too horrible, really. It is a bit much to type all the time, but I have curl rc files for all the formats I deal with on a daily basis. If your service is implemented in Rails there is an even easier way. With Rails you give each format you support a short name that may be used as an “extension” for URLs. For example, if we define the short name for application/vnd.foo.myformat-v1+xml to be mf1 we can say this

curl http://api.example/hello-world.mf1

That is equivalent, from the point of view of a Rails based service, to the previous example. I imagine similar functionality could be implemented in most web frameworks. This effectively puts you back to having the version embedded in the URL, which is convenient for debugging and exploration. (It is still unsuitable for production use, though, for all the same reasons as other approaches to embedding the version in the URL.)

Nonobviousness

Another potential downside of using content negotiated versioning is that the various versions my be less discoverable, compared to a version-in-the-URL approach. I am not entirely sure this is true – after all there is a version token in the media type – but if it is true it would be a Good Thing.

Do you really want people “discovering” a version of the API that was deprecated a year ago? I think it might be better, in either approach, to use version tokens that are not readily guessable. Obviously, previous versions of and API will be documented and remain accessible, but raising some barriers to entry on depreciated parts of a system seems appropriate to me.

Unfamiliarity

This may be the biggest issue of all. People are just not very familiar, and therefore comfortable, with content negotiation. This in spite of the fact that it has been a fundamental part of HTTP since forever. I think this features obscurity is waning now, though, because it is such a powerful feature.

Two years ago Rails got content negotiation support. (That link seems to be broken at the moment. You can see part of the post I am talking about by going here and searching for “The Accept header”.) As frameworks like Rails keep adding and improving their support for this powerful feature the community of developers will become more familiar and comfortable with it. What is needed now is more education in the community on how best to utilize this feature.


If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

Versioning REST Web Services

Managing changes to APIs is hard. That is no surprise to anyone who has ever maintained an API of any sort. Web services, being a special case of API, are susceptible to many of the difficulties around versioning as other types of APIs. For HTTP based REST style web services the combination of resources and content negotiation can be used to mitigate most of the issues surrounding API versioning.

Let’s assume you have a REST/HTTP web service that has some account resources. Say you can make a request like this:

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp+xml

<account>
  <name>Inigo Montoya</name>
</account>

First, you probably noticed that my example uses a vendor MIME media type to describe the representation. Using a more generic MIME media type like application/xml is much more common, at least in my experience. Using generic media types is perfectly legal but a bit silly. You are not really asking for any old XML document, but rather an XML document that has a quite specific schema. Aside from my idealistic rantings, using a specific media type has some strong practical benefits which are at the core of this post.

Backwards compatible changes

Often changes will need to be made to expose new behavior of the system that do not negatively impact correctly implemented clients. Say, for example, you want to start tracking email address for accounts. If the application/vnd.mycompany.myapp+xml format documentation is clear that elements that are not recognized should be ignored you can simply add a email element to the account representation.

<account>
  <name>Inigo Montoya</name>
  <email-address>mailto:prepare-to-die@youkilledmyfather.example</email-address>
</account>

Any client that was created before the addition of the email element will simply ignore it’s presence. Problem solved.

Incompatible changes

Unfortunately, not all changes can be implemented in a way that is backwards compatible. For example, a couple of months after adding email to accounts the sales team sign a deal for 1 bazillion dollars. But the new customer demands that each account be allowed to have more than one email address. After thinking for a while, you decide that the best way to expose that is by changing the account representation as follows.

<account>
  <name>Inigo Montoya</name>
  <email-addresses>
    <email-address priority='1'>mailto:prepare-to-die@youkilledmyfather.example</email-address>
    <email-address priority='2'>mailto:vengeance@youkilledmyfather.example</email-address>
  <email-address>
</account>

That, of course, will break any clients that are expecting the old format – so pretty much all of them. This is a place where we can bring content negotiation to bear. You can simply define a new media type – say application/vnd.mycompany.myapp-v2+xml – and associate new multi-email format with it. Clients can then request whichever format they want. Older clients don’t know the new media type so they get served the older single email format.

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp+xml

<account>
  <name>Inigo Montoya</name>
  <email-address>mailto:prepare-to-die@youkilledmyfather.example</email-address>
</account>

Newer clients do know the new media type so they can have access to the new functionality.

===>
GET /accounts/3 HTTP/1.1
Accept: application/vnd.mycompany.myapp-v2+xml
<===
HTTP/1.1 200 OK
Content-Type: application/vnd.mycompany.myapp-v2+xml

<account>
  <name>Inigo Montoya</name>
  <email-addresses>
    <email-address priority='1'>mailto:prepare-to-die@youkilledmyfather.example</email-address>
    <email-address priority='2'>mailto:vengeance@youkilledmyfather.example</email-address>
  <email-address>
</account>

Everyone gets what they need. Easy as pie.

Alternate approaches

The most commonly proposed approach for versioning REST/HTTP web service interfaces today seems to be to mutilate the URIs by inserting a version. For example,

http://foo.example/api/v1/accounts/3

I really hate this approach as it implies that an account in one version of the API is really a different resource than the account in a different version of the API.

It also forces clients into a nasty choice, either support multiple versions of the API simultaneously or break one of the core constrains of REST. For example, say a client exists for the v1 API that saves references (URIs that include the version indicator) to accounts. Some time later the client is updated to support the new version of the API. In this situation the The client can support both versions of the API simultaneously because all the previously stored URIs point at the old version of the API or it has to mung all the URIs it has stored to the point at the new API. Munging all the URIS breaks the HATEOAS constraint of REST and supporting multiple versions of the API is a maintenance nightmare.

Conclusion

Changes to REST/HTTP web service interfaces come it three basic flavors, changes to the properties associated with a type of resource, additions of new types of resources and deprecation of obsolete types of resources. If you are following the HATEOAS constraint of REST the approach described here can be used to safely handle all three scenarios.

This approach does lead to media types being created, but media types are cheap so we can – and should – have as many as we need. Used properly, content negotiation can be used to solve the problems related to versioning a REST/HTTP web service interface.


If you’re interested in REST/HTTP service versioning be sure not to miss the rest of the series.

RESTful Service Discovery and Description

There has been a great deal of discussion regarding RESTful web service description languages this week. The debate is great for the community but I think Steve Vinoski has it basically right

never once — not even once — have I seen anyone develop a consuming application without relying on some form of human-oriented documentation for the service being consumed

When you start writing an application that makes use of some services you are not writing some sort of generic web services consumer. You are writing a consumer of one very specific web service and the semantics of a service, as with everything else, turn out to be a lot more complicated, subtle and interesting than the syntax.

Human-oriented documentation necessary because only human can understand the really interesting parts of a service description. Based on my experience, it also seems to be sufficient. Sure we could all jump on the full fledged service description language band wagon but I don’t think that service consumers would get much, if any, value out of it.1

Discoverability

Discoverability is the most important capability that interface definition languages bring to the table. However, most service description languages provide discoverability almost as a side effect, rather than it being their primary purpose.

I think it would be better to promote discoverability by working on a more focused capabilities publishing mechanism. To that end, I want to describe what my team has done on this front. It is not entirely suitable general use, but useful standards often emerge from extracting the common parts of many bespoke solutions.

First I want to be clear about the terminology I am using just to make sure we all understand one another

service
A cohesive set of resources that are exposed via HTTP.
resource
An single entity which exposes a RESTful interface via HTTP.
service provider
A process or set of processes that implement one or more services.
container
Another name for a service provider.

Background

We needed the ability to discover the actual URIs of resources at runtime from very early in our project because of our basic architecture. Our system is composed of at least four services2. The containers that provide these services may be deploy in various (and arbitrary) ways. Maintaining the list of top level resources of other services in configuration files became unmanageable long before we every actually deployed the system in production.

We need a way that any component in the system could discover the URIs of resources exposed by other components in the system. We handled this by providing a system wide registry of all the services that are available and a description resource for each service that provides link to the resources contained with that service.

Service Description

Containers that provide a service are responsible for exposing a “service description” for that service.

A service description is a resource that provides links to all the top level resources in service. Currently we support just one type of representation (format) for service description, a JSON format that looks like this

{
  "_type":        "ServiceDescriptor",
  "service_type": "http://mydomain.example/services/something-interesting",
  "resources": [
    {
      "_type": "ResourceDescriptor",
      "name":  "OpenIdProvider",
      "href":  "http://core.ssbe.example/openid"
    },
    {
      "_type": "ResourceDescriptor",
      "name":  "AllAccounts",
      "href":  "http://core.ssbe.example/accounts"
    }
  ]
}

service_type is the globally unique name for the type service that is being described. It should be a URI that is owned by the creator of the service. Each top level resource that is exposed as part of this service has a resource descriptor in the resources set.

If you wanted know about all the accounts of the system you would

1. GET the service descriptor resource 2. iterate over the resources collection until you found the AllAccounts resource descriptor 3. GET the URI found in the href pair of the resource descriptor (http://core.ssbe.example/accounts in this example)

One important thing to note is that each resource is really exactly one resource, and not a type of resource. If you are looking for a particular account you have to get the AllAccounts collection and find the account you are looking for in that set.

Capabilities

The Capabilities resource is the only well known entry point for our system. If a program wants to interact with our system it always starts with the capabilities service and the works it’s way down, using the links in documents, to the resource it actually cares about.

The JSON representation we support looks like

{  
  "_type": "SystemCapabilities",
  "services": [
    {
      "_type":        "ServiceDescriptor",
      "href":         "http://alarm.ssbe.example/service_descriptors/escalations",
      "service_type": "http://mydomain.example/services/something-interesting"
    }
  ]
}

To discover the URI of a particular top level resource a consumer must

1. GET the capabilities document 2. iterate though the objects in services until if it finds the one of the correct service_type 3. GET the full service descriptor using the URI in the href pair 4. iterate of the resources until it finds the one with the correct name 5. extract the URI from it’s href pair

Services are registered with the capabilities resource, by POSTing a service description to it, when the containers that provide those services are started.

Issues

No supported methods or format information

This approach only provides a way to discover the URIs of top level resources. It makes no attempt to describe the representations (formats) or methods those resources support. That sort of thing would not be hard add but so far I have had absolutely not need for it. That information is provided by the human-oriented documentation and since it does not change in each deployment there is no need for it included in the dynamic resource discovery mechanism.

Non-top level resources are not represented

Resources that are not top level – by which I mean resources that not listed in a service description document – are not represented at all. This is a feature, really, but it makes extending this format to include method and data format information less compelling becayse only a relatively minor subset of the resources in the system are surfaced in the service descriptions.

Encourages large representations

The fact that only singleton resources are supported can lead to top level documents that are excessively large. In fact, we have already had to deal with this issue. We have basically punted on the issue but I think the correct approach would be to introduce a ResourceTypeDescriptor that would operate much like a ResourceDescriptor except that the link would be a URI template rather than a concrete URI.


  1. On the other hand service providers might get some value. Something like WADL does give you a way to declaratively define a suite of regression tests. On the other hand, you might be better off using a tool specifically built for that purpose.

  2. That is the base number of services. Additionally functionality is added to the system in the form of additional services so the actually number of services varies based on what you need the system to do.

How REST Can Relieve Your (Lack of) Documentation Guilt

A couple of months ago we hired a contractor to write a reporting interface for our high volume monitoring system. Our system exposes all of it’s data in RESTful web services, and his job has been to take that data and allow users to create reports based on it.

This morning a couple of my teammates and I asked him if he thought our documentation was sufficient to allow supplementary applications, like the one he was finishing up, to be written without having direct access to the developers. He replied to this effect,

To be honest, I did not really look at the documentation. I just fetched the URLs you gave me, and the ones I found in those documents, and so on. It did not take me very long to get a pretty good idea of what kind of data was available and where.

That ability to understand a large system by simply exploring it is one of the most powerful features RESTful architectures. But only if you are using all the precepts of REST, including that resources are represented by documents with links1 (or, hypermedia is the engine of application state, if you prefer a more traditional phrasing).

A RESTful architecture will let you scale, and distribute your application beyond all reasonable expectations. But even better, since you know that anyone who cares can just go exploring, it will also let you feel less guilty about not writing all that documentation that you never quite get around to.


  1. Hat tip to Stefan Tilkov for either reporting that Sanjiva Weerawarana used this phrase in is QCon 07 presentation, or for coining that phrase himself (I cannot tell for sure which it was).

HTTP Authentication with shared identities

Authentication has been bane of my existence lately. By which I mean, it is complicated and interesting and I am loving every minute of it (but, as you can see, I am not going to let that stop me from complaining about it). However tonight I have run into an authentication problem that I am not sure how to solve. I am hoping someone out there can point me toward a solution.

So, Dear Lazyweb, here is my question: is there a mechanism available that allows an HTTP client to have a single identity for several applications and to be able to authenticate itself to each of those applications in such a way that even a malicious application would be unable to impersonate the actor to the other applications in the system?

Oh yeah, and it would be really nice if this were already implemented for libcurl and in Ruby.)

Some background

I have a system composed of a set applications which communicate with one another using RESTful web services. This system supports the addition of arbitrary new applications to system. However,some of these applications maybe written by (relatively) untrusted parties.

All actors, both end user and components of the system, have a single system wide identity. This identity is managed by the one trusted component in the system. This component is responsible for, amongst other things, authentication of actors.

We settled on OpenID as the mechanism for end user authentication. Other than having one of the worst specs I have ever read OpenID is really nice. OpenID solves this problem by forwarding the user’s browser to the identity provider (our trusted component) and the identity provider verifies the user’s identity claim. The application that requested the authentications is then notified of success of failure of the authentication process. This approach has the advantage that the user’s password, even in an encrypted form, never passes though the untrusted components of the system.

Unfortunately, end user authentication is only a subset of the authentication required for this system. There are many automated actors that also make have use of the resources exposed by components in the system. These actors need to the authenticated also, but OpenID is rather unsatisfactory for this purpose. So another solution to delegated authentication is required.

My initial thought was to use MD5-sess based HTTP Digest auth. The spec explicitly mention that it could be used to implement authentication using a third party identity provider. Upon further study, however it only works if the application doing the authentication is trusted. This is because to verify the requester’s identity the application must have the hash of the users account, realm and password. With that bit of information it would quite easy for the application to impersonate the original requester. In my environment of limit trust that is unacceptable.

Another potential, if naive, option is to use HTTP digest auth but to pass the authentication credentials though to the identity provider. The identity provider could then response with an indication of whether the requester proved that they new the password. Unfortunately, the additional load placed on the identity provider by having to verify the requester’s identity for every single request handled by any part o the system is just too great. Not to mention the additional lag this would impose on response times.

Now, the astute reader will by now be fairly yelling something about how this problem was solve by Kerberos years ago. Not only is this true but theoretically, the negotiate HTTP auth scheme supports Kerberos based authenitication. However, I have yet to find any Ruby libraries that support that scheme. Tomorrow, I will probably dive into the RFC to determine if I can implement support myself. If you know of a library that implements this scheme please let me know.

I have also looked at OpenID HTTP authentication. It looks a bit simpler than the Kerberos based negotiate auth scheme, but it seems a bit under cooked for a production system. On the other hand, it does have potential. If there are no other options it might be workable. It would be pretty easy to implement on the Ruby side of the house, particularly since I have spent the last couple of days coming to terms with OpenID, but on the C++ side it might be a bit more of a problem.

Anyway, it is late now so I am going to go to sleep and await your answers.

The power of hypermedia remains non-obvious

Patrick Mueller contemplates whether or not we really need URIs in our documents1. This is a pretty common question in my experience. This question comes up because it is not always immediately obvious just how powerful embedding links in documents is.

What Mr. Mueller suggests is that if you have a client that needs account information for a particular person that is could simply take the account numbers found in the person representations and based on some out of band information, construct the account URIs. For example, if you got a person representation that looked like

<person>
  <accounts>
    <account><id>3242</id></account>
    <account><id>5523</id></account>
  </accounts>
</person>

The client could then make get requests to http://bank.example/accounts/3242 and http://bank.example/accounts/5523 to get the persons account information. The client would have constructed those URIs based on some configuration or compile time information about the structure of account URIs. This is a very common approach. Hell, it is even the one use by the ActiveResource library in Rails. But common does make it good.

Magically creating URIs out of the ether would work at first but say this bank we work for buys another bank. There are some people that have accounts at both banks. Now, if a persons accounts where referenced by URI, rather than just number, you could just add them to the list like this:

<person>
  <accounts>
    <account href="http://bank.example/accounts/3242"/>
    <account href="http://bank.example/accounts/5523"/>
    <account href="http://other-bank.example/accounts/9823"/>
  </accounts>
</person>

The fact that some accounts are served by the original system and some are served by the other banks system is unimportant. However, if the client is constructing URI based on out of band information this approach fails completely. This is just one example of the sort of problems that disappear when you reference resources by URI, rather than some disembodied id.

One of the potential advantages of using just ids, rather than a URI is that it will require less work on the server to generate the document. I suppose ids are less costly, in a strict sense, if the server generating the document is also serves the account resources. But how much faster? Building a URI like the ones above could be as cheap as a single string concatenation. As far I am concerned, that is not really enough work to spend any time worrying about. On the other hand, if the server generating the document does not also serve the account resources, then the accounts should be being referenced by URI internally anyway so using the URI should be cheaper (not to mention safer).

Mr Mueller suggests, as a proof point, that Google Maps must work by URI construction based on a priori knowledge of the shape of tile URIs. It may well, for all I know, but it certainly would not have to. For example, the server could pass the client a tile URI template and the client could then calculate the x and y offsets of the required tiles based on the x and y values of the tiles it already has. Or each tile could include links to the tiles that touch it (which would allow arbitrary partitioning of the tiles which would be nice). No doubt there are other reasonable RESTful choices too.

The more I work with REST based architectures the more enamored of hypermedia. Links make your representations brightly lit, well connected spaces and that will benefit you application in ways you probably have not even imagined yet.


  1. BTW, Mr Mueller, I was unable to post a comment on your blog. After pressing Post I was brought back to the comments page with the message Comment authentication failed! after the Comments text area.

Hierarchical Resources in Rails

Consider a situation where you have a type of resource which always belongs to a resource of another type. How do you model the URI space using Rails? For example, say you have an address resource type. An address is always associated with exactly one user, but a user may have several addresses (work, home, etc).

The simple approach

The simplest approach from a Rails implementation perspective is to just have a flat URI space. In this scenario the URI for the collection of addresses associated with a user and a particular address would be, respectively:

http://example.com/addresses?user_id={user_id}
http://example.com/addresses/{address_id}

From a REST/Web arch standpoint there is absolutely no problem with this URI. It is a bit ugly for the humans around, though. Worse yet, one might reasonably infer from it that http://example.com/addresses references the collection of all the addresses known to the system. While that might be nice from an information modeling point of view, in reality that collection is probably going to be too large to return as a single documents. To be fair, it would be perfectly legal to respond to /addresses with a 404 or 403, but it would be a bit surprising to get that result if you were exploring the system.

The fully hierarchically approach

Edge Rails contains some improvements to the resource oriented route generators. One of the changes adds support for sub-resources. Sub-resources are support via the :has_many and :has_one options to ActiveController::Routing::Map#resources. These options produce fully hierarchically URIs for the resources. For example

http://example.com/users/{user_id}/addresses
http://example.com/users/{user_id}/addresses/{address_id}

The first URI references the collection of all the addresses for the specified user. The second URI references a particular address that belongs to the specified user. These URIs are very pretty, but they add some complexity to the controllers that fulfill them.

The additionally complexity stems from the fact that address_id is unique among all addresses of (in most cases it would be an automatically generated surrogate key). This leads to the potential for the address_id to be valid but that the address it identifies to not belong to the user identified by user_id. In such cases the most responsible thing to do is to return a 404, but doing so takes a couple of extra lines in each of the actions that deal with individual addresses.

The semi-hierarchical approach

After trying both of the previous approaches and finding them not entirely satisfactory. I have started using a hybrid approach. The collection resources are defined below the resources to which they belong but the collection member resources are referenced without an intermediate. For example

http://example.com/user/{user_id}/addresses
http://example.com/addresses/{address_id}

This has the advantage of producing fairly attractive URIs across the board. It also provides an obvious location to add a collection resource containing all the child resources if you have a need for looking at all of them with out the parent resource being involved. And it does not require any extraneous code in the controllers to deal will the possibly of the specified parent and child resources being unrelated.

On the downside, it does requires some changes to the routing system to make defining such routes simple and maintainable. Also, it might be a bit surprising if you are exploring the system. For example, if you request http://example.com/addresses/ you will get a 404, which is probably not what you would expect.

Even with the disadvantages mentioned above I am quite pleased with how the URIs and controllers turn using this technique. If you are looking for a way to deal with hierarchical resources you should give it a try.