Peter Williams

Hierarchical Resources in Rails

Consider a situation where you have a type of resource which always belongs to a resource of another type. How do you model the URI space using Rails? For example, say you have an address resource type. An address is always associated with exactly one user, but a user may have several addresses (work, home, etc).

The simple approach

The simplest approach from a Rails implementation perspective is to just have a flat URI space. In this scenario the URI for the collection of addresses associated with a user and a particular address would be, respectively:

http://example.com/addresses?user_id={user_id}
http://example.com/addresses/{address_id}

From a REST/Web arch standpoint there is absolutely no problem with this URI. It is a bit ugly for the humans around, though. Worse yet, one might reasonably infer from it that http://example.com/addresses references the collection of all the addresses known to the system. While that might be nice from an information modeling point of view, in reality that collection is probably going to be too large to return as a single documents. To be fair, it would be perfectly legal to respond to /addresses with a 404 or 403, but it would be a bit surprising to get that result if you were exploring the system.

The fully hierarchically approach

Edge Rails contains some improvements to the resource oriented route generators. One of the changes adds support for sub-resources. Sub-resources are support via the :has_many and :has_one options to ActiveController::Routing::Map#resources. These options produce fully hierarchically URIs for the resources. For example

http://example.com/users/{user_id}/addresses
http://example.com/users/{user_id}/addresses/{address_id}

The first URI references the collection of all the addresses for the specified user. The second URI references a particular address that belongs to the specified user. These URIs are very pretty, but they add some complexity to the controllers that fulfill them.

The additionally complexity stems from the fact that address_id is unique among all addresses of (in most cases it would be an automatically generated surrogate key). This leads to the potential for the address_id to be valid but that the address it identifies to not belong to the user identified by user_id. In such cases the most responsible thing to do is to return a 404, but doing so takes a couple of extra lines in each of the actions that deal with individual addresses.

The semi-hierarchical approach

After trying both of the previous approaches and finding them not entirely satisfactory. I have started using a hybrid approach. The collection resources are defined below the resources to which they belong but the collection member resources are referenced without an intermediate. For example

http://example.com/user/{user_id}/addresses
http://example.com/addresses/{address_id}

This has the advantage of producing fairly attractive URIs across the board. It also provides an obvious location to add a collection resource containing all the child resources if you have a need for looking at all of them with out the parent resource being involved. And it does not require any extraneous code in the controllers to deal will the possibly of the specified parent and child resources being unrelated.

On the downside, it does requires some changes to the routing system to make defining such routes simple and maintainable. Also, it might be a bit surprising if you are exploring the system. For example, if you request http://example.com/addresses/ you will get a 404, which is probably not what you would expect.

Even with the disadvantages mentioned above I am quite pleased with how the URIs and controllers turn using this technique. If you are looking for a way to deal with hierarchical resources you should give it a try.

JSON Schema Definition Languages

We recently settled on using JSON as the preferred format for the REST-based distributed application on which I am working. We don’t need the expressiveness of XML and JSON is a lot cheaper to generate and parse, particularly in Ruby. Now we are busy defining dialects to encode the data we have, which is happy work. The only problem is there is not a widely accepted schema language for describing JSON documents.

I am not entirely sure a schema language for JSON is necessary in any strict sense. I think that validating documents against a schema is overrated. And, people do seem to be getting along just fine with examples and prose descriptions. Still, the formalist in me is screaming out for a concise way to describe the JSON documents we accept and emit.

I have a weakness for formal grammars. I often write ABNF grammars describing inputs and outputs of my programs, even in situations where most people would just use a couple of examples. I learned XML schema very early in it’s life and I had a love/hate relationship with it for years. Even though it is ugly and complicated I continued to use it because it let me write formal descriptions of the XML variants I created.¹

There are a couple of relatively obscure schema languages for JSON. Unfortunately, I don’t find either of them satisfactory.

Cerny

The Cerny schema validator seem quite functional and while it is not intended as a JSON document schema language, it could be used as one.² Unfortunately, CERNY.schema requires a complete Javascript interpreter and run-time to perform validation. This requirement stems from the fact that CERNY.schema allows parts of the schema to be defined as procedural Javascript code. This approach is completely unacceptable for a language independent data transfer language like JSON.

Such procedural checking is misguided, even beyond the practical problems with requiring a Javascript run-time. Procedural validation code is a powerful technique for validating documents. However, this procedural code greatly reduces the usefulness of schema documents as an informational tool. Schema languages should be designed to communicate the structure of documents to humans, and only incidentally to validator programs.

Kwalify

Another JSON schema language I have tried is Kwalify. Kwalify seems reasonably capable also but it has some warts that really bother me. My main issue with Kwalify is that it is super verbose. This is primarily due to the fact that Kwalify schema documents are written in YAML (or JSON). Schema definitions can be encoded in a generic hierarchical data language, but it is not a very good idea. I find both XSD and the XML variant of RelaxNG to be excessively noise and Kwalify shows a similarly poor signal to noise ratio. Schema language designers should look to RelaxNG’s compact syntax for inspiration and forget trying to encode the schema in the language being described.

Conclusion

I think JSON could benefit from a schema language that is completely declarative and has a compact and readable syntax. If anyone is working on such a thing I would love to know about it. I could roll my own but I would really rather not unless it is absolutely necessary.

REST vs WS-* War

David Chappell declares the REST vs WS-* war over

To anybody who’s paying attention and who’s not a hopeless partisan, the war between REST and WS-* is over. The war ended in a truce rather than crushing victory for one side–it’s Korea, not World War II. The now-obvious truth is that both technologies have value, and both will be used going forward.

In this conflict I am, undeniably, a REST partisan. I know this colors my perceptions, but it is not obvious to me that the war is over. It has become obvious that WS-* will not prevail, but that does not mean it is over. Those who have invested a great deal of time and thought into WS-* may hope that it remains a viable technology, at least for certain problems, but that does not mean it will. I think the situations for which WS-* is the best available technology are vanishingly rare, perhaps even nonexistent. As Mark Baker puts it in his response to Mr Chappell

Perhaps David – or anybody else – could point me towards a data oriented application which can’t fit (well) into such a model (not REST, just the uniform interface part).

I expect that when all is said and done WS-* will still be around. But rather than as a vibrate technology platform, the way Mr Chappell seems to anticipate, I think it survive in a way far more like the way Cobol is still around today: as a zombie, unkillable and ready to eat the brain of anyone who wanders too close the legacy systems.

Mocking

Mr Whitney recently posted an article in which he described mock objects as “bug aggregators”. I once held a similar point of view. Back then my belief was that test doubles (mock, stub, etc) should only be used when real objects would not work, either because they were too hard to setup or because they were too slow. Recently, however, my thinking towards mocks has become a bit more nuanced¹.

A couple of months ago my team switched from Test::Unit to RSpec² for our system validation needs. As part of that switch I read more deeply into the Behavior Driven Development(BDD) movement, which is where tools such as RSpec originate. BDD is, in many ways, just a collection of developer testing best practices combined with much better terminology and some wicked tools (like RSpec). While that may not sound like much it is the biggest improvement in the testing ecosystem since the emergence of systematic automated unit testing.

One of the things that the BDD promotes is the heavy use of test doubles to isolate the tested class from the rest of the system. BDD takes this to such an extreme that Martin Fowler has refers to practitioners of BDD as “mockist”. So when we switched to RSpec I decided to set my previously held notions aside and to give full fledged BDD a try³.

I have been mocking and stubbing my heart out for a couple of months. During that time I have reached the conclusion that test doubles can be very useful as design tools. At the same time they can, as Mr Whitney points out, hide some really nasty bugs.

What are tests for, anyway?

The tension I feel around test doubles comes from the fact that automated unit tests, or executable specification, serve multiple purposes. These purposes are almost never aligned. Improving a test for one purpose will reduce is usefulness for other purposes. As a bare minimum tests serve these purposes:

* tests verify that the code does what you intended * tests provide a set of examples for how to use a class * tests allow for comfortable evolution of the system by protecting against incompatible changes in the future * tests act as a proving ground for the design

While it is the least obvious, experienced practitioners of TDD/BDD often cite that last purpose as one of the most important. Tests have an amazing way of surfacing poor design choices, and test doubles magnify this. You quickly notice interfaces that are excessively complex when you have to mock and stub all the method calls involved. Hard to mock interfaces are a design smell.

Extensive use of test doubles radically enhances the quality of tests as tool for validating the design. At the same time degrades the usefulness of the tests for protecting you from future incompatible changes. I don’t think test doubles aggregate bugs but they are great at hiding them. If you use test doubles extensively you will, quite regularly, end up with two classes that do not work correctly with one another in the real system, but who’s tests pass. On the other hand, those classes will be better designed than if you were just using real objects.

In some languages (Java springs to mind) heavy use of test doubles might also degrade the quality of the tests as examples. If you are using a mocking system that requires a lot of code to mock an object it might tip the against mocking except in extreme cases. However, in Ruby this is not really an issue. The mock/stub library that comes with RSpec has such a clean and pretty syntax that it takes almost nothing away from the readability of the tests.

To counteract the degradation of tests as barrier to the code being inadvertently broken in the future, mockists usually suggest adding higher level of tests that are more about testing the system as a whole rather than the individual parts, or acceptance testing. These sorts of test are usually a good idea in their own right, but they also free you up to make more aggressive use of test doubles to improve the design of your code.

It’s all about tradeoffs

As with most engineering decisions, how much to use test doubles boils down to a set of trade-offs. You have to decide which of the uses of tests are most important for your project and then try to optimize along those dimensions. My currently feeling is that the design benefits of using test doubles usually outweighs the costs. Only you and your team can decide if that is true for your project, but you will never know until you give them a real try.

Hypermedia as the Engine of Application State

One of the least well understood core tenets of the REST architectural style is that “hypermedia is the engine of application state”. Which basically means that responses from the server will be documents that include URIs to everything you can do next. For example, if GET a blog post the response document will have URIs embedded in it that allow you to create a comment, edit the post and any other action that you might want to do. Basically, this allows you to think of your application as a state machine with every page representing a state and links representing every possible transition from the current state.

This approach means that to correctly be access your application the only things a client needs to know is a) a well know starting point URI and b) how to parse one of the document formats (representations) your application supports. For human facing web applications this approach is the one that is always used. Browsers understand how to parse HTML and extract the links to the next action and the user provides starting point URIs. This has become so ingrained in the industry’s culture that most people never really even think about it in these explicit terms.

However, I am now working with a system in which there are many independent automated processes which interact with each other. It was not immediately obvious to me, or my colleagues, that we should be following the same pattern even when there are no humans involved. After all, a program can easily remember how to construct a URI from a couple of pieces of information that it knows. In fact, URI construction is almost always easier to implement than the REST approach of requesting a well known resource, parsing the returned representation and extract the URI you want.¹

After pondering this for a while I did reach the, rather unsurprising, conclusion that Roy Fielding is correct and that hypermedia should be the core of our distributed application. My epiphany came when I decided that I really wanted to change the shape of a URI. I realized that if the system were only slightly bigger (and it will get there soon) there would be the strong probability that I would not know all the places that accessed the resources whose URIs I wanted to change. Therefore, I would not be able to change the shape of those URIs.

A URI that is constructed by a client constitutes a permanent, potentially huge, commitment by the server. Any resource that may be addressed by the constructed URIs must forever live on that particular server (or set of servers) and the URI patterns must be supported forever.² Effectively, you are trading a small one time development cost on the client side for an ongoing, and ever increasing, maintenance cost on the server side. When it is stated like that it becomes obvious that URI construction introduces an almost absurd level of coupling between the client and server.

With a truly REST based architecture you are free to change just about anything about the disposition and naming of resources in the system, except for a few well known start point URIs. You can change the URI shapes (for example, if you decide that you really hate the currently scheme). You can relocate resources to different servers (for example, if you need to partition you data). Best of all, you can do those things without asking any ones permission, because clients will not even notice the difference.

Using hypermedia as the engine of application state has the effect of preserving the reversible for a huge number of decisions that are irreversible in most other architectural styles. As the section 9 of “The Pragmatic Programmer” points out, reversibility is one of the goals of any design should strive for. No one knows for sure what will be needed in the future, so having the ability to easily change your mind is invaluable. Particularly when it can be had so cheaply.