Peter Williams

Why REST

I am in the process of redesigning my blog. The default WordPress theme is nice and all but Charlie and Catherine have shamed me into doing something less mundane.

One of the things that had bothered me was my blog roll. It was out-of-date because WordPress’s default way of handling links make it pretty painful to import a list of links. Actually the first time is easy and great but every time after that you just want to shoot yourself. The other problem was that the list was long and it messed up the way the page looked.

So I decided to trim the list in the side bar to just people that I know. But declarative living requires that I keep my blogroll public. So I decided to create a blogroll page (you can find the results here).

Being lazy like I am, I refuse to create this page by hand. I noticed that FeedLounge can export my subscriptions as OPML.Actually, I noticed this before I switched to FeedLounge. The ability to export my subscriptions is an absolute requirement of any aggregator I use. I would not have switched to FeedLounge if it did not support this. Initially I though I would write a little ruby script that would pull down my OPML document from FeedLounge and the use ruby-xslt to transform the OPML into and HTML document. This was a nice idea because I could have scheduled it to run every night so that blogroll would always up-to-date.

Sadly, FeedLounge completely ignores web architecture. For example, the URI used to export a OPML is http://my.feedlounge.com/export/opml which does not point at my subscriptions but rather to some bit of code that can generate a OPML file. This bit of code figures out who’s subscriptions to export based on, I suppose, a cookie, or something. The details do not matter because it is not something that I can reasonably duplicate in my schedulable ruby script. So instead I have to manually download the OPML file, then transform it and then push it to the server. That process does not take very long but is really annoying that I have to do it. One demerit for FeedLounge for forcing me into a manual process that could have been easily avoided.

So why use REST? Because I want to use your application in ways you have not anticipated or haven’t had the time to implement yet.

RTD Strike — Day 2

This is day two of the RTD strike. That means that today is my second day of commuting to downtown Denver by car. Commuting by car is so much less civilized that riding the bus.

Catherine and I have rearranged our schedules so that I can leave the house earlier than I usually would. That seems to let me find parking for about $8 per day. That is expensive but a lot less than the $20 I paid yesterday.

Anyway, if there is anyone reading this that commutes into downtown along the US-36 corridor let me know and maybe we can carpool. The HOV lanes definitely move faster.

Catherine Blogs

My beautiful wife Catherine has started blogging. So far it is mostly about our children. For example, in Parenthood, part XXVII

“It goes without saying – but, don’t let Elliot lick his underwear.”

But in addition to being a nearly perfect mother Catherine is also an extremely talented database architect so you might see a few technical tidbits from her occasionally.

Tonic — A RESTful Web Application Framework

TonicVia [Sam Buchanan](http://afongen.com/blog/) is a very promising RESTful web application framework for PHP. If you are doing web development in PHP I definitely suggest you take a closer look.

I am always interested in how RESTful behaviors are implemented, regardless of the language and environment used so I went and had a little peek at the docs. I noticed a couple of details that I think are interesting and might be relevant to the RESTful Rails work going on now.

Controller–Resource Type Relationship

Tonic seems to define a controller class (though they call them something different) per resource type. This is different than any of the rails approaches I have seen, which combine the code to support multiple resource types into a single controller class. It is not clear to me, from the documentation I have read so far anyway, exactly how collection resources with addressable members are handled. For example, if http://store.example/products/ is a collection of all known products and http://store.example/products/magik-eight-ball is a particular product how does Tonic handle route those requests to the appropriate resource controllers?

Resource Modifier Resources

Tonic uses the concept of editor and delete resources to manage the modification and deletion of resources. This is quite similar to Charlie’s approach. This is at least one significant difference. It appears–but I am not sure about this so correct me if I am wrong–that you post to the editor to update the resource. From an REST standpoint that is fairly unclean but it would make handling bad form data issues Charlie talked about a lot easier. And if you also supported PUT on the target resource you app would still be RESTful.

Structural Data in Rails

On a recent project I ran into a situation where I needed some structural data. I was writing a conference registration application. Each track at the conference costs a different amount and attendees can sign up for more than one track. We already have an accounting infrastructure that has a concept of a “product”, which is just something for which we charge money. So to support registering attendees for tracks, taking their money and letting the bean counters know that we have taken their money I needed to add some new products representing the registration for each track. However these products are far from mere cruft need by the accounting infrastructure, the conference registration code is completely dependent on the existence of these products. There are a set of check boxes on the registration page that the controller maps into the appropriate products from the products table. That means that the conference registration code will simply not work with out those products.

These products are the sort of data I am referring too when I say structural data. In my mind any data whose absence would cause a failure or that is managed as part of the development process, rather than in the application runtime, is structural data. This sort of data (dare I say “pattern”?) occurs fairly frequently, in my experience, and can be used to great effect

Given the fact that the code is tightly coupled to structural data it makes sense to manage structural data in the same way you manage the database schema. If you are using Rails odds are you are managing your schema with migration( and if you aren’t you should be). Migration is a great way to manage a rapidly changing database schema, and it easily supports creation and modification of structural data. Using migrations in this way has several benefits. It keeps versions of the structural data associated with the versions of the code with which they were intended to work (just check out the project from Subversion and you ready to go). It ensures that those products get inserted when the conference registration code gets deployed (migrating the database is part of the deployment process). And finally it places those vital database records under revision control.

The Problem

This technique works very well with the exception of testing. Unfortunately, the way Rails handles the test data means that you are forced to repeat any structural data in both the migrations and in the fixtures. When a test is started in Rails it purges the test database, then it recreates it by copying the schema of the development databaseThe details of how exactly this schema copying happens vary depending on the schema format you are using and whether you are using migrations but the end result is the same. No matter what you end up with and exact duplicate of your development databases schema.. This approach is not completely unreasonable, your test database always has an identical structure as your development database, however I see several problems with it.

Cloning the development database assumes that the development database is up-to-date. Most of the time development databases are up to date but if you checked out and forgot to do a rake migrate your development database could quite easily be out of date. If this happens you are going to see test failures and the reason is not going to be immediately obvious (I can hear it now, “but it works fine in dev…”).

Cloning assumes that the development database is the authoritative version of the schema. In my world it is not. The migrations are the authoritative version of the schema. When I go to production I am going to do it by executing the migrations not by dumping my development database schema.

The behavior to clone the database is duplicative. We already have a perfectly good way to create the needed database schema. Namely, the migrations that are going to run in production to product the schema against which this code will run. Why have more code to achieve that same result of building a schema?

Cloning the development database assumes that the structure is all that is important. This completely ignores the structural data which is just as important as the physical structure of the database. To work around this you have to duplicate this structural data in both the migrations and the test fixtures. And I despise repeating myself when I am programming…

The solution: schema_format = :migration

I finally got around to creating a solution (you can download the plugin here) that avoids all these problems. This plugin introduces a new schema format of :migration. This schema format treats the migrations you have put so much time and effort into creating as the final authority for what belongs in a database for your application. With this plugin installed and enabled tests will start by purging the existing test database and then running the migrations, in order, from 001 to the most recent migration. This guarantees that the tests will be run against the most recent schema that could possibly make it’s way to production.

This solves the first two issues I raised above. We will ignore the third issue, duplicative code, because the existing code must remain for compatibility reasons and it does not directly impact us, anyway. The fourth issue, structural data, is handled by the plugin also. At first blush it might appear that the behavior I described above would be sufficient to solve this issue also but it is not. This issue remains because the Fixtures code in ActiveRecord actively deletes all rows from a table before loading the fixture data into that table.

Purging a table before loading the fixture data helps isolate tests from one another by ensuring that a test will never get data that has been modified by a previous test. With transactional fixtures this is less of an issue but even with transactional fixture there are situation where modified data will not be removed at the end of a testFor example, if a test fails the data it modified/created will not be removed. This allows for more easy debugging of failed tests because state of the database is exactly as they left them. On the other hand this is only really useful for the very last test that fails.. Unfortunately, we usually want fixture data even for tables that contain structural data. This is because the structural data we currently have may not fully exercise the functionality of the associated model class. Having fixture data means that at some point those tables will get wiped.

To avoid this problem the migration schema format plugin includes functionality to protect records in the database that are not fixture data. This is achieved by changing the table purging behavior of fixture loading. Rather than purging the entire table the fixture loading code only deletes the record that has the same primary key as the fixture it is currently loading. This means that your fixture data and structural data can live in peace and harmony. The only constraint is that fixture data must never use a primary key that is also used by a piece of real structural data. That constraint is easy to deal with simply by using large values for the primary key in fixture data that needs to play nice with structural data.

Enabling the migration schema format is easy:

1. download the tar and unpack it into your vendor/plugins directory 2. edit your config/environments.rb to include the line ”config.active_record.schema_format = :migration ” within the Rails::Initializer.run do |config| block 3. add the line ”require 'shield_nonfixture_data'” to test/test_helper.rb immediately after require 'test_help'

And voila, you can test using migrations and structural data.