Even the errors are happy making

This has got to be my favorite RoR error message:

1) Error: test_truth(Admin::EscalationMgt::EscalationViewsControllerTest): ArgumentError: Admin::EscalationMgt is not missing constant EscalationViewsController!

Only in Rails would the fact that you are trying to use a constant that does exist be an error.

My New Job

I have been at my new job for just over a month now so I suppose it is time to post something about it. I joined Absolute Performance at the end of November. We do “operations management solutions” which is marketing speak for helping you – both reactively and proactively – keep your applications, and systems, available and meeting SLAs.

I am back to developing in Ruby which makes me immensely happy. The technology around here is a mix of C++ and Ruby but my primary role is working on the browser based front-end for our monitoring infrastructure. That means I spend the vast majority of my time in Rails. It has been a great pleasure moving back into Ruby after being exiled into PHP, on which really cannot be overstated.

With the new job I expect my posting frequency to pick up a little. At my previous company there turned out to be some concerns about my some of my posts. That really took the joy out of posting, and therefore reduced the number precipitously. I look forward to working my way back to something like my peak frequency.

DrySQL: Not All I Had Hoped

I happened upon DrySQL the other day and I was immediately interested. DrySQL is an add-on to the standard ActiveRecord support in Rails that uses a lot more of the meta-data in the database to generated the model classes. The standard ActiveRecord classes basically just use the column names to create accessor and modifier methods for the models. DrySQL takes the idea of using the database schema definition as a source of information much further. It figures out the primary key looking for the primary key column in the database, rather than a column named ‘id’. It setups up validations based on the data type of the columns in the database. It defines joins based on the referential constraints. So basically DrySQL is trying to make ActiveRecord what it should have been from the beginning.

I was really happy when I found that someone is working on improving ActiveRecord. I have always been annoyed by in the inconsistencies of Rails ActiveRecord. For example, Rails ActiveRecord requires some of the meta-data to reside in the DB and some to reside in the code has bugged me. Even worse your are actually required to duplicate some of the meta-data by the standard ActiveRecord library, for example validations.

I use the past tense for my happiness because today I installed DrySQL and I am no longer all that happy. It may well do exactly what it says, which I will just assume, but unfortunately I don’t have time to really test it. The reason I do not have time to test it is that it adds a least twenty four (24) seconds to the response time for every single request. One of the simplest requests the application handles goes from

Completed in 0.18736


Completed in 24.24459

simply by adding require_gem 'drysql' to environment.rb.

That twenty four seconds seems to be to build the User model. Requests that instantiate other models incur even greater overhead. This kind of performance hit is completely untenable. While this overhead would be less of an issue in production because this generation will only be done once per model class it is still a complete show stopper for me, and I suspect for most other people. Really, who can give up thirty seconds of their life every single request, even in their development environment?

There is a possibility, and I really hope this is the case, that this is performance issue really just artifact of my environment. I am running a fairly recent version of Edge Rails (a couple of weeks old), postgres 0.7.1 (installed as gem) and postgresql-8.1 all on Edgy. I would love to hear about it if you see a problem with my setup (or if you are using DrySQL without these issues).

Even though this version of DrySQL does not seem to work for today I remain hopeful. DrySQL is definitely the way ActiveRecord should work and there is no reason generating the model class should take an excessive amount of time. Perhaps the next version of DrySQL will be more performant.

RailsConf Ticket

I am not going to be able to attend RailsConf this year. That means I have one RailsConf ticket I would like to sell at face value ($400 US). I also have a conference rate ($100/night) hotel reservation for June 22-26 at the Wyndham O’Hare that I will transfer to whomever buys the ticket. If you are intetest please email me at pezra@barelyenough.org.

Last Day at CFP Board

Today is my last day at CFP Board. That means that this is my last day working with RoR professionally, for a while at least, and that makes me a little sad.

RoR is a sweet piece of work. It is a platform on which a developer can be enormously productive. For example, I fixed three minor bugs before lunch today. By fixed I mean I analyzed the issue, implemented a test, implemented the change, and deployed the new version into production. I have not worked with any other system that supported that kind of velocity with any level of confidence. The dynamic nature of Ruby and Rails allows you to move amazing fast and the tight integration of testing into RoR allow you to do so without having to compromise the quality of the system.

Velocity is like a drug, the more you have the more you want. I have been super productive the last couple of weeks and everyday I wanted just a little more. Over the long run the pace I have had the last couple of weeks is not sustainableAt some point I will remember I have over
2000 unread items in my aggregator and before I know it whole days
will be spent just learning stuff. Useful to be sure but it does not
have the same umph as writing a boat load code.
, but damn it feels good while it lasts.

Tonic — A RESTful Web Application Framework

TonicVia [Sam
is a very promising RESTful web application framework for PHP. If you are doing web development in PHP I definitely suggest you take a closer look.

I am always interested in how RESTful behaviors are implemented, regardless of the language and environment used so I went and had a little peek at the docs. I noticed a couple of details that I think are interesting and might be relevant to the RESTful Rails work going on now.

Controller–Resource Type Relationship

Tonic seems to define a controller class (though they call them something different) per resource type. This is different than any of the rails approaches I have seen, which combine the code to support multiple resource types into a single controller class. It is not clear to me, from the documentation I have read so far anyway, exactly how collection resources with addressable members are handled. For example, if http://store.example/products/ is a collection of all known products and http://store.example/products/magik-eight-ball is a particular product how does Tonic handle route those requests to the appropriate resource controllers?

Resource Modifier Resources

Tonic uses the concept of editor and delete resources to manage the modification and deletion of resources. This is quite similar to Charlie’s approach. This is at least one significant difference. It appears–but I am not sure about this so correct me if I am wrong–that you post to the editor to update the resource. From an REST standpoint that is fairly unclean but it would make handling bad form data issues Charlie talked about a lot easier. And if you also supported PUT on the target resource you app would still be RESTful.

Structural Data in Rails

On a recent project I ran into a situation where I needed some structural data. I was writing a conference registration application. Each track at the conference costs a different amount and attendees can sign up for more than one track. We already have an accounting infrastructure that has a concept of a “product”, which is just something for which we charge money. So to support registering attendees for tracks, taking their money and letting the bean counters know that we have taken their money I needed to add some new products representing the registration for each track. However these products are far from mere cruft need by the accounting infrastructure, the conference registration code is completely dependent on the existence of these products. There are a set of check boxes on the registration page that the controller maps into the appropriate products from the products table. That means that the conference registration code will simply not work with out those products.

These products are the sort of data I am referring too when I say structural data. In my mind any data whose absence would cause a failure or that is managed as part of the development process, rather than in the application runtime, is structural data. This sort of data (dare I say “pattern”?) occurs fairly frequently, in my experience, and can be used to great effect

Given the fact that the code is tightly coupled to structural data it makes sense to manage structural data in the same way you manage the database schema. If you are using Rails odds are you are managing your schema with migration( and if you aren’t you should be). Migration is a great way to manage a rapidly changing database schema, and it easily supports creation and modification of structural data. Using migrations in this way has several benefits. It keeps versions of the structural data associated with the versions of the code with which they were intended to work (just check out the project from Subversion and you ready to go). It ensures that those products get inserted when the conference registration code gets deployed (migrating the database is part of the deployment process). And finally it places those vital database records under revision control.

The Problem

This technique works very well with the exception of testing. Unfortunately, the way Rails handles the test data means that you are forced to repeat any structural data in both the migrations and in the fixtures. When a test is started in Rails it purges the test database, then it recreates it by copying the schema of the development databaseThe details of how exactly this schema
copying happens vary depending on the schema format you are using and
whether you are using migrations but the end result is the same. No
matter what you end up with and exact duplicate of your development
databases schema.
. This approach is not completely unreasonable, your test database always has an identical structure as your development database, however I see several problems with it.

Cloning the development database assumes that the development database is up-to-date. Most of the time development databases are up to date but if you checked out and forgot to do a rake migrate your development database could quite easily be out of date. If this happens you are going to see test failures and the reason is not going to be immediately obvious (I can hear it now, “but it works fine in dev…”).

Cloning assumes that the development database is the authoritative version of the schema. In my world it is not. The migrations are the authoritative version of the schema. When I go to production I am going to do it by executing the migrations not by dumping my development database schema.

The behavior to clone the database is duplicative. We already have a perfectly good way to create the needed database schema. Namely, the migrations that are going to run in production to product the schema against which this code will run. Why have more code to achieve that same result of building a schema?

Cloning the development database assumes that the structure is all that is important. This completely ignores the structural data which is just as important as the physical structure of the database. To work around this you have to duplicate this structural data in both the migrations and the test fixtures. And I despise repeating myself when I am programming…

The solution: schema_format = :migration

I finally got around to creating a solution (you can download the plugin here) that avoids all these problems. This plugin introduces a new schema format of :migration. This schema format treats the migrations you have put so much time and effort into creating as the final authority for what belongs in a database for your application. With this plugin installed and enabled tests will start by purging the existing test database and then running the migrations, in order, from 001 to the most recent migration. This guarantees that the tests will be run against the most recent schema that could possibly make it’s way to production.

This solves the first two issues I raised above. We will ignore the third issue, duplicative code, because the existing code must remain for compatibility reasons and it does not directly impact us, anyway. The fourth issue, structural data, is handled by the plugin also. At first blush it might appear that the behavior I described above would be sufficient to solve this issue also but it is not. This issue remains because the Fixtures code in ActiveRecord actively deletes all rows from a table before loading the fixture data into that table.

Purging a table before loading the fixture data helps isolate tests from one another by ensuring that a test will never get data that has been modified by a previous test. With transactional fixtures this is less of an issue but even with transactional fixture there are situation where modified data will not be removed at the end of a testFor example, if a test fails the data it
modified/created will not be removed. This allows for more easy
debugging of failed tests because state of the database is exactly as
they left them. On the other hand this is only really useful for the
very last test that fails.
. Unfortunately, we usually want fixture data even for tables that contain structural data. This is because the structural data we currently have may not fully exercise the functionality of the associated model class. Having fixture data means that at some point those tables will get wiped.

To avoid this problem the migration schema format plugin includes functionality to protect records in the database that are not fixture data. This is achieved by changing the table purging behavior of fixture loading. Rather than purging the entire table the fixture loading code only deletes the record that has the same primary key as the fixture it is currently loading. This means that your fixture data and structural data can live in peace and harmony. The only constraint is that fixture data must never use a primary key that is also used by a piece of real structural data. That constraint is easy to deal with simply by using large values for the primary key in fixture data that needs to play nice with structural data.

Enabling the migration schema format is easy:

1. download the tar and unpack it into your vendor/plugins directory 2. edit your config/environments.rb to include the line ”config.active_record.schema_format = :migration ” within the Rails::Initializer.run do |config| block 3. add the line ”require 'shield_nonfixture_data'” to test/test_helper.rb immediately after require 'test_help'

And voila, you can test using migrations and structural data.

(Another) Rest Controller for Rails

Charlie has released his take on a RestController for Rails.

That is very sweet. It is great to see more work on RESTful Rails. It seems to me that each attempt gets closer to an approach I could believe in and be proud of. And, I get a warm fuzzy feeling any time I see a domain specific language developing. The resource handler Charlie has created is definitely part of a DSL there.

He brings up a few issue that result from his implementation. Some of which are important and some, IMHO, are not.

Leaky Abstractions

Charlie points out that the abstractions start to leak a little when you get to creating the views for a RestController.

The main issue is the method renaming. You have to know about it since you need to create templates called get.rhtml, get_member.rhtml, etc. It also comes into play if you want to turn on or off filters.

That is very unfortunate. The easiest way to solve this problem might be rethink what makes up a controller. The RestController design seem intent on combining the functionally of a cluster of related resources. In the example he provides the ProductController support interaction with the following resources

* every known product * the collection containing every known product * an editor for product resources * a creator for product resources

This set of resources are very cohesive and quite coupled to one another so combining them into a single controller is reasonable. But it causes this problem that you have know that the RestController is going to take

resource :Member do
  def get

and turn it into an action named get_member.

Perhaps it would be better to conceptualize a controller as a bit of code that mediates interaction with exactly one type of resource. With this view of the world you would end up with more, smaller, controllers. Charlie’s product example would look more like

class ProductController < ApplicationController
  include YarController  # that's YetAnotherRestController
  verb :get do
    @product = Product.find(params[:id])
  verb :put do
    @product = Product.find(params[:id])
      flash[:notice] = 'Product was successfully updated.'
      redirect_to :id => @product
    rescue => e
      # Send the current invalid values to the editor via the flash
      flash[:product] = @product
      redirect_to :resource => :editor, :id => @product
  verb :delete do
    redirect_to :id => nil, :resource => nil

Class ProductsController < ApplicationController
  include YarContoller
  verb :get do
    @product_pages, @products = paginate :products, :per_page => 10
  verb :post do
    @product = Product.new(params[:product])
      flash[:notice] = 'Product was successfully created.'
      redirect_to :resource => :collection
    rescue => e
      flash[:product] = @product
      redirect_to :resource => :editor

And so on… The main benefits of this is that the template to rendered for a get of ‘http://mystore.example/product/243’ is ‘apps/view/product/get.rhtml’ and, I think, the issues with filters go away, too. The down side is that you end up with four controllers for each basic type of resource you expose. One for the basic resource type you want to expose, one for the collection of all of those basic resources, one for the creator resource and one for the editor resource. I don’t know if the extra boiler plate code is worth the benefits but it feels like it might be.


Charlie also points out

a pure REST solution does not work with HTML forms since browsers don’t support PUT and DELETE

He is absolutely correct. However this tunneling PUT and DELETE over POST kludge does not bother me very much. I will now take a moment to revel in being more pragmatic than Charlie, quite possibly for the first time since I meet him seven years ago. Anyway, it is ugly that HTML does not support PUT and DELETE but still very workable.

Handling Bad Data

Finally there is an issue with handling failed attempts PUT/POST. This is the one that bothers me the most. It is not really all that bad from a pragmatic standpoint, storing this info in state works fine. However, it implies a certain weakness in my world view because I did not see it coming.

If the post fails we have to store the ill-formed product into the flash and redirect back to the editor since its at a different URL.

The fundamental problem here is that the separate editor resource will PUT the modified resource when you click save/submit. But what if you messed it up and, say, violated the business rule that blue products must have a price that is divisible by three? In a normal Rails app that proposed change would fail validation and the update action would just re-render the edit page with bad fields highlighted. But in a RESTful world the editor and the validation code are separate and it is wrong from REST stand point to just render the editor resource from in response to a product resource request. However, if you don’t do that you need to get the form data, and which fields are bad, from the previous attempt so that you can re-render the editor with the information the user previously entered and what was wrong with it.

One way you could solve this problem is to allow the creation of “invalid” resources. For example, you require a product to have a description. However, you receive a POST to ‘http://mystore.example/products’ without a description. You could issue the product an ID and store it in it’s invalid state (without a description) and then redirect the browser to the editor resource for that newly created, but invalid, product. That feels really clean from a design stand point but I am not sure how difficult it would be to implement. And you would certainly end up to permanently invalid resources, which might be hard to manage in the future.