05 Apr 2013
•
Software Development
It seems that most APIs — particularly internal ones — are not designed for ease of use but rather to be easy to implement. No one would expect a human facing product designed that way to be successful. We should not expect APIs to be any different.
Web APIs are products in their own right. That means all those rules for building great products, like understanding your users and their use cases, apply. APIs are not just high latency, bandwidth hogging database connections. Rather an API should expose an application and the business value it provides. This means understanding what clients want to accomplish and then affording those uses in easy, intuitive ways.
Communication with the users is the key to designing a great API. As with other types of products, it is often necessary to build the first version of an API before there are any developers using it. We are on shaky ground until our design is validated actual clients. As soon as there are actual, or even potential, client developers listening to, and integrating their feed back should be priority number one.
Listening doesn’t mean reflexively implementing every whim of users — users are not always right about the details — but by understanding what they are trying to accomplish we as API designers can build systems that afford those goals with a minimum of effort on the part of client developers. Facilitating that value creation should be our main goal as API designers.
27 Feb 2013
•
Software Development
Dear library developers, please knock that shit off immediately.
We all seem to accept the wisdom of semantic versioning these days (thank goodness). Somehow, though, it has not occurred to many library developers that locking the first slot of the version to 0 means you give up all those benefits. Incrementing the first slot is how clients are informed of incompatible changes. If you never change the first slot you necessarily stop communicating this information.
If you have released you library to the rest of the world it should not have a ‘0.’ version. Period. If you think most people probably should not be using your library, add a pre-release tag to the version. If you want to tell the world that the API is likely to change use the docs/readme, that is why it exists. Or you could skip telling us altogether because everybody already knows the API is likely to change. That is why we came up with semantic versioning in the first place.
05 Dec 2012
•
Software Development
Designing the messages (or representations, i’ll use the terms interchangeably) is the most important part of API design. If you get the messages right everything else will flow naturally from them. Of course, there are trade offs that must be made when designing messages. One of those trade offs is how much data to put in each message. If they are too small clients must make too many calls to be performant. If they are too big generating, transferring and parsing the messages will be excessively slow.
Any entity worth discussing should have a URI of it’s very own. That is, it should be a resource. This means that we often (read: almost always) end up with a lot of resources that don’t really have much data directly. The usual pattern is that they have a small number of properties and then link to a bunch of other resources. For example consider an invoice: a few properties like purchase date, etc and then links to the customer, billing address, shipping address, and line items. The line items would, in turn, link to a product.
We often bulk up the representations of these lightweight resources by directly embedding representations of the other resources to which they link. This tends to reduce the number of requests needed because those embedded representations don’t need to be requested explicitly. This approach has substantial downsides, at least if implemented naively. Consider the following representation of an invoice with embedded representations.
{"purchase_date" : "2012-10-29T4:00Z",
"customer" :
{"uri" : "http://example.com/custs/42",
"name": "Peter Williams",
//...
},
"billing_address" :
{"uri" : "http://example.com/addrs/24",
"line1" : "123 Main St",
//...
},
// etc, etc
"line_items" :
[{"uri" : "http://example.com/li/84",
"quantity": 3,
"product" :
{"uri" : "...",
"name": "Blue widget",
"desc": "..."
},
},
// other line items here
]
}
This approach is very appealing. All the data needed to display or operate on a invoice is right there at our fingertips which nicely manages the number of requests that need to be made. The data is also arranged in a logical way that makes sense to our human brains.
For all of its upsides, the downsides to this approach are substantial. The biggest issue, to my mind, is that it limits our ability to evolve this message over time. By directly embedding the line item and product data, for example, we are signalling that they are fundamentally part of this representation. Clients will implement code assuming those embedded resources are always there. That means we can never remove them without breaking clients.
There are many reasons we might want to remove those embedded representations. We might start seeing invoices with a lot of line items thereby resulting in excessively large messages. We might add a lot of properties to products and make the messages too large that way. We might move products to a different database and find that looking the all up takes too long. These are just a few of the innumerable reasons that we might want change our minds about embedding.
How small is too small?
Given that removing a property from a representation is a breaking change are there ways to design representations that reduce the possibility that we will need to remove properties in the future? The only real way is to make representations as small as possible. We will never need to remove a property that was never added in the first place. We already discussed how messages that are too small can result in excessive numbers of requests but is that really true?
Applying the yagni principle is in order when thinking about embedding. Embedding is easy to do and very super extremely hard to undo. It should be avoided until it is absolutely necessary. We will know it is absolutely necessary when, and only when, we have empirical evidence showing that now is the time. This will happen quite rarely in practice. Even when we have empirical evidence that our request volume is too high, solutions other than embedding are usually a better choice. Caching, in particular, can ameliorate most of the load problems we are likely to encounter. The fastest way to get a representation is not to embed it into another message that is passed over the wire but to fetch it out of a local cache and avoid the network altogether.
Embedding one representation inside another is an optimization. Be sure it is not premature before proceeding.
sometimes – not often, but sometimes – i like the idea of embedding
Annoyingly, sometimes optimizations really are required. In those situations where we have clear empirical evidence that the current approach produces too many requests, we have already implemented caching and we cannot think of another way to solve the problem embedding can be useful. Even in these situations embed should not done hierarchically as in the example above. Rather we should sequester the embedded representations off to the side so that it is clear to clients that they are an optimization. If we can signal that clients should not assume they well always be embedded all the better.
The following is an example of how this might be accomplished using our previous example.
{"purchase_date" : "2012-10-29T4:00Z",
"customer_uri" : "http://example.com/custs/42",
"billing_address_uri" : "http://example.com/addrs/24",
"shipping_address_uri": "http://example.com/addrs/24",
"line_item_uris" :
["http://example.com/li/84",
"http://example.com/li/85"],
"embedded":
[{"uri" : "http://example.com/custs/42",
"name": "Peter Williams",
//...
},
{"uri" : "http://example.com/addrs/24",
"line1" : "123 Main St",
//...
},
{"uri" : "http://example.com/li/84",
"quantity": 3,
"product_uri" : "http://example.com/prods/12"
},
{"uri" : "http://example.com/prods/12",
"name": "Blue widget",
"desc": "..."
},
// and so on and so forth
]
}
The _uri and _uris properties are links. A client looks for the relationship it needs and then first looks for a representation in the embedded section with the required uri. If it finds one then a network communication has been avoided, if not it can make a request to get the needed data. This approach clearly identifies representations that are embedded as an optimization and makes it easy for clients to avoid relying on that optimization to behave correctly.
This flat embedding is the approach taken by both HAL and Collection+JSON (albeit with some slightly different nuances). I suspect that the developers of both of those formats have experienced first hand the pains of having representations getting too big but not being able to easily reduce their size without breaking clients. If one of those formats work you use them, they have already solved a lot of these problems.
Other considerations
Avoiding hierarchical embedding also makes documenting your representations easier. With the sidecar style you can keep each representation to a bare mimimum size and only have to document one “profile” of representation for each flavor of resource you have. With this approach there is no difference between the representation of a customer when it is embedded vs when it is the root representation.
26 Nov 2012
•
Personal
•
Software Development
Where are the cool innovations of the next decade happening so i can find a job working on them?
I am starting to look for a job and for the first time in many years i am not sure what sort of job i want. I have made a good career being on the bleeding edge of web development. I believed in the superiority and inevitability of the web long before most people. I have advocated for hypermedia APIs to bring many of the web’s benefits to application integration. These beliefs allowed me to work in some of the most innovative communities in existence, the ones around Rails, Ruby and web APIs.
The web was once untested, immature and respectably revolutionary. That has all changed. Web applications are now the norm, as are the technologies that grew up with it. More recently hypermedia APIs have gained real traction. This is a good thing but it makes me nostalgic. I don’t find implementing “best practices” using “proven technology” all that appealing. Marginal improvements are good — great, even — but the are not particularly inspiring.
I need to find a new revolution to join. But finding nascent revolutions is not easy. They are by definition small and hard to see. I am not even sure if i am looking for a revolution in technology, or a revolution in an important domain that is enabled by technology.
Big data?
On the revolution in technology side, one possibility is data science and big data. These are getting a lot of hype these days and the tools seem pretty immature over there. The immaturity is very appealing; the hype not so much. The fact that it is heavily hyped might mean that it is close to peaking. Joining a revolution that is already over is no fun.
Cloud?
Cloud technologies feel similar to big data to me, just further along. Lots of good ideas but the heavy lifting is already done. Pretty much everyone accepts that cloud anything is a good idea and most of the important capabilities already exist. Obviously there are tons of improvements to be made but from the outside they all seem pretty incremental.
Specific problem domain?
Another possibility is to focus on a problem domain, rather than a technology set. Finding the right domain is a huge challenge, though. I would want the domain to be one that was ripe for some drastic innovation and could make the world a much better place. I think that means either a new, immature sector or a industry that has stagnated for many years. Education and energy spring to mind immediately but i am sure there are others.
I really don’t know what to focus on. The more i think about it the more i like the idea of working on a problem domain but how do i decide which one? I need to decide pretty quickly, not having a long term direction makes it hard to take immediate steps. If you have any thoughts or advice i’d love to hear them.
11 Nov 2012
•
Uncategorized
We are having an API meetup at Lucky Pie in Louisville, CO on nov 15th @ 6:30pm. Come share a tasty beverage, a slice of pizza and your opinions on all things API with your fellow API crafters. If you are an API practitioner we’d love to meet you in person.
BTW, if you are in Denver for Defrag this drinkup is a short drive from the Omni Interlocken so please join us.
If you register it’ll help ensure we have enough space, but all are welcome, registered or not.