That approach has the advantage of being simple and easy to understand. Its main downside is that it makes it difficult for existing clients to switch to a newer version of the if one becomes available. The difficultly arises because most existing clients will have bookmarked certain resources that are needed to accomplish their goals. Such bookmarks complicate the upgrade quite significantly. Clients who want to use an upgraded API must choose to rewrite those bookmarks based on some out of band knowledge, support both the old and new version of the API, or force the user to start over from scratch.

None of these are good options. The simplest, most attractive approach is the first. However, forcing clients to mangle saved URIs reduces the freedom of the server to evolve. The translation between the two versions of the API will have to be obvious and simple. That means you are going to have to preserve key parts of the URI into the new structure. You cannot switch from a numeric surrogate key to a slug to improve your SEO. Likewise, cannot move from a slug to a numeric surrogate key to prevent name collisions. You never know when the upgrade script will be executed. It could be years from now so you will also need to maintain those URIs forever. Some clients have probably bookmarked some resources that you do not think of as entry points, you will need to be this careful for every resource in your system.

The second option, forcing clients to support both versions of the API, is even worse that the first. This means that once a particular instance of a client has used the API it is permanently locked into that version of that API. This is horrible because it means that early users cannot take advantage of new functionality in the API. It is also means that deprecated versions of the API must be maintained much longer than would otherwise be necessary.

The third option, forcing users to start over from scratch, is what client writers must do if they want to use functionality which is not available in the obsolete version when there is no clear upgrade path between API versions. This is not much work for the client or server implementers but it seriously sucks for the users. Any configuration, and maybe even previous work, is lost and they are forced to recreate it.

A way forward

Given that this style of versioning is the most common we need a solution. The link header provides one possible solution. We can introduce a link to relate the old and new versions of logically equivalent resources. When introducing a breaking API change the server bumps the API version and changes the URIs in any way it likes, eg the new URI might be http://example.com/v2/products/super-widget. In the old version of the API a link header is added to responses to indicated the equivalent resource in the new API, eg http://example.com/v2/rels/v2-equivalent.

>>>
GET /v1/orders/42 HTTP/1.1
...

<<<
HTTP/1.1 200 OK
link: <http://example.com/v2/orders/super-widget>; rel="alternate http://example.com/v2/rels/v2-equivalent"
...

Older clients will happily ignore this addition and continue to work correctly. Newer clients will check every response involving a stored URI for the presences of such a link and will treat it as a redirect. That is, they will follow the link and use the most modern variant they support.

If you are really bad at API design you can stack these links. For example, the v1 variants might have links to both the v2 and v3 variants. Chaining might also work but it would require clients to, at least, be aware that any intermediate version upgrade link relations so that they could follow that chain to the version they prefer.

You could also add links to the obsolescent variant’s body. This would be almost equivalent except that it requires clients to be able to parse older responses enough to search for the presence of such a link. Using the HTTP link header field nicely removes that requirement by moving the link from the arbitrarily formatted body to the HTTP header which will be supported by all reasonable HTTP clients.

Using URIs to version APIs may not be the cleanest way to implement versioning but the power of hypermedia allows us to work around its most obvious deficiencies. This is good given the prevalence of that approach to versioning.

And a lot of that performance, Prasad said, came from removing unnecessary design wankery (our verbiage, not his) — the rounded corners, the omnipresent gradients. By making things simple, clean, modern, flat, and even print magazine-like, the LinkedIn app only got faster and better on the performance side, as well.

— Venture Beat

In the past there has been a sense of disconnection between the business, QA and development parts of our team. (An all too common situation in my experience.) If quality actually matters there must be good connections between all the stakeholders and quality must be considered at every step of the process. We don’t have a product manager, or a large QA team². That means that if quality is going to be considered at every step it is going to be because the developers are doing it, there is no one else at many of the steps.

Our solution is to acknowledge developer responsibility for all aspects of feature development. That unification reduces the chances of things being overlooked. It does change the workload of developers, though. Now developers must:

• elicit requirements from all stakeholders
• write testable acceptance criteria
• develop automated acceptance tests
• implement the feature
• verify all acceptance (included pre-existing ones) pass
• verify that the feature, as implemented, matches the desire of the champion
• deploy changes to production

This has lightened the load on QA to the point that our one QA resource can keep up with four developers. QA still has a lot of responsibility:

• verify the acceptance criteria make sense
• verify the acceptance criteria is testable
• verify the acceptance criteria cover all important variations of the feature
• verify the acceptance tests cover all important variations
• refine acceptance tests
• verify all tests pass in a production like environment
• perform visual inspections in various browsers
• accept features into the production branch

We have been using this version³ of the process for a while now. I think is has worked really well. In fact, during our last retrospective our QA guy stated that he thought our quality management processes belonged in the “Good things” category. I think that is a first i have ever heard a QA person be so positive about processes.

Only time will tell if this approach produces significantly better results but I am very optimistic. So far it seems to have created a world of difference.

The exact scenario was as follows, the information we store about a file is potentially invalidated with every change to the contents of that file. For example, some code might be added that has different licensing requirements. Or all the code that has a particular licensing requirement might be removed. To make sure that the information in and SPDX file is valid for a particular file it must be possible to verify that its contents are identical to the contents of the file that was analyzed.

In SPDX we support verification of file contents by providing message digests¹ of every file. However, message digest (hash) functions come and go. MD/5 used to be the norm but has fallen out of favor. SHA1 is currently very popular but is steadily being replaced with SHA256 and SHA512. We definitely need to support more than one digest algorithm.

We had three basic choices:

• have separate properties for each digest algorithm, each of which would be a sub-property of the checksum property
• define datatypes for each digest algorithm and have single checksum property
• define a class for digests that encapsulates all the data and have a single checksum property

For the first option the graph would look like

<http://zlib.net/zlib-1.2.5.tar.gz#Makefile> a spdx:file;
  spdx:sha1 "1fac389…"^^xs:hexBinary;
  spdx:sha256 "4aa8223…"^^xs:hexBinary.

The resulting graphs are simple and self explanatory. Support for new digest algorithms is achieved by the addition of one new properties for each algorithm.

One potential downside is that some tools might not be able to pass through digests for algorithms the do not understand. For example, SPDX provides a tool to translate SPDX RDF data to and from spreadsheets. The digests are inserted into the spreadsheet by the tool looking for the known digest properties and putting that data into the appropriate column in the spreadsheet. This means that novel digest types would be lost in the translation. This could be avoided if the tool supported OWL inferencing but it is unlikely we will implement that in the near future. I think requiring OWL inferencing to work properly is a design smell.

A graph for the second option would look like

<http://zlib.net/zlib-1.2.5.tar.gz#Makefile> a spdx:file;
  spdx:checksum "1fac389…"^^spdx:sha1Hex;
  spdx:checksum "4aa8223…"^^spdx:sha256Hex.

The moves the digest algorithm into the datatype specification of the literals. This approach seems quite elegant. There is a single property so it is easy for tools deal with. It is extensible, anyone could define a new datatype. It is relatively compact.

However, there very few ontologies that use xml datatypes in this way. This could be because there are subtle problems with this approach. Or it could be that it is just uncommon. This approach would break down for algorithms that have any secondary parameters. In that case you could combine it with the third option, though.

A graph for the third option would look like

<http://zlib.net/zlib-1.2.5.tar.gz#Makefile> a spdx:file;
  spdx:checksum [spdx:algorithm <spdx:sha1>;
                 spdx:checksumValue "1fac389…"^^xs:hexBinary];
  spdx:checksum [spdx:algorithm <spdx:sha256>;
                 spdx:checksumValue "4aa8223…"^^xs:hexBinary].

In many ways this approach is very similar to the datatype based one. The introduction of anonymous resources does, potentially, allow the addition of additional parameters to the digest algorithm. However, tools that do not understand the algorithm would probably not pass that information though correctly. One practical upside is that the label for particular algorithms can be stored in the RDF. The <spdx:sha1> resource can have a dc:title property with value “SHA1”. This would mean that tools don’t necessarily have to implicitly understand a digest type in order to display it to humans.

In the end we decided to use the third approach on SPDX. The additional flexibility was generally found appealing. Having to define a new property for each new digest algorithm was generally viewed as a bit of a kludge. When i presented this issue to the semantic web mailing list there was only one response which preferred the first option, but found the third option acceptable. Most of the people involved in the SPDX effort are not highly experienced RDF modelers. I am not sure if the distaste for defining new properties reflects our relative lack of experience with RDF or if it is more fundamental.

Feedback on this choice is welcome but this post is more of an exploration of the possibilities and implications of those approaches.

A large part of the appeal of XML based languages is XML provides a reasonable balance between those two factors – for programmers. It can be read easily by computers and understood reasonably well by a programmer with very limited tooling.

For the non-programmer the story is somewhat different. Most XML based languages are complete gibberish to people without significant technical expertise. A business person will need non-trivial tools to allow them to consume the information locked up in an XML file.

Data interchange formats implicitly value the wider distribution of information. Why else would you be exchanging data. It is disappointing that so many of these formats are based on technology the excludes all but those with sophisticated tools or deep technical knowledge. Data interchange formats should be designed first for people, and second for computers. A properly designed data interchange format should be consumable, using commonly available tools, by any person who is familiar with the domain.

This means that XML is pretty much right out.

Fortunately there is HTML+RDFa. RDFa allows RDF data sets to be serialized into HTML documents. The information can then be consumed by humans using any web browser. The raw data can be readily extracted by tools.

Consider the following two examples. Each is part of an SPDX¹ file. In first example, HTML+RDFa, both groups are easily supported. The information is displayed in a way that it can be understood by most human and the data is machine readable. In the second the information is machine readable but quite difficult for humans to interpret.

The following is a similar amount of information expressed in RDF/XML

<spdx:File rdf:about="https://olex.openlogic.com/package_versions/download/9423?path=openlogic/zlib/1.2.5/openlogic-zlib-1.2.5-all-src-1.zip=3690#CMakeLists.txt">
  <spdx:Copyright>unknown</spdx:Copyright>
  <spdx:License rdf:resource="http://spdx.org/licenses/Zlib"/>
  <spdx:Name>CMakeLists.txt</spdx:Name>
  <spdx:Type>source</spdx:Type>
  <spdx:sha1>L_NaoTEuHjO8uI1VLGIai2rDk7wPkoyeSe5vVD1gxOc</spdx:sha1>
</spdx:File>
<spdx:File rdf:about="https://olex.openlogic.com/package_versions/download/9423?path=openlogic/zlib/1.2.5/openlogic-zlib-1.2.5-all-src-1.zip=3690#ChangeLog">
  <spdx:Copyright>unknown</spdx:Copyright>
  <spdx:License rdf:resource="http://spdx.org/licenses/Zlib"/>
  <spdx:Name>ChangeLog</spdx:Name>
  <spdx:Type>other</spdx:Type>
  <spdx:sha1>rxKcRCSHu8.4tHMdiJRINMatY4efBCMz.PmHpo.gj9s</spdx:sha1>
</spdx:File>
<spdx:File rdf:about="https://olex.openlogic.com/package_versions/download/9423?path=openlogic/zlib/1.2.5/openlogic-zlib-1.2.5-all-src-1.zip=3690#FAQ">
  <spdx:Copyright>unknown</spdx:Copyright>
  <spdx:License rdf:resource="http://spdx.org/licenses/Zlib"/>
  <spdx:Name>FAQ</spdx:Name>
  <spdx:Type>other</spdx:Type>
  <spdx:sha1>qNaKL48VknhfAA7NgpOLjMoyHlxv45x.hMu8UfAy0Fc</spdx:sha1>
</spdx:File>
<spdx:File rdf:about="https://olex.openlogic.com/package_versions/download/9423?path=openlogic/zlib/1.2.5/openlogic-zlib-1.2.5-all-src-1.zip=3690#INDEX">
  <spdx:Copyright>unknown</spdx:Copyright>
  <spdx:License rdf:resource="http://spdx.org/licenses/Zlib"/>
  <spdx:Name>INDEX</spdx:Name>
  <spdx:Type>other</spdx:Type>
  <spdx:sha1>HMjPi3ZRY2Anq1vc4Lfl5x77JWj3hhaWWT.I34QPK9g</spdx:sha1>
</spdx:File>

The extreme accessibility of HTML+RDFa for both humans and machines makes it an obviously superior choice for data interchange formats. HTML+RDFa is a relatively new entry into the arena. Hopefully we will see more data formats use this superb technology.

Underneath all of those embarrassing braces and semicolons, JavaScript has always had a gorgeous object model at its heart.

That sums up my feelings about javascript almost exactly.

Synopsis

This library provides parsing of SAML 2.0 artifacts. For example.

artifact = Saml2::Type4Artifact.new_from_string(params['SAMLart'])
# => #<Saml2::Type4Artifact ...>
artifact.source_id    # => 'a314Xc8KaSd4fEJAd8R'
artifact.type_code    # => 4

Once you have an artifact you can resolve it into it’s associated assertion:

assertion = artifact.resolve     # => #<Saml2::Assertion>

With the assertion you can identify the user and retrieve attributes:

assertion.subject_name_id        # => '1234'
assertion['mail']                # => 'john.doe@idp.example'

Configuration

If you are using Rails the SamlSp will automatically load configuration info from config/saml_sp.conf.

For non-Rails apps the saml-sp configuration file can be place in the application configuration directory and loaded using the following code during application startup.

SamlSp::Config.load_file(APP_ROOT + "/config/saml_sp.conf")

Logging

If you are using saml-sp in a rails app it will automatically log to the Rails default logger. For non-Rails apps you can specify a Logger object to be used in the config file.

logger MY_APP_LOGGER

Artifact Resolution Service

For artifact resolution to take place you need to configure an artifact resolution service for the artifacts source. This is done by adding block similar to the following to your saml-sp config file.

artifact_resolution_service {
  source_id         'opaque-id-of-the-idp'
  uri               'https://samlar.idp.example/resolve-artifact'
  identity_provider 'http://idp.example/'
  service_provider  'http://your-domain.example/'
  http_basic_auth {
    realm    'the-idp-realm'
    user_id  'my-user-id'
    password 'my-password'
  }
}

The configuration details are:

• source_id: The id of the source that this resolution service can resolve. This is a 20 octet binary string.

• uri: The endpoint to which artifact resolve requests should be sent.

• identity_provider: The URI identifying the identity provider that issues assertions using the source id specified.

• service_provider: The URI identifying the your software (the service provider) to the identity provider.

• http_basic_auth: (Optional) The credentials needed to authenticate with the IdP using HTTP basic authentication.

Promiscuous Auth

If the IdP does not provide proper HTTP challenge responses you can specify the HTTP auth in promiscuous mode. For example,

http_basic_auth {
  promiscuous
  user_id  'my-user-id'
  password 'my-password'
}

In promiscuous mode the credentials are sent with every request to this resolutions service regardless of it’s realm.

Requirements

• Nokogiri
• Resourcful
• uuidtools

Install

• sudo gem install saml-sp

Fortunately, it turns out that many of these very large work loads are embarrassingly parallel problems. And look, you have several (dozen) workers just waiting to do your bidding. It just makes sense to break up these large blocks of work into many smaller chunks so that it can be processed in parallel.

Breaking a task up into many small parts comes with some issues. Any task that takes long enough to run asynchronously is probably going to take long enough that you need to track its progress.

Also the problem is probably not completely parallelizable. Most problems seem to have a large portion of easily parallelized work followed by a bit of work that can only happen after all the parallel work has been complete.

These patterns show up often enough that i have gotten tired of repeating myself. Hence was born resque-multi-step. Resque-multi-step is a Resque plugin that provides compound job support complete with progress tracking, error handling, and a completely serial finalization sequence.

Example

Say you want to reindex all the posts in a blog. However, committing solr for each post would be excessively slow. (Trust me, it really is.)

Resque::Plugins::MultiStepTask.create("reindex-#{blog.name}") do |task|
  blog.posts.each do |post|
    task.add_job ReindexWithoutCommit, post
  end

  task.add_finalization_job CommitSolr
end

This reindexs all the posts in parallel. Any available workers will pick up a job to reindex a specific blog post. Once all those reindex jobs have completed, the finalization job will be executed.

If you have more that one finalization job, they are executed serially in the order they were added to the task.

Administrivia

If these issues sound familiar give resque-multi-step a try. It is available as a gem so installing is just

gem install resque-multi-step

If you want to contribute head on over to the github project and hack away. If you come up with something useful i’ll integrate it post haste.

Resque checks the queues for jobs to process in a fixed order. (In alphabetic order, to be precise.) This turns out to be a problem is you want predictable handling time for jobs. For example, consider a system which has queues aaa and zzz. If you add 100 jobs to aaa and 1 job to zzz, the job on zzz will wait a long time before being processed.

This problem is easily solved by just checking the queues in random order. Over time, any particular queue will be checked early so a few deep queues will not starve the other queues in the system.

resque-fairly is a Resque plugin which provides that behavior. Just install the gem, add require 'resque-fairly' and Resque will handle queues with approximate fairness.