Comments on: Pipes Dreams

By: Mat Morrison

Mat Morrison — Thu, 12 Mar 2009 06:48:15 +0000

“The inability to create custom reusable transformations is a real missed opportunity.”

As Brendan says, you can use pipes in pipes (but only in the loop module.) Indeed, it’s considered best practice to do this — and lots of pipes developers create re-usable objects like this. Most of the ones I’ve built recently around APIs like Technorati’s or Twitter’s use a modular pipe like this.

But I agree — the ability to edit a pipe in a text editor would be brilliant. There’s no real reason that I can see why there shouldn’t be a text representation of a pipe that’s accessible to the user, and I can see how powerful that would be.

You might like to look at the FriendFeed Yahoo! Pipes list (http://friendfeed.com/list/pipes)

Or see what I’ve been trying to do w/ pipes (http://mediaczar.com/blog/category/pipes)

By: Brendan Cooper

Brendan Cooper — Wed, 11 Mar 2009 16:48:40 +0000

You can insert pipes as modules within other pipes, when you put them into a loop operator. Check out Mat’s Technorati Authority pipe at http://mediaczar.com/blog/2009/01/technorati-authority-yahoo-pipe/.

But I agree with you on the interface. It does frequently go wrong and it’s very frustrating.

Instead of going back to fully manual, why not subscribe to the feeds using something like Google Reader which you can then filter? Or even create your aggregated feeds in Google Reader, send that to a Yahoo Pipe which has the filters you want in place, then subscribe to the output RSS from that?

Good luck with the hunt.

By: Pipes in Text « Authsider

Pipes in Text « Authsider — Sun, 01 Mar 2009 20:39:02 +0000

[...] March 1, 2009 Cloud Computing , IT , SaaS , Trends A few days ago, Stefan joined a rant of Peter Williams on Yahoo! Pipes lack of a text representation: While for many models (and programs, and anything in [...]

By: Evgeny

Evgeny — Sun, 01 Mar 2009 16:14:12 +0000

Ohh, and just stumbled on this Pipes and YQL (didn’t even know about that one) : http://www.infoq.com/news/2009/02/Pipes-YQL-Jonathan-Trevor

Maybe YQL is actually what you are describing.

By: Robert Brewer

Robert Brewer — Thu, 26 Feb 2009 17:33:01 +0000

Juju.com seems to be trying to tackle that problem commercially (but with search indexing, not Yahoo! pipes). Give ‘em a spin.

Oh, and if your Python is up to par, apply for a job there too: http://juju.standoutjobs.com/jobs/932-python-developer They could use a RESTevangelist PDQ. ;)

By: Catherine

Catherine — Thu, 26 Feb 2009 15:27:17 +0000

Hardly an expert, you’re biased.

By: Peter Williams

Peter Williams — Thu, 26 Feb 2009 15:09:56 +0000

What free time? At the moment I am a stay-at-home dad with 2 kids to watch and a job to find. :) I could do what you are describing. In fact, I have considered it. I think it might be a cool project to use to explore Erlang a little.

By: Evgeny

Evgeny — Thu, 26 Feb 2009 09:26:05 +0000

Call it rb-pipes, host it on github, and everyone will say thank you.

By: Evgeny

Evgeny — Thu, 26 Feb 2009 09:25:03 +0000

Hmm... Peter???

How about you use your spare free time, your textual editing skills, and your ruby experience to construct a dsl for pipes. Then you can run it with hpricot/nokogiri/mechanize and the output will be an rss feed.

Just like pipes, but not visual and buggy and restricted, but open, flexible, fast and reliable.

You can use it to save on the time where you go enter information into 40 websites.