Flume: A Web Application Concept for Duplication Elimination

FlumeComp1
click to enlarge

I wrote a quick post to Google Buzz recently that dealt with a concept and issue that’s been rattling in the back of my mind lately. We consume so much information from so many sources that we’re bound to run into the same stories. The news that becomes popular does so because of it being shared, telling friends, sending it to others, and spreading the word.

Virality is the term that’s been associated with this for some time. Getting things to “go viral” is a key to success, but the value of any story is its exclusivity, or who publishes it first. The above concept image is a visualization of my thoughts on how this issue might be dealt with.

I call it – Flume.

1. flume – noun. – A narrow gorge, usually with a stream flowing through it.

The goal is to group all similar information from throughout the web, from social networks and from all of your friends, and distill it to just the original sources.

You’d have a list of news stories that are by default presented by popularity. Each story would get a score that represents how often it’s cited across the web – based on tweets and retweets from Twitter, inbound links from reactionary blog posts, and personal status update reactions from friends and colleagues on social networks like Facebook and Google Buzz. The presentation would be consistent regardless of the source, with simple links to click and visit any story in detail. You’d have at-a-glance, with no interaction needed from the user, an importance-ranked delivery of information stemming from the people and content sources they value most.

Behind the scenes, the application would detect when a story all points to the same common news event, and instead of you having to look elsewhere through your various feeds, walls, streams, etc., the story would show up only as a single source origin story – all within the confines of one application.

For example, if CNN gets the story first, that’s the one that would show up in Flume. All other stories that were posted anywhere on the web afterwards would be consolidated underneath it. Anyone posting about it on Twitter – hidden. Anything on Facebook about it – not seen. All of the reactions would fall in a toggleable area below each story, where if you desired to drill down into to see what people are saying, you’d know from where it came from and where it was being talked about. But only if you want to.

So how would it work?

You’d use a combination of keyword density, trackback links, un-shortened URL detection, and possibly the Salmon protocol to algorithmically distinguish the content streaming in as either original or reactionary. As more content arrives, whichever source has the content origin first gets the credit. So if your friend on Twitter was the first to post about a story before it arrived to you via someone else on Facebook, you’d only see the tweet. If the blog post came before the Buzz, you’d get the blog post alone.

The mockup design is obviously heavily influenced by TweetDeck. Where the multicolumn view works in TweetDeck’s favor, I’d like to take all the separated information and throw out anything other than the first place something appeared. Rather than having separate columns for each service, I’d want a single column for all news, regardless of source. If you wanted to of course, you could always re-sort it by source if you desired.

It’s a bit like Google Buzz or FriendFeed, where you have a single aggregated stream of information, but instead of being source-centric, the default is popularity centric. It’s throwing TweetDeck, Google Buzz, FriendFeed, Facebook, Brizzly, Redux, Amplify, Google Reader, and countless other web tools into a blender to create an ultimate single point-of-entry news reading mechanism.

HTML5 and web application standards would be ideal for the client to allow for both desktop and cross-platform mobile usage, while a robust database application layer in the cloud would be constantly evaluating and delivering the ranked information requested by the client.

What do you think? Would a single view of content without the same stories popping up over and over interest you? What other features would you expect to see in this application? What are some of the limitations that would prevent an application like this from existing, and how might they be overcome?


5 Comments

  • Kol Tregaskes |

    All I can say is yes please, Nathan. :-)

    Duplication is annoying. It’s a pain in Google Reader for sure but the likes of TweetMeme and Google Blogs do a fairly good job of this. To have this as an application would be good. To have it based on my Google Reader subscriptions would be better. GReader is were I get all my news but the duplicates are numerous. Having it work in Buzz too would be great but we need better filtering (among other major features) anyway.

  • AJ Kohn |

    I like the idea … a lot. For me personally, I’d want to rely on the people in my network far more than anything else. Perhaps I’m not the norm, but I follow people (particularly on FriendFeed) because they’re great filters of information.

    Pick the right people in a network – the ones with subject matter expertise – and you are delivered a very relevant stream of information. Yet, there is a lot of overlap between these human filters. FriendFeed does a decent job of this but it could be better.

    So … I like the idea but the most interesting thing would be the ability to add and subtract people from the network and – on the fly – see how the information would change. That type of instant visualization of value would be a better way to manage and optimize your network.

  • Michael Britt |

    I love the idea. I have an app that curates content and it’s definitely a challenge to get the best stuff and eliminate duplicates at the same time.

    Go for it and let me know when I can either it test it for you or buy it from you.