Collecta Blog – Real-time Streaming Search

Collecta’s new home page showcases the best pictures, stories, updates and comments from CES and other trending topics as they unfold

San Francisco, Calif. – January 7, 2010 – Collecta, the streaming real-time search company that has been accelerating the pace of Internet search, is leveraging its network of more than 10 million sites to reveal hot stories as they develop across the web. The site’s new ‘Hot Now’ section features the most representative stories, photos, and comments for each hot topic — providing a unique multi-dimensional snapshot on the pulse of the web at any point in time. For example, with the start of CES 2010, Collecta’s ‘Hot Now’ shows the hottest stories, launches, and buzz from Las Vegas.

At a glance, Collecta visitors can discover the latest breaking events, news, and other trends on the web — from the floor of CES to a cold blast gripping the east coast or Google’s Nexus One. For each topic, Collecta includes a sampling of articles, images, Twitter updates, and comments to give people a holistic view of what’s being said and shown across the web. The content updates as new relevant information and images come in — giving the fastest, most up-to-date view of topics in real time.

Collecta’s editorial process has been designed to ensure that the hottest topics and trends are showcased, as well as the most representative and useful pieces of media for each topic.

“Previously, the Collecta home page displayed a list of the hottest keywords found in Collecta searches. Our users responded really well to that feature and often launched their own searches to discover what’s happening related to any given keyword,” said Gerry Campbell, CEO of Collecta. “We decided to liven things up by adding sample stories, comments, and photos up front for each topic. As a result, visitors don’t just see a handful of terms. They see the most useful pieces of content right away, and instantly have a multi-faceted view of what’s going on at any given moment.”

In addition, technology fans can turn to Collecta to track all the latest happenings from the floors of CES in real time. By running an extended search on CES on Collecta, people can literally watch events and news unfold. Collecta’s XMPP-based platform shoots information out the instant it has been published, so users can watch a live results stream without having to hit refresh or restart their query. Collecta encompasses a broad content network of both social media and established news sources, offering a single place to view product reviews from bloggers and journalists, Tweets and photos from exhibitors and attendees, articles, comments and more — all in real time.

About Collecta
Collecta represents a new way to experience search, in real time. Collecta is the Web’s most powerful real-time search engine, posting matching stories, blogs, photos, and comments as they happen. By aggregating content in real time, Collecta offers a new and more comprehensive view of what’s going on in the world right now. For more information, visit www.collecta.com.

We hope by now you’ve had a chance to see our real-time results in action in the Collecta client at Collecta.com. But we also understand that life is about convenience, and that a lot of people use RSS readers to keep track of their daily news, blogs and breaking stories too. Feed readers may not be real-time (yet) but we understand the convenience, so we’re here for you – the instructions below explain how to use the Collecta REST API to get results from your favorite Collecta searches, right to your feed reader.

There are really only two basic steps required:

1) Create your query using the Collecta REST API. You can test your query in any browser to make sure it is working properly.

2) Add the URL for your query to your favorite feed reader, and start receiving results.

That’s it.

The Collecta REST API is really very simple, it’s just some parameters on the URL, but since not everyone is used to working with an API, we have provided the following simple instructions to get you started in writing your own queries.

Creating search queries with the Collecta REST API.

Since the Collecta API is just a specifally formatted URL, all your queries will be in the basic format of http://api.collecta.com/search?q= followed by your query terms.

As a simple example, lately there have been some terrific snowfalls in the eastern US and in Britain. So to see what people are saying about snow I could use this query: http://api.collecta.com/search?q=snow

But lets say you wanted to only see photos, not the updates and blog comments and other stuff.  Since the query language is the same as that used on Collecta.com, you can use the category field. So the Collecta query would be: snow category:photo -  and the API link would be: http://api.collecta.com/search?q=snow category:photo

Go ahead and click the link above to see the results you would get for your query. They may not look so great in the browser, but most feed readers will display the images nicely, as you can see below.

Here is an example from Google Reader:

Another example query using categories: since it’s football season, to limit to just stories and blog posts about the NFL the Collecta query would be: NFL category:story So the API link would be http://api.collecta.com/search?q=NFL category:story

Remember – to get results for all categories, just run use the same query without specifying a category field.

Other common search operators.
Finally, a quick run through of some other common search operators, most of which should already be familiar from other search engines. Let’s start with ‘AND’ which the easiest because it is implied.  For example – http://api.collecta.com/search?q=Florida State will search for results containing both the word Florida and the word State. So you never need to type “AND” as part of your query.

Collecta also supports the ‘OR’ operator: http://api.collecta.com/search?q= cats OR dogs
This will return results with either word.  Please note he capital OR – you must capitalize OR for it to be treated as a search operator.

Collecta also supports queries by phrase.  Phrases are used when you want to get an exact match on a set of words together in a specific arrangement.  Using a phrase can be helpful when AND or OR is returning extra information and you want to narrow your query results.  Names are often good examples of this.  For example you would want to use http://api.collecta.com/search?q=”Pat White” instead of  http://api.collecta.com/search?q=Pat White This will insure that you get the exact phrase you are looking for, as in “Pat White of the Miami Dolphins” but not just results with both the words Pat and White as in “My grandpa Pat has white hair.”

And of course you can use the previously operators together for example http://api.collecta.com/search?q=”Pat White” OR “Miami Dolphins”

There are a few other limitations to the Collecta API.  One being that right now some special characters will not work or will need to be URL escaped.  An exception to the rule is the # sign, which you should just leave out altogether.  So if you are looking for items tagged with a certain hash tag, you can leave out the # and just search for the term, and Collecta will also find the hash tags for that term.

Did you notice something new on Collecta today? What you are seeing is our new real-time display of the latest hot trends from around the web – we call it Hot Now.

If you’ve used Collecta before you may have noticed the Hot Now topics displaying as an updating list in the Collecta client. But we decided to make things a little more fun and show sample results for each of the current trends in Hot Now.

We include samples from each of our content categories for images, stories, updates and comments from across the web. The sample items for each topic will change from time to time as new relevant information comes in – giving you the fastest, most up-to-date view of the topics in real time.

So now Collecta is not just the place for real time search, it’s the place to track the pulse of the web. Check back from time to time to see what’s Hot Now, and discover breaking news and events – in real time.

Last week we posted a very specific case where the residents of Charleston SC used Collecta to work their way around town during a big storm. This week we are going to take a step back and give you a broad view of ways real-time search can be used. Watch below as CTO Jack Moffitt explains the what and why of real-time search in five minutes.

CTO Jack Moffitt with “New Things Now: Intro to Real-time Search”

Want to share how you use Collecta? Email us at help at collecta dot com.

Read Write Web has invited people to vote for their favorite Web Products of 2009. We are excited to be on the list and are asking all you to help make us #1.

Here is the link to the voting.

Thanks!
Team Collecta

We’re never satisfied with just being the fastest. We always look for ways to make Collecta even faster. With that in mind we set out looking for ways to not only scale but to get faster as we did. Our quest led us to Basho’s Riak Search, a new kind of search indexing system.

CTO Jack Moffitt explains, “Indexing at Collecta is there to provide historical context; it is only part of our search technology. Our real-time technology is powered by streaming (using XMPP, not indices). As far as I’m aware, we are the only real-time company that is not based on indexing systems, and that fact probably explains why our search results appear in fractions of a second after, not minutes after, content is published.”

He continues, “Our users would not be impressed if the results page showed them nothing until the next time someone mentioned their query term, so we store a historical archive and index it for queries. Unfortunately, the traditional indexing systems fall well short of our needs.”

We are excited to see what this new opportunity brings not only us but all developers and administrators.

Read Jack’s entire in-depth blog post or Basho’s press release.

One of the questions we get here at Collecta from those who are new to real-time search is “What would I use it for?” A fair question, for those used to static web search, the importance of real-time information may not always be clear – until you need it. Which is why we have decided to present some case studies from real Collecta users. We could easily tell you all the ways you could use Collecta – to track your favorite sports team during the game, follow breaking news or get the latest scoop on your favorite celebrities, stay ahead of the curve in business by tracking the market buzz – but we feel like real life examples speak most clearly to the importance of real time, and how the real-time web combined with social media provides information that you can’t always get from traditional media sources.

On Dec 2nd, 2009 the city of Charleston SC had some severe weather causing road closures, flooding and damages. If you’ve ever visited Charleston, you know that the main city is on a peninsula, very close to sea level, so flooding potentially shuts down large parts of the city. Quite the bummer if you have somewhere to go. Suddenly knowing which streets are flooded and which streets are passable – right now – takes on a whole new importance.


Photo found via Collecta from The Post and Courier – Tony Bartelme/Staff

The Digitel Charleston started asking people to use a specific hash tag to post their information to Twitter, and to follow it on Collecta.com and used this to feed important real time information about a sever storm to its citizens helping them keep track of the storm, and to help users re-route around the most flooded areas when travel was a must.

By aggregating information from Twitter, local weather blogs, and traditional media sources from the rest of the web all in one place, Collecta provided a one-stop place to get the information Charleston residents needed about the storm, in real time. As a further example, below Christopher Zorn, a Collecta developer who lives in Charleston, used this real time feed from Collecta to guide his family through the storm.

Have you used Collecta to make your life easier, help you have more fun, or follow a breaking event that matters to you? Let us know by email or @collectadotcom on Twitter.

Six weeks ago we began our second API challenge. This time we challenged developers to build a creative, user-facing application on our XMPP API. There were some great entries into the contest and in the end it came down to two finalists:

David Bello’s CollectaAlerts lets you follow and receive realtime updates directly to your desktop and mobile phone.
Alvaro Videla’s Dashboard Widget for Mac OS X that allows you to search for terms using the Collecta XMPP API.
**Please follow the links and try them for your self.

But just like in the Highlander, there can be only one. Collecta is proud to announce the winner of its XMPP Real-Time Challenge…

Alvaro Videla! Your $2,000 Amazon gift certificate will be on it’s way shortly.

We thank each and every developer that took the time to use the XMPP API and submit to our contest, and we hope to see continuing use of the XMPP API in the future. Stay tuned for more fun announcements and
challenges from Collecta.