Webnomena

So you built a new Web 2.0 like service. It gets some traction and people are crowding in. The site just published an open API. Before you know it, the system crumbles down under the weights of its own success. Sounds familiar?

If your API only exposes “read only” API i.e. an option to pull some of the data out of the system you’re only in half of a trouble. In a case where your API allows the “writing” option too, i.e. modifying system records in the database, now things gets really interesting. Example for read only: Technorati provide blog, bloggers and posts information to Internet Bots and badges. Twitter is an example for both read and write API. Bots built using Twitter API can get members’ status updates as well as automating posting status messages.

Btw, both of te above examples are for Data Producers that are working night and day to scale these open API support.

The problems are generally the same and so does the solutions: performance (throughput and latency)hardware sizing and cost, traffic pattern predictability, load balancing, throttling, caching, stateless web nodes, multi-casting, table partitions (having skilled=$$$ DBA for building high availability database), backup, API format (there are too many of them), message queuing, redundancy, recovery, security, quality of service (premium services), statistics, logging, error handling, monitoring, abuse protection, you name it.

Gnip is a new start-up founded by Eric Marcoullier that is working to address some of these common problems. Reading their blog shows how much thought and sweat is put into addressing some of these common scalability problems. They aim at addressing some the other pain point in the open API arena like consistent API and Identity discovery. Having a consistent/normalized entity ID across multiple web services can solve one of the biggest obstacle today for using WYSIWYG mashup tools like Popfly – but this is for another post:).

If you fit the profile description from the first paragraph reading more about them icould help, but this is not all this post is about.

Let’s assume that the “read” part is getting better due to service like Gnip (ping in reverse) same way that blogging platforms improved new posts indexing using ping service. Now, bots and mashup services don’t need to be “busy waiting” on the API. What about the “write”? What can be done to make this reusable and scalable?

I think that it all come to a new opportunity here to outsource the entire open API development and support, and to save a bundle. Here are some ways to save on this effort through consolidation.

• Reusing hardware through hosting solutions whether physical or virtual like Amazon EC2 or Google App Engine
• Reusing technologies implementation and integration like using memcached, terracotta and many more
• Reusing expertise – Database Administrator and Security experts
• Protocol and meta data standards
• Monitoring tools and technics

Saving: blood, sweat, tears, grief and reputation (in other words avoiding embarrassment).

Bottom line is, that in my opinion, Gnip take it a good distance forward but there is a room for another reusable, consistent and scalable layer between the Data Producer and Gnip.

What do you think?

Its time for putting the pieces together!

Using the combination of location aware social networks, recommendation and search engines a new service can contribute to successful, pleasant and safe business trip. Most of the pieces exists today and some implementations are closer than others. In this case again what that is available to the consumers is ahead of what that is available for the business.

The objective is getting the relevant information to someone on a business trip when it is most useful. The success factors are relevance, timing and quality.

Systems that are built around location detection technology should be capable of supporting both “off-location” setting and “at-the-location” configuration. Having the option to prepare for a trip ahead of time can help in providing a great travel experience once there and after. The richer the options to pre-configure the traveler preferences the easier to meet each of the above success factors.

The benefits using  location based system over manually entering zip code is twofold.

• The first is accuracy – hybrid systems today using multiple ways to locate a mobile device (and you) can do way better job than the zip code scheme. In emergency try to find somebody in an area size defined by a zip code. Another case is providing traffic or even crime information near you. On a less dramatic case maybe you have no rental in this trip and the only option is walking so few mile range could be useless.
• The second benefit is asynchronous communication – if the system knows where you are the service can push information to you and not wait till you pool the data (after entering the zip code).

It is not about pushing “quality” spam at you. It is about eliminating the noise.

Success factors

Relevance and timing

• I’m here now- what options are both possible and available for me?
• These are my needs and objectives – how can you help me here and now? 
• Share my location and find who’s near me 
• Here is my agenda – I can consume information during these times.
• I’m in one of these possible trip states: planning, soon to go, arrival, following agenda, still there, leave (back home or another destination), not my first visit here. There are probably more options.
• Communication preferences:
  • Immediacy – alert me: small pop-up windows for notification like the ones used by Instant Massaging client tools or Twhirl are a better option. And SMS.
  • RSS feeds,
  • Email is not so great but still an option for detailed information.
• Events -  what is happening around me with enough time to plan for.
• Maps and directions
• Emergency holler

Quality

• Filter by my preferences – exclude or include words/tags something like filtrbox
• Filter by others – recommendations, rating, bookmarking
• Add my own recommendations and annotations
• Stared(bookmark)/rated information
• No fluff – give me the bottom line first. I can’t read a page long information in the first scan.

 

Additional ideas

Things to do for a trip preparation

• Itinerary – accommodation, sport, weather (gallery?)  food, entertainment, shopping.
• Agenda – blackout periods (meetings)
• Using past information or the “wisdom of the crowed” – other people that went there.
• Arrival – traffic, news (what local sports team won/lost)
• Lunch Break – allow some information flow – alerts
• End of business day – shuttle, near by connections (social), events.
• Next day (few days trip to the same spot case) – show me something new please.
• Next destination – something changed that I should be aware of?
• Recurrent trip (save settings  – rating, more of this)
• I can go on here…

Social/sharing

• Other travelers shares their favorites bits of information.

 

Are we there yet? …. Are we there yet?

I think that Microsoft, Powerset deal is a great step and accomplishment for both companies.

For Microsoft this acquisition could help to offer something unique in the search arena. They are trying hard to fight their declining market share. This may be the way around Google.

Yet,  it is even better deal for Powerset! In my opinion this is the right exit strategy for someone building, yet another, search engine today. Starting “cheap” by crawling Wikipedia only, apparently, was a very smart move. It is spam free and a data source rich with information. I’m sure that there will be an integration effort due to the different way that Powerset index text, yet, now they will be able to access a larger data source and reuse some of the already built infrastructure. This is aligned with merger as an exist strategy.

Ironically, there is another gain for Microsoft . They will be getting through the Powerset deal some of the former Yahoo’s search expert (like Tim Converse).  I wonder if this recent move by Microsoft will change something in the Yahoo, Microsoft on-going merger deal discussions.

If you want to read more about Powerset and the deal read these two great posts by VentureBeat :

• Microsoft to buy semantic search engine Powerset for $100M plus
• Powerset’s search technology scoop, may scare Google