Lifted from here:

It appears that within the past 2-3 days the popular social book-marking site del.icio.us has started blocking the major search engine spiders from crawling their site.  This isn’t a simple robots.txt exclusion, but rather a 404 response that is now being served based on the requesting User-Agent.

While I was doing some Photoshop work for a site of mine tonight I needed to grab some custom shapes to use to make some icons.  I recalled having bookmarked a good resource for custom shapes in del.icio.us, but after searching my bookmarks using my del.icio.us add-in for Firefix, I couldn’t find it, so I pulled up my browser and went to my profile page on del.icio.us to do a search.  To my surprise, I was greeted with this:

After confirming I hadn’t mistyped the URL, I checked out the del.icio.us homepage and found that all was fine there.  However, upon trying to perform a search, I was confronted with the same 404 error, and received the same response when trying to navigate to any page other than the homepage.

At this point I was thinking that there might have been some server issues going on with del.icio.us, but that didn’t line up with my Firefox add-in still showing my bookmarks.  I then noticed that my User-Agent switcher add-in was active (not sending the default User-Agent header), and remembered that I had set it to switch my User-Agent to Googlebot earlier because I was checking another site earlier today to see if it was cloaking (it was).

I reset the User-Agent switcher so it was sending my normal User-Agent header and tried accessing my del.icio.us page again and I was surprised to see that it was no longer responding with a 404 error.  Puzzled by this, I took a look at del.icio.us’ robots.txt and found that it was disallowing Googlebot, Slurp, Teoma, and msnbot for the following:

Disallow: /inbox
Disallow: /subscriptions
Disallow: /network
Disallow: /search
Disallow: /post
Disallow: /login
Disallow: /rss

Seeing that the robots.txt was blocking these search engine spiders, I tried accessing del.icio.us with my User-Agent switcher set to each of the disallowed User-Agents and received the same 404 response for each one.  I thought that there might have been some obscure issue with the add-in that was leading to this behaviour, so I popped open Fiddler, a nifty HTTP debugging proxy that I use to sniff HTTP headers.  Fiddler has a convenient feature that allows you to create HTTP requests manually, so I created a simple set of request headers and made HEAD and GET requests using the different User-Agents listed in the robots.txt.  I received the same responses as before.

My interest was definitely piqued at this point.  I ran a site command against del.icio.us in Google restricted to the past 24 hours and found results as fresh as 15 hours old.

Running a normal site command on del.icio.us revealed numerous results that Google had a cached version of, many of which were as fresh as only three days ago.

This evidence seems to be indicating that del.icio.us has recently started blocking the major search engine spiders from crawling their site, by way of the requesting User-Agent.  Given the recent crawl dates and cache dates, it looks like this started happening within the past 2-3 days.  This raises some questions as to the intentions of del.icio.us, and perhaps Yahoo!  With Yahoo! recently integrating del.icio.us bookmarks into its search results this could an attempt to enhance the effectiveness of that new feature by preventing competing search engines from indexing content from del.icio.us.  While Yahoo!’s Slurp bot is also blocked, it’s unlikely that Yahoo! would need to crawl the content of one of its own sites, as Yahoo! actually owns del.icio.us.

Google Lays Groundwork for TV Scatter Sales Sortie

Advertises for TV Ad Chief, Negotiates to Peddle Upfront Spillover

By Claire Atkinson

Published: February 26, 2007

NEW YORK (AdAge.com) — Google is looking for a head of national TV sales. The recruitment ad, posted on its website, says the winning candidate

Google is maneuvering aggressively as it seeks to find an entry point into the $65 billion U.S. TV-ad-sales market.

will build a “world-class national TV-advertising-sales team and lead the effort to both sell television solutions and shape a next-generation advertising platform.”

TV ad market
The ad goes on to say the search giant is looking for a candidate with “extensive senior-level relationships with advertisers, both with agencies and clients directly.” What it doesn’t say, at least not in so many words, is that Google is desperately trying to wedge itself into the $65 billion U.S. TV-ad-sales market.

Those close to some of the talks the search giant has had with TV executives said Google’s first priority is to persuade broadcast networks to let it sell their 30-second spots. To that end Google has had talks with the broadcast networks, cable programmers and several cable operators, including Comcast and Cablevision.

Start with scatter
“They want to start with scatter first and then move to local,” said one executive familiar with Google’s plans. Scatter refers to TV ad inventory that is not sold to marketers ahead of time, during what is known as the upfront, but is negotiated and bought each quarter. The price of scatter inventory rises and falls in relation to whether there is a lot or a little available, making it a market perfect for the auction-style ad system at which Google excels. The search giant has already attempted to apply its sales tools to radio and print-ad sales, and TV executives have long been wary of allowing Google to get a toehold in their business.

TV executives argue that they don’t want to see their products commoditized, and that auctions could depress pricing. Google argues the opposite, claiming it will bring new customers to the market, possibly increasing prices.

“We’ve acknowledged publicly we’re interested in pursuing TV,” a Google spokesman said. “But we haven’t disclosed any specific product direction. It’s an area we’re looking into.”

Worldwide talks
“Television remains the single most important source of information and entertainment for billions of people around the world,” reads one of the Google classified ads for a TV software engineer, underscoring just how important it sees the TV-ad-sales business. Its ambitions are by no means limited to the U.S. Google is also talking to the BBC about offering the broadcaster’s video content on a worldwide basis.

Google already has hired two senior executives from NBC — Michael Steib, who was general manager of strategic ventures at NBC Universal and is believed to be heading the TV auction business, and Jordan Hoffner, who was tapped to help Google become better at partnering on content.

Of course, there’s already a player in the online-TV-auction business: eBay, whose Media Exchange service is backed by a number of major advertisers, including Microsoft, Intel and Lexus. Media Exchange was tested in late January but has yet to win a commitment from a TV entity to sell live inventory.

Architecture of TV-ad-sales business
But Google’s TV ambitions go beyond auctioning 30-second spots. Google also is looking to compete with Microsoft by offering the architecture of such ad-sales ventures. Last week, the Television Bureau of Advertising unveiled plans to automate the archaic business of buying local airtime. Google is a contender to build that platform.

More significantly, Google is pitching a potential venture with Discovery Communications, now headed by former NBC cable chief David Zaslav. While details about that venture are scant, one executive close to the project said Discovery was “open” to a content partnership. But another executive said Discovery had to be careful about how such a joint venture would be received by the cable operators.

Google has many hurdles to overcome if it is to succeed in the TV business. First and foremost, it must resolve the copyright issues it faces with its newest unit, YouTube. Content companies appear to have a newfound confidence in the value of their video clips, something that executives say derailed talks between YouTube and CBS.

According to one analyst, Google is willing to offer somewhere between $70 million and $140 million to compensate media companies for copyright violations connected with YouTube, but CBS and others feel that offer is too low. The analyst said Wall Street is waiting on Google’s first deal with an entertainment conglomerate to decide what value to ascribe to traditional TV content as it migrates to the web. That may be a long wait.

“No one wants to be the one that lets Google into the henhouse,” said one network executive.

Apple Macs standard browser Safari could be making huge gains or maybe its holding steady in the browser wars; it all depends on whose stats you use. The same can be said of Mozilla Firefox, while Internet Explorer is losing ground in the US but seems to be holding its own world-wide.

The conflicting figures come courtesy of two web metrics  firms, Net Applications and OneStat. Net Applications’ figures come from its live stats customers and sample over 40,000 URLs. OneStat uses a daily sample of 2 million visitors spread across 100 countries to come up with its numbers.

Net Applications’ numbers cover the US only and paint rosy pictures for both Firefox and Safari. Firefox has seen its market share bounce from 9.5 percent in January 2006 to 13.67 percent a year later, although the latter figure is down a bit from December 2006’s 14.0 percent.

Safari has seen even stronger growth, according to Net Applications. In January 2006, 3.0 percent of all web surfers were using Safari. A year later, that figure had grown by over 56 percent, to 4.70 percent.

Over the same time period, Microsofts Internet explorer is slowly falling despite releasing its seventh incarnation this year

. Starting at of 85.31 percent at the beginning of 2006,Internet explorer usage dipped to 83.56 percent in July and ended December at 79.64 percent, recovering modestly to 79.75 percent for January 2007.

OneStat’s figures tell a different story, both in the US and around the world. For the US, OneStat reports a much more modest decline for Internet Explorer, which dipped to 78.13 percent last month from 80.91 percent a year ago. Firefox saw a significant gain of 14.3 percent in the US, while Safari saw a much more modest rise of 3.7 percent, from 3.55 percent in January 2006 to 3.68 last month.

In contrast, the worldwide browser market share figures are much more static. Internet Explorer usage has remained almost static over the past year, dropping a mere 0.01 percentage point over the last year. Firefox has shown modest growth during the past year. The open-source browser was the choice of 11.69 percent of web surfers in January 2007 compared to 11.23 percent the year previous.

The biggest surprise in OneStat’s worldwide market share figures comes courtesy of Safari. Apple’s web browser, the choice of 2.02 percent of web surfers tracked by the firm in May 2006, dropped to 1.64 percent at the beginning of 2007. Overall, that’s a 12.7 percent decline from January 2006’s 1.88 percent figure.

There’s a big difference, especially for Safari. While Apple’s home-rolled web browser is doing very well in the US, it’s slipping around the world. That can be explained in part by the rapid growth of the PC market in countries where Apple isn’t a big player, most significantly, India and China.

Firefox continues to grow, no matter what the venue. In the US, Firefox is definitely making gains at the expense of Internet Explorer. Worldwide, some of those more modest gains appear to have come at the expense of Opera and Netscape. Firefox jumped 0.46 percentage points between January 2006 and 2007, while Opera dropped from 0.77 percent to 0.58 and Netscape dropped 0.03 percentage points to 0.13 percent. Meanwhile, IE’s 0.01 percentage point share dip was statistically insignificant.

Curiously, the release of Internet Explorer 7 last summer didn’t provide any additional momentum for Microsoft. IE7’s browser share soared from 3.18 percent in October 2006 to 25.01 percent last month, but all of that gain came at the expense of older versions of the browser, especially IE6, which dipped from 77.17 percent to 54.04 percent, according to Net Applications.

If there is any clear takeaway from the morass of statistics we just waded through, it is that competition between web browsers is alive and well. In the US and some Western European countries, users are increasingly willing to ditch Internet Explorer. In the US, some of those old IE users are apparently showing up on a different platform as well as a different browser.

Thats all for this week stats fans.

Have a great weekend.  I really am off to the pub this time.

I don’t know if you have ever heard about netvibes; don’t worry if you haven’t heard about it. Despite the fact it has four million subscribers plenty are still in the dark about it. And I was one of them until Christmas 2006, and that’s saying something, as I know pretty much everything about anything.

So what is Netvibes?

Netvibes in their own words is:

A custom made web 2.0 home page solution

* Browse, modify, and import your RSS feeds with our integrated RSS/ATOM feedreader. You can easily import an OPML file as well.
* To import, download and listen podcasts without any additional software
* To check your mail on one or many gmail account, to stick webnotes, weather and many more to come!

I have no idea how Netvibes expect to attract your average web user with all this RSS/ATOM /OPML nonsense. But I will try and explain it; basically Netvibes helps you read the Internet quicker. Obviously you can only read as quick as you can read, but this site cuts down the time on actually visiting websites, because Netvibes visits the sites for you and then displays just the news headlines from the sites you tell it.

It can check your email/myspace/bebo/ebay/flickr/traffic and travel and countless other sites all at the same time and then present the information for you in easy to digest chunks. So you can scan read the headlines quickly and see if there is anything of interest to you. It can also listen to Podcasts, but I have no idea why you would want to do that from within your web browser.

You can also go one step further and have tabs, so you can have multiple pages within the one homepage. For example you may enjoy reading about your favourite football team, or the soaps. Well you could have all your info for each topic on separate tabs.

Here’s an example of a tab on my homepage for the greatest football team in the world. Click on it for a big version

So make sure you do yourself a favour and check netvibes out. It WILL improve your life.