A Golden Needle in a Haystack

Wednesday, 30 January 2013   |   Creative Thinking

4 cluesThe internet really is an amazing place.  It is phenomenally huge, and unlike the real world, if you have access you may reach any part of it instantly and with virtually no effort or cost.  It is also growing constantly and more importantly, evolving.  We no longer have just static pages but rich interactive experiences and the ability to construct services that can manipulate and connect data in virtually any way we can think of.  Searching for a needle in a haystack is easy compared to trawling the interwebs.

Now this is a problem that many people are constantly trying to tackle.  We started off with directories.  Soon outgrew them and moved on to search engines.  Now we have a lot of links shared by way of social media (Facebook, Twitter and the like).  The problem is, each of them has it's weaknesses and so a lot of potentially valuable work is not connected with all the people that would really like to benefit from it.

Directories were quite limiting.  New websites were soon popping up quicker than people could add them.  Biases crept in and you were limited in the way you could interact with it to find sites.

Search engines improved on this as the adding of sites was automated which solved the speed problem and also the human bias problem.  It also meant you could index pages on many different characteristics to make search more natural.  But now the problem was gaming the system.  If it is a process, and people know the process then it is natural that some of them will use that knowledge to their advantage.  Search engines are constantly trying to stay one step ahead of the spammers.

Another problem which search engines face, which is also one that social media faces is that of self reinforcing popularity.  Sites and content that ranks well is going to continue to rank well as a result.  Studies have shown that, when doing a search, about 60% of people will click on the first link, this goes down to 20% for the second link and rapidly down after that.  If there is an even spread of people that will share or link to what they find then by virtue of being in the number one spot, you will gain more publicity which will reinforce your number one position and so on.  Newer material that is potentially more relevant can find it quite hard to be seen amongst the older material.

What this all means is that while there is a great deal of truly awesome stuff out there, the reality is, a lot of it is potentially not being found.

A lot of people like to sit back and think that is all well and good but it's not a problem as someone will come along sooner or later and find a better way to do it.  And then all will be ok.

I think that this is a very dangerous path to take.  Especially as, more and more, people are inclined to alter what is returned to us for various reasons, mostly financial but there are other nefarious reasons.

We as creative types need to be proactive in how we trawl through the internet and find those golden needles.  What techniques have you found useful for locating those hard to find but very valuable resources out there on the net?


You're on your way to becoming a Highly Creative Person.