Over the weekend I discovered that the National Library of New Zealand had been trawling the net and bulk downloading harvesting,

every publicly accessible website that falls under the nz country code, as well as certain other websites that are owned by New Zealanders or legally considered New Zealand publications.

This is because

The internet is always changing, and uses a myriad of technologies, so it is impossible to make a perfect copy. Despite this, we are hoping to harvest 100 million URLs during October 2008, giving us a snapshot of the internet at that time. link

I also discovered that if I took the usual measure, to avoid being crawled and archived, by using a robots.txt file inside my website, the National Library would ignore my wishes right to object to outside crawling of my site and take it anyway “to enhance the likelihood of our being able to harvest as full a snapshot of the .nz domain as possible“.

That’s OK though because I know that the National Library has my best interests at heart and knows better than I do about the archiving of my website. They know that although I might consider my few pages of rants, raves and links to be temporary, my grandchildren will want to benefit from the crazy thoughts that I had way back in October 2008.

But what about the people who have developed applications which generate dynamic webpages based on entry data – I’ve seen countless webpages like this. The astute developer will have done the right thing and placed an anti crawl robots.txt file in their website to avoid crawlers doing huge damage to their server traffic. I know you can opt in to opt out of being crawled, but you have to have seen it at the right time, followed up the user agents, worked out what was happening and put in your request. I didn’t notice until it was a bit life. Hu hum, life got in my way.

I’m in two minds about this idea of the National Library archiving everything in the .nz webosphere. As much as anyone I want the stories collected and collated for the future, couldn’t they respect respect some of our wishes for autonomy over our material? I have no robots.txt file on any of my websites at the moment – although I do want the right as a relatively savvy citizen to have the option. As it happens, it appears the biggest internet archive site in the world agrees with me.

EDIT: Aha! A rewrite rule with a Rickroll. Nice. Very. Nice.

No related posts.

On the other hand the same teacher instinct was apalled by how little progress we have made teaching people how to search in a way that will return the information they are seeking. Particularly using appropriate key words to refine our searches.

I have been constructing a similar post in my head. Beware the river of consciousness that follows.

Recent work that I have done with teachers has lead me to wonder about how we seem to have hit skills in a roundabout sort of way. I’m often asked to work to help teachers get skills in using ICTs to present kid’s work or to show off the finished product – PowerPoint, Photo Story, iMovie or Movie Maker. Or teachers want a bunch of websites to use in their classroom programme.

If I suggest spending time looking at search techniques or alternatives to Google, they say that they know how to use Google. And then, I watch as they type a web address into the Google’s search bar.

Just over a year ago I wrote a post about the core things that computer users of any age need to know. Turning the beast on, where and how to use right click context menus and saving and retreiving files. I also said that we need to have some internet skills. And one of the most important of those skills is to understand the difference between an address bar and the search box.

As I see it we’ve taken a whole language approach to the concept of information and knowledge literacy and while we bemoan the fact that many kids can’t sort out the fact from fiction we can’t see that at the moment many adults don’t know how to tell either.

A few weeks ago my son’s science teacher was horrified that one of her students might use the Wikipedia for information. “Don’t let him use the Wikipedia,” she told me, “It can be changed!” We were in the middle of the mid-year interviews and I was asking why she was accepting material from him that contained unreferenced facts and information.

I am regularly informed that the Wikipedia is an unsafe place to look for information. Often times the person who kindly tells me this has just discovered this at a professional development course. My response depends on the environment but I have been known to tell people that, “Yes, I know, they have even let me edit Wikipedia articles.”

The Wikipedia is no better or no worse than any other website – if you understand its purpose and its place in the ecology of the internet and information. It’s a great place to start your research but a bad place to end.

In her post Dorothy talks about the way that Google has gently removed us from the reality that poor spelling and loose thinking won’t get us what we want.

I guess what I was seeing at the Googleplex was a testimony to the failure of this approach. When Google came along with their user-friendly search engine they not only made it simple to do an advanced search (just click the Advanced Search button!) but they seem to make intuitive sense of our paltry efforts. Whether it is poor spelling and grammar or people who simply type in a question, Google seems to be able to to supply a list of intelligent results.

I think we have to go one step further back – yet again – and look at search engine design and search engine purpose. It’s not always about helping you find the information that you want, but about directing you to what it wants you to see. We need to step away from the Google (and Google Advanced) is good / Wikipedia is bad mode and take a look at the myriads of other options that are out there.

Related posts:

1. Google Maps Mashup – NZ!
2. Searching
3. Linkeracy

and …

Try WikiMindMap for a visual representation of a Wikipedia search:

WikiMindMap is a tool to browse easily and efficiently in Wiki content, inspired by the mindmap technique. Wiki pages in large public wiki’s, such as wikipedia, have become rich and complex documents. Thus, it is not allways (sic) straight forward to find the information you are really looking for. This tool aims to support users to get a good structured and easy understandable overview of the topic you are looking for.

It’s a useful way to start a search or to look at related concepts.

Related posts:

1. Linkeracy 18 January 2009
2. Linkeracy – 21st June

The report looked at the websites returned for 1,394 popular keywords searches found via Google, Yahoo, MSN, AOL and Ask.

The results returned for each search term were then analysed using the Site Advisor security tool. Once installed this piece of software warns users when they browse websites known to be dangerous.

The most benign of the pages that Site Advisor flags up try to change browser settings (to redirect people to ad sites) and the most dangerous deluge users with spam or bundle adware and spyware in with downloads.

In one case signing up with one site led to a test e-mail address getting more than 300 spam messages per week.

So what were the most dangerous search terms?

Ouch.

The moral of the story? Be vigilant! Safe surfing behaviour. Keep your antivirus and antispyware tools up to date. You don’t get anything for nothing – especially on the internet.

Related posts:

1. I Still Haven’t Found What I’m Looking For