Free Large datasets to experiment with Hadoop
Do you know any large dataset to experiment with Hadoop which is free/low cost? Any pointers/links related are appreciated. Preference: At least one GB of data. Production log data of webserver. Few of them which I found so far: Wikipedia dump http://wiki.freebase.com/wiki/Data_dumps http://aws.amazon.com/publicdatasets/ Also can we run our own crawler to gather data from sites e.g. Wikipedia? Any pointers on how to do this is appreciated as well. Few points about your question regarding crawling and wikipedia. You have linked to the wikipedia data dumps and you can use the Cloud9 project from