data-mining

How do I data mine a pile of text to get keywords by usage? ("Jacob Smith" or "fence") And is there a software to do this already? even semi-automatically, and if it can filter out simple words like "the", "and", "or", then I could get to the topics quicker. The general algorithm is going to go like this: - Obtain Text - Strip punctuation, special characters, etc. - Strip "simple" words - Split on Spaces - Loop Over Split Text - Add word to Array/HashTable/Etc if it doesn't exist; if it does, increment counter for that word The end result is a frequency count of all words in the text. You can

Possible Duplicate: How do I determine k when using k-means clustering? How can i choose the K initially, if i do not know about the data? Can someone help me in choosing the K. Thanks Navin The base idea is to evaluate cluster scoring on sample data, usally it is distance inside cluster and distance between clusters. The more this measure the better clustering, based on this mesure you can select best clustring paramters. One of metrics can be found here http://alias-i.com/lingpipe/docs/api/com/aliasi/cluster/ClusterScore.html Felix Kling Seriously, what do you want to know? Do you want us to

This question already has an answer here: How do I determine k when using k-means clustering? 17 answers I am attempting to apply k-means on a set of high-dimensional data points (about 50 dimensions) and was wondering if there are any implementations that find the optimal number of clusters. I remember reading somewhere that the way an algorithm generally does this is such that the inter-cluster distance is maximized and intra-cluster distance is minimized but I don't remember where I saw that. It would be great if someone can point me to any resources that discuss this. I am using SciPy for

I have noticed that when One Hot encoding is used on a particular data set (a matrix) and used as training data for learning algorithms, it gives significantly better results with respect to prediction accuracy, compared to using the original matrix itself as training data. How does this performance increase happen? Many learning algorithms either learn a single weight per feature, or they use distances between samples. The former is the case for linear models such as logistic regression, which are easy to explain. Suppose you have a dataset having only a single categorical feature