cluster-analysis

This question already has an answer here: Fast computation of pairs with least hamming distance 1 answer Finding Minimum hamming distance of a set of strings in python 4 answers I have a database with n strings (n > 1 million), each string has 100 chars, each char is either a , b , c or d . I would like to find the closest strings for each one , closest defines as having the smallest hamming distance . I would like to find the k-nearest strings for each one (k < 5). Example N = 5 i1 = aacbdbbb i2 = abcbdbbb i3 = bbcadabd i4 = bbcadabb HammingDistance(i1,i2) = 1 HammingDistance(i1,i3) = 5

I'm clustering a sample of about 100 records (unlabelled) and trying to use grid_search to evaluate the clustering algorithm with various hyperparameters. I'm scoring using silhouette_score which works fine. My problem here is that I don't need to use the cross-validation aspect of the GridSearchCV / RandomizedSearchCV , but I can't find a simple GridSearch / RandomizedSearch . I can write my own but the ParameterSampler and ParameterGrid objects are very useful. My next step will be to subclass BaseSearchCV and implement my own _fit() method, but thought it was worth asking is there a simpler

So, I've been mulling over these concepts for some time, and my understanding is very basic. Information retrieval seems to be a topic seldom covered in the wild... My questions stem from the process of clustering documents. Let's say I start off with a collection of documents containing only interesting words. What is the first step here? Parse the words from each document and create a giant 'bag-of-words' type model? Do I then proceed to create vectors of word counts for each document? How do I compare these documents using something like the K-means clustering? Try Tf-idf for starters. If

I am using Mallet api to extract topic from twitter data and I have already extracted topics which are seems good topic. But I am facing problem to estimating K. For example I fixed K value from 10 to 100. So, I have taken different number of topics from the data. But, now I would like to estimate which K is best. There are some algorithm I know as Perplexity Empirical likelihood Marginal likelihood (Harmonic mean method) Silhouette I found a method model.estimate() which may be used to estimate with different value of K. But I am not getting any idea to show the value of K is best for the

I have five points and I need to create dendrogram from these. The function 'dendrogram' can be used to find the ordering of these points as shown below. However, I do not want to use dendrogram as it is slow and result in error for large number of points (I asked this question here Python alternate way to find dendrogram ). Can someone points me how to convert the 'linkage' output (Z) to the "dendrogram(Z)['ivl']" value. >>> from hcluster import pdist, linkage, dendrogram >>> import numpy >>> from numpy.random import rand >>> x = rand(5,3) >>> Y = pdist(x) >>> Z = linkage(Y) >>> Z array([[ 1.

I have a list of points P=[p1,...pN] where pi=(latitudeI,longitudeI). Using Python 3, I would like to find a smallest set of clusters (disjoint subsets of P) such that every member of a cluster is within 20km of every other member in the cluster. Distance between two points is computed using the Vincenty method . To make this a little more concrete, suppose I have a set of points such as from numpy import * points = array([[33. , 41. ], [33.9693, 41.3923], [33.6074, 41.277 ], [34.4823, 41.919 ], [34.3702, 41.1424], [34.3931, 41.078 ], [34.2377, 41.0576], [34.2395, 41.0211], [34.4443, 41.3499],