Efficient string similarity grouping

Question

Setting: I have data on people, and their parent\'s names, and I want to find siblings (people with identical parent names).

 pdata<-dat         


        

Answer 1

My suggestion is to use a data science approach to identify only similar (same cluster) names to compare using stringdist.

I have modified a little bit the code generating "parents_name" adding more variability in first and second names in a scenario close to reality.

num<-4e6
#Random length
random_l<-round(runif(num,min = 5, max=15),0)
#Random strings in the first and second name
parent_rand_first<-stringi::stri_rand_strings(num, random_l)
order<-sample(1:num, num, replace=F)
parent_rand_second<-parent_rand_first[order]
#Paste first and second name
parents_name<-paste(parent_rand_first," + ",parent_rand_second)
parents_name[1:10]

Here start the real analysis, first extract feature from the names such as global length, length of the first, length of the second one, numeber of vowels and consonansts in both first and second name (and any other of interest).

After that bind all these feature and clusterize the data.frame in a high number of clusters (eg. 1000)

features<-cbind(nchars,nchars_first,nchars_second,nvowels_first,nvowels_second,nconsonants_first,nconsonants_second)
n_clusters<-1000
clusters<-kmeans(features,centers = n_clusters)

Apply stringdistmatrix only inside each cluster (containing similar couple of names)

dist_matrix<-NULL
for(i in 1:n_clusters)
{
  cluster_i<-clusters$cluster==i

  parents_name<-as.character(parents_name[cluster_i])

  dist_matrix[[i]]<-stringdistmatrix(parents_name,parents_name,"lv")
}

In dist_matrix you have the distance beetwen each element in the cluster and you are able to assign the family_id using this distance.

To compute the distance in each cluster (in this example) the code takes approximately 1 sec (depending on the dimension of the cluster), in 15mins all the distances are computed.

WARNING: dist_matrix grow very fast, in your code is better if you will analyze it inside di for loop extracting famyli_id and then you can discard it.