adding language filter to twitter popularhashtags - scala

问题

I am new to Spark and Scala. I ran the Spark streaming job-twitter popular hash tags.

I added a filter for some words and was able to filter out tweets :

val filter = Array("spark", "Big Data")
val stream = TwitterUtils.createStream(ssc, None, filter) 

Likewise I want to add a language filter so that only English tweets are streamed. Twitter4j has Track() and Locations. Does it have a language filter? If so, how does it work in Scala?

回答1:

I'm repeating what's already been said in this Spark thread.

Spark uses Twitter4J for the feed. Twitter4J as of version 3.0.6 has getLang (doc) which allows you to:

.filter(_.getLang == "en")

which can be used against the DStream of twitter4j.Status.

But unfortunately Spark uses an older version of Twitter4J (doc) which doesn't have getLang.

Either upgrade Twitter4J within Spark to 3.0.6, wait for Spark to upgrade Twitter4J, or an altogether different approach.

来源：https://stackoverflow.com/questions/31014369/adding-language-filter-to-twitter-popularhashtags-scala