How to find the most common bi-grams with BigQuery?

问题

I want to find the most common bi-grams (pair of words) in my table. How can I do this with BigQuery?

回答1:

BigQuery now supports SPLIT():

SELECT word, nextword, COUNT(*) c 
FROM (
SELECT pos, title, word, LEAD(word) OVER(PARTITION BY created_utc,title ORDER BY pos) nextword FROM (
SELECT created_utc, title, word, pos FROM FLATTEN(
  (SELECT created_utc, title, word, POSITION(word) pos FROM
   (SELECT created_utc, title, SPLIT(title, ' ') word FROM [bigquery-samples:reddit.full])
  ), word)
))
WHERE nextword IS NOT null
GROUP EACH BY 1, 2
ORDER BY c DESC
LIMIT 100

回答2:

The standard SQL version:

SELECT word, nextword, COUNT(*) c FROM (
SELECT pos, title, word, LEAD(word) OVER(PARTITION BY created_utc,title ORDER BY pos) nextword FROM (
SELECT created_utc, title, word, pos FROM ( 
    SELECT created_utc, title, SPLIT(title, ' ') word FROM `bigquery-samples.reddit.full`), UNNEST(word) as word WITH OFFSET pos))
WHERE nextword IS NOT null
GROUP BY 1, 2
ORDER BY c DESC
LIMIT 100

When unnesting an ARRAY you can retrieve the position of that element using the following syntax:

UNNEST(word) as word WITH OFFSET pos

来源：https://stackoverflow.com/questions/24172994/how-to-find-the-most-common-bi-grams-with-bigquery