BigQuery UDF memory exceeded error on multiple rows but works fine on single row

前端 未结 3 946
小蘑菇
小蘑菇 2020-12-20 03:13

I\'m writing a UDF to process Google Analytics data, and getting the \"UDF out of memory\" error message when I try to process multiple rows. I downloaded the raw data and f

3条回答
  •  余生分开走
    2020-12-20 03:42

    A UDF will fail on anything but very small datasets if it has a lot of if/then levels, such as:
    if () {
    .... if() {
    .........if () {
    etc

    We had to track down and remove the deepest if/then statement.

    But, that is not enough. In addition, when you pass the data into the UDF run a "GROUP EACH BY" on all the variables. This will force BQ to send the output to multiple "workers". Otherwise it will also fail.

    I've wasted 3 days of my life on this annoying bug. Argh.

提交回复
热议问题