Is it possible to partition data based on column which may contain empty value sometimes? how to handle it?

问题

I am using spark-sql-2.4.1v. I need to join two datasets ds1 and ds2 (new-column) based on some field. val resultDs = ds1.join(ds2, , "inner");

resultDs now contain "new-column" but for the records which not met does not have this "new-column". Hence for those records I need to set this "new-column" null/empty.

But as per my business requirement I need to parition the resultDs on "new-column". so how generally these kind of scenarios are handled ? pleas advice.

来源：https://stackoverflow.com/questions/61607228/is-it-possible-to-partition-data-based-on-column-which-may-contain-empty-value-s