Pyspark random forest feature importance mapping after column transformations

Question

I am trying to plot the feature importances of certain tree based models with column names. I am using Pyspark.

Since I had textual categorical variables and numeri

Answer 1

The transformed dataset metdata has the required attributes.Here is an easy way to do -

1. create a pandas dataframe (generally feature list will not be huge, so no memory issues in storing a pandas DF)

   pandasDF = pd.DataFrame(dataset.schema["features"].metadata["ml_attr"] 
   ["attrs"]["binary"]+dataset.schema["features"].metadata["ml_attr"]["attrs"]["numeric"]).sort_values("idx")
   

2. Then create a broadcast dictionary to map. broadcast is necessary in a distributed environment.

   feature_dict = dict(zip(pandasDF["idx"],pandasDF["name"])) 
   
   feature_dict_broad = sc.broadcast(feature_dict)