How to set parameters for a custom PySpark Transformer once it's a stage in a fitted ML Pipeline?

Question

I\'ve written a custom ML Pipeline Estimator and Transformer for my own Python algorithm by following the pattern shown here.

However, in t

Answer 1

There is no getStages() method on PipelineModel but the same class does have an undocumented member called stages.

For example, if you've just fitted a pipeline model with 3 stages and you want to set some parameters on the second stage, you can just do something like:

myModel = myPipelineModel.stages[1]
myModel.setMyParam(42)
# Or in one line:
#myPipelineModel.stages[1].setMyParam(42)

# Now we can push our data through the fully configured pipeline model:
resultsDF = myPipelineModel.transform(inputDF)