Pruning Decision Trees

Question

Below is a snippet of the decision tree as it is pretty huge.

How to make the tree stop growing when the lowest value in a node is under 5. H

Answer 1

Interestingly, min_impurity_decrease doesn't look as if it would allow growth of any of the nodes you have shown in the snippet you provided (the sum of impurities after splitting equals the pre-split impurity, so there is no impurity decrease). However, while it won't give you exactly the result you want (terminate node if lowest value is under 5), it may give you something similar.

If my testing is right, the official docs make it look more complicated than it actually is. Just take the lower value from the potential parent node, then subtract the sum of the lower values of the proposed new nodes - this is the gross impurity reduction. Then divide by the total number of samples in the whole tree - this gives you the fractional impurity decrease achieved if the node is split.

If you have 1000 samples, and a node with a lower value of 5 (i.e. 5 "impurities"), 5/1000 represents the maximum impurity decrease you could achieve if this node was perfectly split. So setting a min_impurity_decrease of of 0.005 would approximate stopping the leaf with <5 impurities. It would actually stop most leaves with a bit more than 5 impurities (depending upon the impurities resulting from the proposed split), so it is only an approximation, but as best I can tell its the closest you can get without post-pruning.