To get the current selection use:
Dim rng As TextRange = New TextRange(YourRtfBox.Selection.Start, YourRtfBox.Selection.End)
And then set the fontstyle:
rng.ApplyPropertyValue(Inline.FontSizeProperty, YourFontSizeValue)
rng.ApplyPropertyValue(Inline.FontFamilyProperty, YourFontFamilyValue)