You can't set the style, but you can simulate a pattern like a dotted line by using a pattern image and a UIColor.
UIView *item;
item.layer.borderColor = [UIColor colorWithPatternImage:[UIImage imageNamed:@"checkerboard.png"]].CGColor;
Don't forget to add this to your headers:
#import