NSOutlineView Changing disclosure Image

You've got the basic idea but what you will need to do is draw the image yourself. Here's the code I use:

- (void)outlineView:(NSOutlineView *)outlineView willDisplayOutlineCell:(id)cell forTableColumn:(NSTableColumn *)tableColumn item:(id)item {
    NSString *theImageName;
    NSInteger theCellValue = [cell integerValue];
    if (theCellValue==1) {
        theImageName = @"PMOutlineCellOn";
    } else if (theCellValue==0) {
        theImageName = @"PMOutlineCellOff";
    } else {
        theImageName = @"PMOutlineCellMixed";
    }

    NSImage *theImage = [NSImage imageNamed: theImageName];
    NSRect theFrame = [outlineView frameOfOutlineCellAtRow:[outlineView rowForItem: item]];
    theFrame.origin.y = theFrame.origin.y +17;
    // adjust theFrame here to position your image
    [theImage compositeToPoint: theFrame.origin operation:NSCompositeSourceOver];
    [cell setImagePosition: NSNoImage];
}

You will need 3 different images as you can see, one for the ON state, one for the OFF state and also one for the MIXED state which should be halfway between the two. The mixed state makes sure you still get the opening and closing animation.

After researching this issue myself, and trying some of the answers here, I have found that the other approaches mentioned will work, but will require that you perform a lot more manual intervention in order to avoid screen artifacts and other strange behavior.

The simplest solution I found is the following, which should work in most cases.

This solution has the added benefit of the system automatically handling a great many other cases, such as column movement, etc, without your involvement.

- (void)outlineView:(NSOutlineView *)outlineView willDisplayOutlineCell:(id)cell
     forTableColumn:(NSTableColumn *)tableColumn
               item:(id)item
{
    [cell setImage:[NSImage imageNamed: @"Navigation right 16x16 vWhite_tx"]];
    [cell setAlternateImage:[NSImage imageNamed: @"Navigation down 16x16 vWhite_tx"]];
}

In your case, you would wrap this up with your class detection logic, and set the cell images appropriately for your cases.

A nice way to change the disclosure image is to use a view based outline view:

In your ViewController with NSOutlineViewDelegate:

- (NSView *)outlineView:(NSOutlineView *)outlineView viewForTableColumn:(NSTableColumn *)tableColumn item:(id)item
{
    CustomNSTableCellView *cell    = [outlineView makeViewWithIdentifier:tableColumn.identifier owner:self];
    cell.item                      = item;

    return cell;
}

You have to subclass your NSOutlineView and overide the method:

- (id)makeViewWithIdentifier:(NSString *)identifier owner:(id)owner
{
    id view = [super makeViewWithIdentifier:identifier owner:owner];

    if ([identifier isEqualToString:NSOutlineViewDisclosureButtonKey])
    {
        // Do your customization
        // return disclosure button view

        [view setImage:[NSImage imageNamed:@"Disclosure_Categories_Plus"]];
        [view setAlternateImage:[NSImage imageNamed:@"Disclosure_Categories_Minus"]];
        [view setBordered:NO];
        [view setTitle:@""];

        return view;
    }

    return view;
}

//Frame of the disclosure view
- (NSRect)frameOfOutlineCellAtRow:(NSInteger)row
{
    NSRect frame = NSMakeRect(4, (row * 22), 19, 19);
    return frame;
}

For who looks for Swift2 Solution. Subclass NSRow of your outlineview and override didAddSubview method as below.

override func didAddSubview(subview: NSView) {
    super.didAddSubview(subview)
    if let sv = subview as? NSButton {
        sv.image = NSImage(named:"IconNameForCollapsedState")
        sv.alternateImage = NSImage(named:"IconNameForExpandedState")
    }
}

This is what i have tried and working so far,

/* because we are showing our own disclose and expand button */

- (NSRect)frameOfOutlineCellAtRow:(NSInteger)row {

    return NSZeroRect;
}
- (NSRect)frameOfCellAtColumn:(NSInteger)column row:(NSInteger)row {
    NSRect superFrame = [super frameOfCellAtColumn:column row:row];

    if ((column == 0) && ([self isGroupItem:[self itemAtRow:row]])) {
        return NSMakeRect(0, superFrame.origin.y, [self bounds].size.width, superFrame.size.height);
    }
    return superFrame;
}

I have subclassed NSOutlineView class and override these methods,

[self isGroupItem] is to check whether its group or not. but got one problem, now looks like mousehandling i need to do :( , on double clicking group row is not toggling