Actually this is a classic problem as SO user Victor put it (in another SO question regarding which tasks to ask during an interview).
I couldn\'t do it in an hour
Here's another method, not necessarily the best, but sure to impress any interviewer.
First, call the point with the lowest X co-ord 'L', the point with the highest X co-ord 'R', and the remaining point 'M' (Left, Right, and Middle).
Then, set up two instances of Bresenham's line algorithm. Parameterize one instance to draw from L to R, and the second to draw from L to M. Run the algorithms simultaneously for X = X[L] to X[M]. But instead of drawing any lines or turning on any pixels, count the pixels between the lines.
After stepping from X[L] to X[M], change the parameters of the second Bresenham to draw from M to R, then continue to run the algorithms simultaneously for X = X[M] to X[R].
This is very similar to the solution proposed by Erwin Smout 7 hours ago, but using Bresenham instead of a line-slope formula.
I think that in order to count the columns of pixels, you will need to determine whether M lies above or below the line LR, and of course special cases will arise when two points have the same X or Y co-ordinate. But by the time this comes up, your interviewer will be suitably awed and you can move on to the next question.