how to extract the borders of an image (OCT/retinal scan image)

Question

I have an (OCT) image like shown below (original). As you can see, it mainly has 2 layers. I want to produce an image (shown in the 3rd picture), in which the red line indic

Answer 1

A full working implementation in Octave:

pkg load image
pkg load signal

median_filter_size = 10;
min_vertical_distance_between_layers = 35;
min_bright_level = 100/255;

img_rgb = imread("oct.png");% it's http://i.stack.imgur.com/PBnOj.png
img = im2double(img_rgb(:,:,1));
img=medfilt2(img,[median_filter_size median_filter_size]);

function idx = find_max(img,col_idx,min_vertical_distance_between_layers,min_bright_level)
  c1=img(:,col_idx);

  [pks idx]=findpeaks(c1,"MinPeakDistance",min_vertical_distance_between_layers,"MinPeakHeight",min_bright_level);

  if ( rows(idx) < 2 )
    idx=[1;1];
    return
  endif

  % sort decreasing peak value
  A=[pks idx];
  A=sortrows(A,-1);

  % keep the two biggest peaks
  pks=A(1:2,1);
  idx=A(1:2,2);

  % sort by row index
  A=[pks idx];
  A=sortrows(A,2);

  pks=A(1:2,1);
  idx=A(1:2,2);
endfunction

layers=[];
idxs=1:1:columns(img);
for col_idx=idxs
  layers=[layers find_max(img,col_idx,min_vertical_distance_between_layers,min_bright_level)];
endfor
hold on
imshow(img)
plot(idxs,layers(1,:),'r.')
plot(idxs,layers(2,:),'g.')

my_range=1:columns(idxs);

for i = my_range
  x = idxs(i);
  y1 = layers(1,i);
  y2 = layers(2,i);
  if  y1 > rows(img_rgb) || y2 > rows(img_rgb) || x > columns(img_rgb) || y1==1 || y2==1
    continue
  endif
  img_rgb(y1,x,:) = [255 0 0];
  img_rgb(y2,x,:) = [0 255 0];
endfor 

imwrite(img_rgb,"dst.png")

The idea is to process every column of the image as a curve (of grey levels) and looking for two peaks, each peak is on the border of a layer.

The input image is the original linked by the OP: http://i.stack.imgur.com/PBnOj.png

The image saved by the code as "dst.png" is the following: