Pure Prolog programs that distinguish between the equality and inequality of terms in a clean manner suffer from execution inefficiencies ; even when all terms of relevance
The implementation of occurrences/3 below is based on my previous answers, namely by profiting from the clause-indexing mechanism on the 1st argument to avoid some choice-points, and addresses all the issues that were raised.
Moreover it copes with a problem in all submited implementations up to now, including the one referred to in the question, namely that they all enter an infinite loop when the query has the 2 first arguments free and the 3rd a list with different ground elements. The correct behaviour is to fail, of course.
Use of a comparison predicate
I think that in order to avoid unused choice-points and keeping a good degree of the implementation declarativity there is no need for a comparison predicate as the one proposed in the question, but I agree this may be a question of taste or inclination.
Implementation
Three exclusive cases are considered in this order: if the 2nd argument is ground then it is visited recursively; otherwise if the 3rd argument is ground it is checked and then visited recursively; otherwise suitable lists are generated for the 2nd and 3rd arguments.
occurrences(X, L, Os) :-
( nonvar(L) -> occs(L, X, Os) ;
( nonvar(Os) -> check(Os, X), glist(Os, X, L) ; v_occs(L, X, Os) ) ).
The visit to the ground 2nd argument has three cases when the list is not empty: if its head and X above are both ground and unifiable X is in the head of the resulting list of occurrences and there is no other alternative; otherwise there are two alternatives with X being different from the head or unifying with it:
occs([],_,[]).
occs([X|R], Y, ROs) :-
( X==Y -> ROs=[X|Rr] ; foccs(X, Y, ROs, Rr) ), occs(R, Y, Rr).
foccs(X, Y, ROs, ROs) :- dif(X, Y).
foccs(X, X, [X|ROs], ROs).
Checking the ground 3rd argument consists in making sure all its members unify with X. In principle this check could be performed by glist/3 but in this way unused choice-points are avoided.
check([], _).
check([X|R], X) :- check(R, X).
The visit to the ground 3rd argument with a free 2nd argument must terminate by adding variables different from X to the generated list. At each recursion step there are two alternatives: the current head of the generated list is the current head of the visited list, that must be unifiable with X or is a free variable different from X. This is a theoretic-only description because in fact there is an infinite number of solutions and the 3rd clause will never be reached when the list head is a variable. Therefore the third clause below is commented out in order to avoid unusable choice-points.
glist([], X, L) :- gdlist(L,X).
glist([X|R], X, [X|Rr]) :- glist(R, X, Rr).
%% glist([X|R], Y, [Y|Rr]) :- dif(X, Y), glist([X|R], Y, Rr).
gdlist([], _).
gdlist([Y|R], X) :- dif(X, Y), gdlist(R, X).
Finally the case where all arguments are free is dealt with in a way similar to the previous case and having a similar problem of some solution patterns not being in practice generated:
v_occs([], _, []).
v_occs([X|R], X, [X|ROs]) :- v_occs(R, X, ROs).
%% v_occs([X|R], Y, ROs) :- dif(Y, X), v_occs(R, Y, ROs). % never used
Sample tests
?- occurrences(1,[E1,1,2,1,E2],Fs).
Fs = [1,1],
dif(E1,1),
dif(E2,1) ? ;
E2 = 1,
Fs = [1,1,1],
dif(E1,1) ? ;
E1 = 1,
Fs = [1,1,1],
dif(E2,1) ? ;
E1 = E2 = 1,
Fs = [1,1,1,1] ? ;
no
?- occurrences(1,L,[1,2]).
no
?- occurrences(1,L,[1,E,1]).
E = 1,
L = [1,1,1] ? ;
E = 1,
L = [1,1,1,_A],
dif(1,_A) ? ;
E = 1,
L = [1,1,1,_A,_B],
dif(1,_A),
dif(1,_B) ? ;
...