问题
Related questions:
- Java PriorityQueue with fixed size
- How do I use a PriorityQueue?
- get indexes of n smallest elements in an array
- Scala: Is there a way to use PriorityQueue like I would in Java?
I have a very large data set (more than 5 millions items) and I need to get N largest items from it. The most natural way to do it is to use heap/priority queue storing only top N items. There are several good implementations of priority queue for JVM (Scala/Java), namely:
- scala.collection.mutable.PriorityQueue
- java.util.PriorityQueue
- lucene.util.PriorityQueue
First 2 are nice, but they store all the items, which in my case gives critical memory overhead. Third (Lucene implementation) doesn\'t have such a drawback, but as I can see from documentation it also doesn\'t support custom comparator, which makes it useless for me.
So, my question is: Is there a PriorityQueue implementation with fixed capacity and custom comparator?
UPD. Finally I\'ve created my own implementation, based on Peter\'s answer:
public class FixedSizePriorityQueue<E> extends TreeSet<E> {
private int elementsLeft;
public FixedSizePriorityQueue(int maxSize) {
super(new NaturalComparator());
this.elementsLeft = maxSize;
}
public FixedSizePriorityQueue(int maxSize, Comparator<E> comparator) {
super(comparator);
this.elementsLeft = maxSize;
}
/**
* @return true if element was added, false otherwise
* */
@Override
public boolean add(E e) {
if (elementsLeft == 0 && size() == 0) {
// max size was initiated to zero => just return false
return false;
} else if (elementsLeft > 0) {
// queue isn\'t full => add element and decrement elementsLeft
boolean added = super.add(e);
if (added) {
elementsLeft--;
}
return added;
} else {
// there is already 1 or more elements => compare to the least
int compared = super.comparator().compare(e, this.first());
if (compared == 1) {
// new element is larger than the least in queue => pull the least and add new one to queue
pollFirst();
super.add(e);
return true;
} else {
// new element is less than the least in queue => return false
return false;
}
}
}
}
(where NaturalComparator is taken from this question)
回答1:
You could use a SortedSet e.g. TreeSet with a custom comparator and remove the smallest when the size reachs N.
回答2:
How can you say Lucene's doesn't support a custom comparator?
Its abstract and you must implement the abstract method lessThan(T a, T b)
回答3:
Though an old question but it may be helpful to somebody else. You can use minMaxPriorityQueue of Google's Java library guava.
回答4:
I can't think of a ready-to-use one, but you can check my implementation of this collection with similar requirements.
The difference is the comparator, but if you extend from PriorityQueue you'll have it. And on each addition check if you haven't reached the limit, and if you have - drop the last item.
回答5:
Below is the implementation I used before. Complies with Peter's suggestion.
public @interface NonThreadSafe {
}
/**
* A priority queue implementation with a fixed size based on a {@link TreeMap}.
* The number of elements in the queue will be at most {@code maxSize}.
* Once the number of elements in the queue reaches {@code maxSize}, trying to add a new element
* will remove the greatest element in the queue if the new element is less than or equal to
* the current greatest element. The queue will not be modified otherwise.
*/
@NonThreadSafe
public static class FixedSizePriorityQueue<E> {
private final TreeSet<E> treeSet; /* backing data structure */
private final Comparator<? super E> comparator;
private final int maxSize;
/**
* Constructs a {@link FixedSizePriorityQueue} with the specified {@code maxSize}
* and {@code comparator}.
*
* @param maxSize - The maximum size the queue can reach, must be a positive integer.
* @param comparator - The comparator to be used to compare the elements in the queue, must be non-null.
*/
public FixedSizePriorityQueue(final int maxSize, final Comparator<? super E> comparator) {
super();
if (maxSize <= 0) {
throw new IllegalArgumentException("maxSize = " + maxSize + "; expected a positive integer.");
}
if (comparator == null) {
throw new NullPointerException("Comparator is null.");
}
this.treeSet = new TreeSet<E>(comparator);
this.comparator = treeSet.comparator();
this.maxSize = maxSize;
}
/**
* Adds an element to the queue. If the queue contains {@code maxSize} elements, {@code e} will
* be compared to the greatest element in the queue using {@code comparator}.
* If {@code e} is less than or equal to the greatest element, that element will be removed and
* {@code e} will be added instead. Otherwise, the queue will not be modified
* and {@code e} will not be added.
*
* @param e - Element to be added, must be non-null.
*/
public void add(final E e) {
if (e == null) {
throw new NullPointerException("e is null.");
}
if (maxSize <= treeSet.size()) {
final E firstElm = treeSet.first();
if (comparator.compare(e, firstElm) < 1) {
return;
} else {
treeSet.pollFirst();
}
}
treeSet.add(e);
}
/**
* @return Returns a sorted view of the queue as a {@link Collections#unmodifiableList(java.util.List)}
* unmodifiableList.
*/
public List<E> asList() {
return Collections.unmodifiableList(new ArrayList<E>(treeSet));
}
}
I would appreciate any feedback btw.
EDIT: It seems like using a TreeSet is not very efficient after all because the calls to first() seem to take sublinear time. I changed the TreeSet to a PriorityQueue. The modified add() method looks like this:
/**
* Adds an element to the queue. If the queue contains {@code maxSize} elements, {@code e} will
* be compared to the lowest element in the queue using {@code comparator}.
* If {@code e} is greater than or equal to the lowest element, that element will be removed and
* {@code e} will be added instead. Otherwise, the queue will not be modified
* and {@code e} will not be added.
*
* @param e - Element to be added, must be non-null.
*/
public void add(final E e) {
if (e == null) {
throw new NullPointerException("e is null.");
}
if (maxSize <= priorityQueue.size()) {
final E firstElm = priorityQueue.peek();
if (comparator.compare(e, firstElm) < 1) {
return;
} else {
priorityQueue.poll();
}
}
priorityQueue.add(e);
}
回答6:
Exactly what I was looking for. However, the implementation contains a bug:
Namely: if elementsLeft > 0 and e is already contained in the TreeSet. In this case, elementsLeft is decreased, but the number of elements in the TreeSet stays the same.
I would suggest to replace the corresponding lines in the add() method by
} else if (elementsLeft > 0) {
// queue isn't full => add element and decrement elementsLeft
boolean added = super.add(e);
if (added) {
elementsLeft--;
}
return added;
回答7:
Try this code:
public class BoundedPQueue<E extends Comparable<E>> {
/**
* Lock used for all public operations
*/
private final ReentrantLock lock;
PriorityBlockingQueue<E> queue ;
int size = 0;
public BoundedPQueue(int capacity){
queue = new PriorityBlockingQueue<E>(capacity, new CustomComparator<E>());
size = capacity;
this.lock = new ReentrantLock();
}
public boolean offer(E e) {
final ReentrantLock lock = this.lock;
lock.lock();
E vl = null;
if(queue.size()>= size) {
vl= queue.poll();
if(vl.compareTo(e)<0)
e=vl;
}
try {
return queue.offer(e);
} finally {
lock.unlock();
}
}
public E poll() {
return queue.poll();
}
public static class CustomComparator<E extends Comparable<E>> implements Comparator<E> {
@Override
public int compare(E o1, E o2) {
//give me a max heap
return o1.compareTo(o2) *-1;
}
}
}
回答8:
Here is one I put together if you have guava. I think it is is pretty complete. Let me know if I missed something.
You can use the gauva ForwardingBlockingQueue so you don't have to map all the other methods.
import com.google.common.util.concurrent.ForwardingBlockingQueue;
public class PriorityBlockingQueueDecorator<E> extends
ForwardingBlockingQueue<E> {
public static final class QueueFullException extends IllegalStateException {
private static final long serialVersionUID = -9218216017510478441L;
}
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
private int maxSize;
private PriorityBlockingQueue<E> delegate;
public PriorityBlockingQueueDecorator(PriorityBlockingQueue<E> delegate) {
this(MAX_ARRAY_SIZE, delegate);
}
public PriorityBlockingQueueDecorator(int maxSize,
PriorityBlockingQueue<E> delegate) {
this.maxSize = maxSize;
this.delegate = delegate;
}
@Override
protected BlockingQueue<E> delegate() {
return delegate;
}
@Override
public boolean add(E element) {
return offer(element);
}
@Override
public boolean addAll(Collection<? extends E> collection) {
boolean modified = false;
for (E e : collection)
if (add(e))
modified = true;
return modified;
}
@Override
public boolean offer(E e, long timeout, TimeUnit unit)
throws InterruptedException {
return offer(e);
}
@Override
public boolean offer(E o) {
if (maxSize > size()) {
throw new QueueFullException();
}
return super.offer(o);
}
}
回答9:
Create a PriorityQueue that has size limit. It stores N max numbers.
import java.util.*;
class Demo
{
public static <E extends Comparable<E>> PriorityQueue<E> getPq(final int n, Comparator<E> comparator)
{
return new PriorityQueue<E>(comparator)
{
boolean full()
{
return size() >= n;
}
@Override
public boolean add(E e)
{
if (!full())
{
return super.add(e);
}
else if (peek().compareTo(e) < 0)
{
poll();
return super.add(e);
}
return false;
}
@Override
public boolean offer(E e)
{
if (!full())
{
return super.offer(e);
}
else if (peek().compareTo(e) < 0)
{
poll();
return super.offer(e);
}
return false;
}
};
}
public static void printq(PriorityQueue pq)
{
Object o = null;
while ((o = pq.poll()) != null)
{
System.out.println(o);
}
}
public static void main (String[] args)
{
PriorityQueue<Integer> pq = getPq(2, new Comparator<Integer>(){
@Override
public int compare(Integer i1, Integer i2)
{
return i1.compareTo(i2);
}
});
pq.add(4);
pq.add(1);
pq.add(5);
pq.add(2);
printq(pq);
}
}
来源：https://stackoverflow.com/questions/7878026/is-there-a-priorityqueue-implementation-with-fixed-capacity-and-custom-comparato