Any time you're chaining generics or nesting data structures like this, it's time to take a step back and think about what you're really trying to do.
Would this be better if it were encapsulated in a wrapper class? It will probably make things less complex later. A week from now when you have to debug something in your array of hashsets of arraylists of arrays of some simple datatype because you thought you'd just keep wrapping them, you'll be sorry.
-edit: arrays are illegal? Point noted. I still feel the argument here stands.
You're using Java; don't be afraid to make some extra classes.
public class MyIntegers {
private HashSet theIntegers;
// wrap methods
public boolean add(Integer i) { return theIntegers.add(i); }
public boolean contains(Integer i)...
public boolean remove(Integer i)...
}
// later...
public static void main(String[] args) {
MyIntegers[] rows = new MyIntegers[NUM_ROWS];
}