So I have 6 panels, all of them used a grid layout. And I put them together using gridbaglayout, here is the design I wanted
I am not familiar enough with the GridBagLayout to quickly spot what you can improve to your code to fix this. But another approach might be possible, using nested BorderLayouts.
Panels 'first', 'third' and 'fourth' can be placed inside one JPanel Awith a BorderLayout in the NORTH, WEST and EAST respectively. The same can be done for the panels 'second', 'fifth' and 'sixth' in a JPanel B.
Then you group those 2 panels (A and B) in another panel with a BorderLayout in the WEST and EAST