Remember that the preferred size of components is used, if the size is never changed (for example, using layouts).
It is not straightforward, but minimal size is not checked when you put a component into a BorderLayout left position, for example. Create an empty panel, define non-zero minimal size of that panel, put it there, and the panel will still not be displayed.
So, when changing minimal size, always update also the preferred size of the panel.