AWT does not allow part of a frame to be transparent. But, you can grab the screen background and show it in the frame. For code snippets, look at Irregular Frame Shape.
There is a new java.awt.noerasebackground property since 1.4, but I have not been able to make a frame transparent by changing that property and setting opaque property of all contained components to true. Perhaps this will be fixed in a newer version of Swing.