Irregular shaped Frame

AWT top level windows (and frames) are always rectangular, but there are some workarounds which enable you to create the ilusion of non-rectangular windows.

You can use a trick and create a rectangular, pseudo-transparent window that will look like a irregular-shape window. Use the Robot class to grab the screen image before displaying your window, and put it into the background. Then, synchronize the background every time the window moves. This method is not perfect, because it does not take into account background activities of other windows. Here is a snippet from two classes displaying a round window (used as a GUI for a CD archive)

import javax.swing.*;
 import javax.swing.event.*;
import javax.swing.text.html.HTMLEditorKit;
import javax.swing.tree.*;
 
import java.awt.*;
import java.awt.event.*;
import java.io.*;
import java.net.*;
import java.util.ResourceBundle;
 
 
public class CDWindow extends JFrame {
	Image img1, img2;
	BgPanel jpl;
	Dimension d = Toolkit.getDefaultToolkit().getScreenSize();
	Robot r;
	Rectangle rect = new Rectangle(0, 0, d.width, d.height);
	private boolean standalone;
	private String rootDir;
	public CDWindow(boolean stdAlone, String rootDirectory) {			
		setTitle("my title");// set task bar label
		setUndecorated(true);
		setBounds((d.width - 500) / 2, (d.height - 500) / 2, 500, 500);
		try {
			r = new Robot();
			img1 = r.createScreenCapture(rect); // get screen image
		} catch (AWTException awe) { 
			awe.printStackTrace();
			System.out.println("error reading screen");
			System.exit(0);
		}
		try {
			img2 = // load image, for example a round CD
			(new ImageIcon(ClassLoader.getSystemResource("cd.gif")))
			.getImage();
		} catch (Exception e) {
			e.printStackTrace();
			System.out.println("error loading picture:" + e);
		}
		jpl = new BgPanel(img1, img2, this);
		jpl.setOpaque(false);
		setContentPane(jpl);
		requestFocus();
		addFocusListener(new FocusAdapter() {
			public void focusLost(FocusEvent fe) {		
				img1 = null;
			}
			public void focusGained(FocusEvent fe) {		
				//refreshBg();	
			}
		});
		addWindowListener(new WindowListener() {
			public void windowActivated(WindowEvent e) {
				refreshBg(); 
 
			}
			public void windowClosed(WindowEvent e) {}
			public void windowClosing(WindowEvent e) {}
			public void windowDeiconified(WindowEvent e) {}
			public void windowOpened(WindowEvent e) {}
			public void windowDeactivated(WindowEvent e) {
				img1 = null;
			}
 
			public void windowIconified(WindowEvent e) {
				img1 = null;
			}
		});
	}
 
	private synchronized void refreshBg() {
		if (img1 != null)
			return;
		setSize(0, 0);
		img1 = r.createScreenCapture(rect);
		setSize(500, 500);		
		jpl.setImg(img1);
	}
};

Panel with two background images - screen background and irregular frame shape image.

class BgPanel extends JPanel {
	Image img1, img2;
	Window f;
	public BgPanel(Image im1, Image im2, Window fr) {
		img1 = im1;
		img2 = im2;
		f = fr;
	}
	public void paintComponent(Graphics g) {
		g.drawImage(
			img1,
			0,
			0,
			getWidth(),
			getHeight(),
			f.getX(),
			f.getY(),
			f.getX() + getWidth(),
			f.getY() + getHeight(),
			this);
		g.drawImage(img2, 0, 0, null);
 
	}
	public void setImg(Image i) {
		img1 = i;
		repaint();
	}
};

IFrame class from IBM developerworks implements this solution http://www-128.ibm.com/developerworks/java/library/j-iframe

Windows in OS X implementation of Swing can have a transparent background. Use this to create a transparent window:

 window.setBackground(new Color(0, 0, 0, 0));

You can put a rectangular transparent image with a non-rectangular visible shape into the background, and turn of window border and title bar, to simmulate non-rectangular windows.

A native windows skin look and feel from l2fprod can be used to create non-rectangular frames on MS Windows only. See also skins_for_swing.

Java Native Access (JNA) includes a demo program which implements non-rectangular windows on OSX, windows, and X11-based systems. You can run the demo here. There is also example code for constant and per-pixel transparency which composites your window with whatever is beneath it.