My Outlook PST file recently reached 3GB in size and, while Outlook will still performing ok, I needed to reduce the size of the file to keep the size of my nightly backup reasonable.
I removed old and large messages from the Sent and Deleted messages folders and estimated that I freed up at least 1/3 of my used space. I tried compacting the file through Outlook and through the Control Panel Mail applet. The compaction finished too quickly and the file remained the same size.
This was solved by rebooting and going directly to the Control Panel Mail applet (double click Personal folders line and click the Compact Now button) before opening Outlook. My PST file is now 1.2GB.
I hope this helps someone else.