I suspect with such a size discrepency, the file that is open in Outlook is not the file you're looking at in Explorer.
1. Open Outlook, right click the top folder level of the PST file you need to compact, Properties, Advanced, File Name. Take note of the file name and location.
2. Go to Control Panel, Mail, Data Files. Is that file listed here? If it is, remove it. You can add it back after #3.
3. Run Scanpst.exe to repair the file. See Microsoft article 287497
4. Add the PST file back to Outlook, using the instructions for creating a new file, except that you choose one instead of naming a new one (see #4). Open Outlook, expand the folders for that PST file, delete as many items as you can (out of Sent, Deleted, Inbox, etc). Try to compact it again.
Did any of this help? If not,
4. Create a new PST file (that you will move all the existing items from the problem PST file into). Go to Start, Control Panel, Mail, Data Files, Add, give the file a name, and click OK. Drag/drop items from the old PST into the new PST.
5. It may be that your Mail Profile is damaged. See Microsoft article 829918
for instructions on creating a new one.
Please post the results of what worked and what didn't.