There really isn't any way of telling what form factor that case takes for the motherboard. From looking it seems to be an ATX or mATX, but without physically being there its too hard to tell. Also, every Dell I've worked on (and most other OEM machines) have special proprietary connectors for the front panel connectors and such. So you would have to do some wire splicing along with soldering new connectors and probably some trial an error to figure out which wires do what. i.e power button, reset button, hdd activity light, power light, etc.
This is more along the lines of a "case mod" and I wouldn't suggest it since it's your first build. You would have a much easier time if you went with a cheap case like THIS HERE
for $18. It's not the greatest case in the world, but compatibility should be all there. Other cheap cases HERE
There are so many different motherboards out there that I don't know what to choose!
Yes there are. Generally what works best for me to help you is if you give me a budget on what you plan to spend on the entire build. From there I can make suggestions for the best bang for your buck and see what we can put together for your build within your budget. Once I know that, I can put together a "mock" build for you.
Should I get something with PCI and PCI express slots?
PCI Express slots are the newest fastest slots available on a motherboard. More specifically, PCI-E x16 2.1 is the fastest right now. So picking a video card that is PCI-E x16 2.0 or higher is the way to go. As for the sound card, PCI or PCI Express will do just fine. Some sound cards get really expensive and unless you are a DJ, doing some sound recording, or some sort of mixing artist... I think onboard sound with the newer motherboards will do you just fine. A dedicated sound card is overkill and eliminating it will probably keep you within budget easier.
And can I use my existing power supply?
I wouldn't suggest that AT ALL. OEM power supplies that come with systems like Dells, HPs, and Gateway's are just barely powerful enough to power the current system. If you go with newer more power consuming technology, the existing power supply just won't handle it. It may start up, but you risk blowing every single new component up in the system if it fails and goes "pop".
Also, most video cards in the range or higher than what you are looking at require a more powerful power supply even at which point I would recommend a 80 plus certified PSU to ensure your rig gets clean and stable power even under load like when gaming.
In a nutshell, give me a dollar amount and I'll tell you what is possible or what's not possible. Also, mentioning what games you plan to play now and in the future will help me as well. Any other specifics as to what you will use this computer for will also help.
Let me know.