In general, with the NVIDIA naming scheme, GTX>GTS>GT. However, this is not always the case. For example, the 512MB 8800GT performs better than the 640MB 8800GTS. But then again, the 512MB 8800GTS performs better than the 8800GT.
As for the original question of why is the 8800GT better than the 9600GT... Well, a high-end card from a previous generation is not always better than a mid-range card from the next generation. For example, the 7600GT outperforms the 6800GT. However, the 7600GT was built on a whole new microarchitecture than the 6800GT. In comparison, the 9600GT was built on a similar, but crippled version of the 8800GT architecture, so it has less performance. So, whether or not the the next generation will significantly outperform the previous generation (as the 7th did to the 6th, but the 9th didn't do to the 8th) depends on how big the changes in the generations were. For example, there is no difference between the 8800GT and the 9800GT other than the 9800GT has Tri-SLI support. In contrast, the 8800GT absolutely demolishes the 7800GT. But how can you know all these little facts about each card? Well, you don't need to; that's where benchmarks come in.
As for what card you should get, while video RAM should not be the deciding factor (as noted above, the 512MB 8800GT and the 512MB 8800GTS both outperform the 640MB 8800GTS), the HD4850 is the sweet spot for performance for the price right now. Oh, and the HD4850, at least by reference design, only has 512MB of video RAM. But again, don't base your decision solely on the amount of video RAM.