I'm not sure what firewall you have, if any, however a good (free) firewall that I am currently using (and would recommend) is Kerio Personal Firewall. With this firewall, you'll be able to control the number of connections you're making to the Internet.
Did you mention to your ISP that this issue is happening on all of your computers? Seems a little odd to me that both of your computers have too many connections... Oh well.
Try installing Kerio and see if it helps.