Problems apply to the new 64-bit Vista Windows, which will eventually supplant the 32-bit version. Vista is due to be shipped to corporate users next month and sold to the public in January.
Security companies accounting for a majority of the market say they need access to the core, or "kernel" of 64-bit Windows to fully protect customers.
Software security companies now have access to the kernel in 32-bit Windows, but Microsoft has put up a wall called "PatchGuard" to protect the 64-bit kernel from hackers. Security software firms say that keeps them out, too.
The Gartner group, which issued a report this week assessing problems Microsoft would have in making needed changes to Vista security, said security software "will not deliver full functionality" for 64-bit Windows under current conditions.
Microsoft last week promised the European Commission that it would change the Vista operating system to meet concerns of security software makers.
Gartner recommended that companies tell Microsoft they would not make a commitment to 64-bit Vista until a firm release is set for the first set of kernel-control" software.
Gartner said that might be in 2008, and even then there might not be full functionality.
Microsoft said on Thursday after briefing software makers that working with them to come up with a solution would be time-consuming.
"We believe we have a long runway to work with," said Adrien Robinson of Microsoft's security technology unit, because companies will be slow to shift to the 64-bit version.
But McAfee security software said it had customers who would already have adopted 64-bit technology.
Microsoft's Robinson said the new technology may not provide access to the kernel, but instead would allow security companies to peer into the kernel. "There is no longer direct access," Robinson said.
Windows 64-bit Vista is supposed to crash if any malicious software tries to get access to the kernel.
But Helmuth Feericks, chief technology officer of Authentium software in Palm Beach Gardens, Florida, said his group had figured out a way to turn off PatchGuard protection, install its own software, and then turn it back on.
Feericks said that if he could get such access so could sophisticated hackers. "It's going to be a continuous battle between Microsoft and the security industry and the hackers," he said.