[PhysicsNinja]

I got my hands on a Dell Poweredge R900, with 64 GB of RAM and 5x 73 GB drives set up as a RAID5.  It has 4x guadcore Xeons and is running Windows Server 2008 R2.  The machine is pretty snappy...  boots up lightning fast and is incredibly responsive. 

 

I do predictive modeling for a living, and I'm trying to use it as a compute pool to play with some models of my own at home (pet-projects).  16 cores should rip through my simulations pretty quickly.  I'm running custom .net code (compiled, not run in Debug mode in the Visual Studio GUI).  I can launch an arbitrary number of simulation thread with each thread computing for 10 minutes or so.  But that is where things got interesting. 

 

Running 16 threads on my 16 cores I expected the CPUs to rail, the fans to spool up and some serious numbers to be crunched...   Instead I saw an average CPU utilization of 20%.  Each core was at the 20% level, and the speed per thread was way slower than my desktop PC (yup, about 5x slower).  I tried 7 threads as a test...  7 cores jumped to 20%, and each of the 7 threads ran about the same speed as each thread did in the 16 thread case.  So I figured why not try 48 threads?  3 threads per core should put each core at 60% right?  Nope...  20% on all cores again.  Each thread now visibly ran 3x slower than the 16 thread case too.  It seems the Performance monitor isn't lying to me. 

 

So Curious if this was a hardware limit, I fired off 2 instances of my code, with both running 16 threads, my utilization jumped to 40% on all cores.  A third instance?  Yep, 60%.  The hardware can do it. 

 

It seems that either BIOS, or windows server is artificially capping applications to no more than 20% utilization per core.  The 20% thing I cannot find anything on.  My web searches for "low cpu utilization" turn up suggestions to look for malware and faulty hardware. This is a fresh re-build never connected to the web, so malware is out of the picture, and we already learned from the multi-instance test that faulty hardware isn't crippling the system. 

 

I'm a windows guy, and I've built every PC I have, but I don't know a thing about server hardware, and I don't know a thing about windows server.

 

Any help would be greatly appreciated!

 

 

[Advertisements]

I found this article but it's about setting limits

 

https://technet.micr...8(v=ws.10).aspx

 

Suppose it wouldn't hurt to verify that the registry values don't exist and perhaps experiment with higher values in an attempt to up the default limit.

 

Another one says Process Management isn't enforced until CPU usage is 70%

https://technet.micr...y/cc753280.aspx

[RKinner]

I found this article but it's about setting limits

 

https://technet.micr...8(v=ws.10).aspx

 

Suppose it wouldn't hurt to verify that the registry values don't exist and perhaps experiment with higher values in an attempt to up the default limit.

 

Another one says Process Management isn't enforced until CPU usage is 70%

https://technet.micr...y/cc753280.aspx

[PhysicsNinja]

Thanks for the response.

 

I played with management profiles for a few days, and ultimately removed the resource manager entirely.  No avail.  Out of curiosity I dug up an old copy of super-pi and launched it.  One core railed! 

 

So naturally I assumed it was something not optimized correctly in visual studio, so I wrote my own pi calculator using the dart-throwing method (10 lines of code).  Fired it off, and one core railed.  So, I decided to put my pi calculator in a thread and launch 16 instances...  16 cores railed, 100% CPU utilization.  So it's not throttling, and it is possible.

 

I do shuttle a lot of data around from memory.  I'm thinking what I'm seeing is a convolution of the reduced frequency at low CPU usage and the total bandwidth of the FSB.  I still can't quite figure out why multiple instances use more CPU than single instances since the speed at which information can be pulled out of RAM doesn't change much (The extra CPU load of additional instances does raise the clock frequency a bit).  Maybe there is a little CPU overhead to just sit and wait for data to get off the bus?  I did try tuning on memory mirroring to see if I got a speed boost like you do from a RAID-1, but it made no noticeable change, so the FSB is probably the bottleneck.  Running only a single thread of my simulation did use 100% of a core.  I turned on one thread a time and saw per-core utilization drop down from 100% at about 4 threads. 

 

I wrote a distributed-compute version of my simulation.  Basically, I had 4 instances all communicating with a control instance (Same code I'd run if I had 5 compute nodes connected by a network, but all were running on the same server), so I'm running 80 threads now...   It did hit 80+% utilization, but was not even close to the wall-clock speed of my quad-core machine.  It was apparently just a lot of CPU overhead, not much useful compute. 

 

I'm going to need to get a lot smarter about how I store information in memory and how I structure my code to utilize all 16 cores on this server.  Boo.  Hopefully it will prove more useful in training neural networks (more math, less data) else I wasted money on it. 

[RKinner]

Process Monitor might help figure out what's going on:

 

Download Process Monitor http://live.sysinter...com/Procmon.exe

 

By default it stores to RAM so eventually runs out of memory if you leave it up too long but perhaps you can get it to show you where it's wasting time.