| Author |
 Topic  |
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
 Posted - 02/12/2009 : 2:03:06 PM
|
Has anybody done any testing as far as the differences between a bare metal 2008 install and 2008 inside a Hyper-V VM? How much slower are you seeing VM hard disk performance compared to physical?
I've run some HD benchmarking tools, but they tend to skew inside the VM and show 700-800% improvements due to caching or whatever. So, I thought I'd setup a quick and dirty test by just robocopying some files.
The hardware is an ASUS M2N-Deluxe with an AMD Dual core 5600 and 6GB of RAM. Two Samsung SATA-II drives (see below for details). My test was simply to robocopy the Server 2008 DVD from the C: drive to the D: drive.
Bare Metal 2008x64: 3988/Min (66MB/sec)
C: Drive - Physical disk 1
D: Drive - Physical disk 2
2008x64 VM inside Hyper-V: 2500MB/min (41MB/sec)
C: Drive - VHD File located on Physical disk 1
D: Drive - VHD File located on Physical disk 2
I'm aware that RAID-5/10 will certainly help, but this is for my home server. I'm likely to be installing a 3Ware SATA-II RAID5 at some point to host my data VHD's, but nonetheless I figured comparing bare metal to VM using the same physical drives would give a good indication of the overhead imposed by the VHD's.
Do these numbers sound about right or match what anybody else is seeing?
|
Edited by - sixdoubleo on 02/12/2009 10:55:58 PM |
|
|
wkasdo
Moderator
Netherlands
6140 Posts
Status: offline |
Posted - 02/12/2009 : 2:06:00 PM
|
| Sounds a bit low. Did you use fixed .vhd's to optimize performance? Did you make sure the physical disks were otherwise idle? |
 |
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/12/2009 : 2:19:18 PM
|
Yeah, this is the only VM running on the box and I made sure everything was nice and quiet. However, the host OS for Hyper-V is on the same physical disk as the VM's C: drive VHD file.
Would there be a big benefit to introducing a third disk... So I'd have:
Physical Disk 1: Server 2008 Hyper-V Host OS Only
Physical Disk 2: VM OS Drive VHD Files (C:)
Physical Disk 3: VM Data Drive VHD Files (D:)
I'll check whether they're fixed or dynamic...I know for a fact the D: drive is fixed, but the C: drive might be dynamic since the wizard created it. Also, the D: drive is on a Virtual SCSI adapter whereas the C: drive is on the Virtual IDE Adapter. Is there a huge difference in performance between the two? |
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 02/12/2009 : 3:18:51 PM
|
I'm benchmarking exactly this situation right now, albeit using ESXi and external iSCSI RAID. I will blog the results next week. From what I gathered so far your figures for virtual do seem a bit low.
Re VirtSCSI vs. VirtIDE. As of Virtual Server 2005 R2, there was a huge difference in performance between the two and there certainly is one in the VMWare world.
BTW: I am well aware of you asking for Hyper-V related, not ESX related, experience. Sorry for not being able to provide any but this will have to wait the way the things are here. |
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
|
|
|
wkasdo
Moderator
Netherlands
6140 Posts
Status: offline |
Posted - 02/12/2009 : 3:23:38 PM
|
| No, for hyper-v there is NO difference between virtual IDE and SCSI. They share the same IO stack, if you have the Hyper-v additions of course. |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/12/2009 : 3:34:37 PM
|
quote:
Originally posted by cj_berlin
I'm benchmarking exactly this situation right now, albeit using ESXi and external iSCSI RAID. I will blog the results next week. From what I gathered so far your figures for virtual do seem a bit low.
Re VirtSCSI vs. VirtIDE. As of Virtual Server 2005 R2, there was a huge difference in performance between the two and there certainly is one in the VMWare world.
BTW: I am well aware of you asking for Hyper-V related, not ESX related, experience. Sorry for not being able to provide any but this will have to wait the way the things are here.
You are right about the difference between IDE and SCSI with 2005...but my understanding is that Hyper-V uses a pass-through driver on the IDE and thus its performance is equal to that of the Synthetic SCSI driver. Furthermore, to the best of my knowledge you can't boot from SCSI in Hyper-V...so for the system drive, you're stuck with IDE.
I'm actually VERY much interested in your findings as I'm doing the same as you. I haven't decided on Hyper-V or ESX yet for my home server. I'm going to install ESXi and perform the same tests as well.
Maybe we can sort of agree on a common set of tests so we can compare numbers. If you find a simple HD benchmarking tool that seems to report properly inside a VM, let me know. I was using HDTune to check VM's at work (we use ESX) and it was working properly. I fired it up in Hyper-V and it's showing insanely high numbers (900MB/sec transfer). So for now I'm robocopying a standard set of files.
As it stands, I'm leaning towards Hyper-V simply because consumer motherboards and RAID-5 controllers that are in my price range are not well supported with ESX or ESXi, whereas Hyper-V supports them by virtue of the large Vista/2008 driver base.
Definitely let me know how it goes! :) |
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 02/12/2009 : 4:22:12 PM
|
quote:
Originally posted by wkasdo
No, for hyper-v there is NO difference between virtual IDE and SCSI. They share the same IO stack, if you have the Hyper-v additions of course.
This is good to know. Aidan should include this in his white paper. |
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 02/12/2009 : 4:32:44 PM
|
quote:
Originally posted by sixdoubleo
If you find a simple HD benchmarking tool that seems to report properly inside a VM, let me know. I was using HDTune to check VM's at work (we use ESX) and it was working properly.
For the first impression we are using
http://www.estesoft.com/diskbenchmark/default.aspx
because it lets you view multiple usage profiles at a glance. Definitely worth a try. I have NO idea, though, how this one will behave in a Hyper-V VM.
If it comes to benchmarking for specific applications (SQL, Exchange, Web server etc.)... we tend to use the very application and a stress generator. It has proved very difficult to deduce the suitability of a storage subsystem for large SQL databases from the classical benchmarking data whereas putting a 200 GB heavily fragmented MDF on the array and shuffling BLOBs around while searching for a short string within text fields gives you an impression of what to expect. |
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
Edited by - cj_berlin on 02/12/2009 4:34:03 PM |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/12/2009 : 5:07:03 PM
|
OK, tweaked things a bit and now I'm getting better numbers.
Turns out both of my disks were actually dynamic. I remember converting them but evidently I never mounted the converted drives. So I setup a new VM with both the C and D setup as fixed.
Also, in the VM I checked "enable advanced performance" in the Policies tab of the disk drive under Device Manager. (I assume this is write caching)
My updated numbers are:
Bare Metal 2008x64: 3988MB/Min (66MB/sec)
C: Drive - Physical disk 1
D: Drive - Physical disk 2
2008x64 VM inside Hyper-V (Original): 2500MB/min (41MB/sec)
2008x64 VM inside Hyper-V (Tweaked) : 3971MB/min (66.1MB/sec)
C: Drive - VHD File located on Physical disk 1
D: Drive - VHD File located on Physical disk 2
MUCH more acceptable and almost identical numbers physical vs virtual and that's what I was after. But I thought the difference between Dynamic and Fixed was more like 10-15%... In this case it's a little more like 40%.
|
Edited by - sixdoubleo on 02/12/2009 5:15:44 PM |
|
|
|
joe_elway
Honorable But Hopeless Addict
Ireland
6673 Posts
Status: offline |
Posted - 02/12/2009 : 6:50:33 PM
|
Mark Russinovich said at TechEd EMEA that MS tests showed VHD having a 2% drop in performance over the same disk as bare metal, way better than originally hoped. I think they wanted to get within 10%.
IDE is supposed to be slower than SCSI based on this:
http://blogs.msdn.com/tvoellm/archive/2007/12/12/which-is-better-ide-or-scsi-windows-server-virtualization-08-code-name-viridian-controller-performance.aspx
"The IDE controller implements a well-known IDE controller and this means there is extra processing before the I/O is sent to the disk. This processing occurs in vmwp.exe (a user mode process that exists for each started VM. More on this in a later post). Once the IDE emulation is complete the I/O is sent into the Root Partition’s I/O Stack. I/O completion requires a trip back to vmwp.exe.
The SCSI controller is not emulated. The SCSI controller uses VMBUS (Virtual Machine BUS. More on this in a later post). The I/O's pass from the Child (aka Guest) Partition to the root over VMBUS and enter the I/O stack. You can already see one less process/machine context switch is required because vmwp.exe does not get invoked. Once and I/O completes its completion is sent over VMBUS."
I don't have the internals knowledge that Willem has so I will always bow to Willem's word. |
Aidan Finn
MCSE, MVP (Virtual Machine: Systems Administration)
IT Blog: http://www.aidanfinn.com
My Photography: http://www.aidanfinnphoto.com/
My Hyper-V Book: Mastering Hyper-V Deployment
Twitter: http://twitter.com/joe_elway |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/12/2009 : 7:36:31 PM
|
Yeah, a 2% drop is awesome...way better than I was hoping for. Now that I have my VM's tweaked, I'm seeing pretty much negligible differences between VHD and bare-metal.
Regarding the IDE vs. SCSI, that certainly makes sense. But if the IDE doesn't perform as well, why did they make it the only option for your boot drive? So you're basically forced to stick your OS on the inferior interface. While it will perform more than adequately for my uses, I'd be concerned about how it would scale to enterprise-class loads. Does a virtual IDE interface cap out at physical IDE speeds?
|
Edited by - sixdoubleo on 02/12/2009 7:44:22 PM |
|
|
|
joe_elway
Honorable But Hopeless Addict
Ireland
6673 Posts
Status: offline |
Posted - 02/13/2009 : 03:40:56 AM
|
quote:
Originally posted by sixdoubleo
But if the IDE doesn't perform as well, why did they make it the only option for your boot drive?
Unless we work in that team, I think we'd be guessing. I have my suspicions but they are guesses.
quote:
Does a virtual IDE interface cap out at physical IDE speeds?
I would say no. Remember that we get a hit (if my previous link was correct) but that's a software hit. We're still using the speeds of the underlying physical hardware.
For data loads that require better performance, e.g. DB log files, exchange database files then you should use additional SCSI disks. You don't need to create one monster IDE drive and partition it. What I do, even for small loads because they may grow, is add additional SCSI disks. So a SQL box might be:
- C: IDE 1
- Database Log on D: SCSI 1
- Database File on E: SCSI 2
I could take that further (but haven't yet):
- C: IDE 1
- Paging File on D: SCSI 1
- Database Log on E: SCSI 2
- Database File on F: SCSI 3
(All fixed size VHD)
And for those virtual machines where we need the best possible I/O performance, just use passthrough disks. |
Aidan Finn
MCSE, MVP (Virtual Machine: Systems Administration)
IT Blog: http://www.aidanfinn.com
My Photography: http://www.aidanfinnphoto.com/
My Hyper-V Book: Mastering Hyper-V Deployment
Twitter: http://twitter.com/joe_elway |
|
|
|
wkasdo
Moderator
Netherlands
6140 Posts
Status: offline |
Posted - 02/13/2009 : 07:22:38 AM
|
| I have a whitepaper with extensive IO measurements. In all of them the enlightened IDE and SCSI drives are equal (<1%) in performance: IOPS, throughput and latency. Legacy IDE is waaay slower of course. |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/13/2009 : 1:50:11 PM
|
Just wanted to throw an update in here...Now that I've got Hyper-V matching bare metal hard disk performance on my box, I decided to turn my attention to ESXi and try to get a good comparison.
I went ahead and pulled out the Server 2008/Hyper-V disks and set them aside so I could go back to them later. Then I put some fresh disks on the same hardware and installed ESXi to perform the same tests. After about 12 installs of ESXi (mostly fighting with IDE/SATA driver issues and it not recognizing controllers) I finally got a solid setup. I ended up actually going with THREE physical disks just to keep everything isolated...so in this respect my ESXi setup has a slight advantage over Hyper-V in that the host OS has its own drive:
Disk 1 (Samsung SATA-II 500GB) - ESXi and its related partitions
Disk 2 (Samsung SATA-II 1TB)- VMFS volume to hold OS VMDK's
Disk 3 (Samsung SATA-II 1TB)- VMFS volume to hold Data VMDK's
I setup a Windows Server 2008 64-bit VM just like I did with Hyper-V
1 CPU
1024MB RAM
127GB C: Drive - VMDK on VMFS volume on Physical Disk 2
127GB D: Drive - VMDK on VMFS volume on Physical Disk 3
Installed VMWare Tools
Latest MS Patches
Test Procedure:
Copy the contents of the Server 2008 64-bit DVD to C:\Server2008DVD
Reboot (to clear any cache)
Issue the command: "robocopy C:\Server2008DVD D:\Server2008DVD /E" and note the MB/Min value reported at the end.
After about a dozen tests, tweaking, rebooting the VM, trying new VM's, rebooting the ESXi server, trying to let everything stabilize, I am simply not able to get the speed up above 1500MB/Min (25MB/sec).
Again my tests with 2008 Bare Metal and 2008 VM in Hyper-V are more than twice as fast....with Hyper-V matching the bare-metal performance.
Bare Metal 2008x64: 3988MB/Min (66.5MB/sec)
2008x64 VM inside Hyper-V : 3971MB/min (66.1MB/sec)
I'm pretty certain that what I'm seeing is simply ESX's lack of support for consumer-based SATA controllers. It seems like the controller is running in compatibility mode. I had to modify the install ISO and insert my own customized oem.tgz to allow it to load the correct SATA driver for my particular PCI ID. Given enterprise-class hardware, I'm sure the numbers would be much different. But since I"m building a home server, I care about SATA performance.
Anybody have any ideas for increasing the ESXi numbers? Googling around for ESX/ESXi and SATA performance, you see a lot of complaints, but not much in the way of solutions. I get the sense that many of the folks doing this are just happy to get it running at ALL and are just using it as a lab/learning tool and aren't too concerned about performance. |
Edited by - sixdoubleo on 02/13/2009 1:52:18 PM |
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 02/13/2009 : 1:59:11 PM
|
Dunno.
On a Proliant ML115, using the onboard Intel SATA controller, I am getting numbers close to what you have quoted above for Hyper-V.
On a ThinkCentre M57 with a Sil 312 PCI-X SATA Controller I get about 50 MB/s both on ESXi and bare metal.
Trying to cheat ESXi on the storage side has always been asking for trouble, or at least for degraded performance.
I'll post my results next week. |
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 02/13/2009 : 2:04:34 PM
|
quote:
Originally posted by cj_berlin
Trying to cheat ESXi on the storage side has always been asking for trouble, or at least for degraded performance.
Very true...but I don't think I'm necessarily cheating it...I'm giving it three dedicated drives. Given that Hyper-V is able to produce respectable numbers with the same exact hardware makes me suspicious of ESXi's driver support for that controller. I have a feeling their Intel SATA support might be a little more mature.
|
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 03/10/2009 : 10:01:33 PM
|
quote:
Originally posted by cj_berlin
Dunno.
On a Proliant ML115, using the onboard Intel SATA controller, I am getting numbers close to what you have quoted above for Hyper-V.
Hey Evgenij,
I just thought I'd check and see if you've made any headway in your ESXi testing?
I ended up putting a Dell PERC 5 SAS controller in this thing in hopes that it would improve ESXi's performance...no luck. I'm still getting around 2000MB/Min Robocopy speeds from ESXi, whereas I see close to 4000MB/Min with Hyper-V.
|
|
|
|
deandownsouth
Old Timer
USA
415 Posts
Status: offline |
Posted - 03/10/2009 : 10:25:00 PM
|
quote:
Originally posted by sixdoubleo
Just wanted to throw an update in here...Now that I've got Hyper-V matching bare metal hard disk performance on my box, I decided to turn my attention to ESXi and try to get a good comparison.
I went ahead and pulled out the Server 2008/Hyper-V disks and set them aside so I could go back to them later. Then I put some fresh disks on the same hardware and installed ESXi to perform the same tests. After about 12 installs of ESXi (mostly fighting with IDE/SATA driver issues and it not recognizing controllers) I finally got a solid setup. I ended up actually going with THREE physical disks just to keep everything isolated...so in this respect my ESXi setup has a slight advantage over Hyper-V in that the host OS has its own drive:
Disk 1 (Samsung SATA-II 500GB) - ESXi and its related partitions
Disk 2 (Samsung SATA-II 1TB)- VMFS volume to hold OS VMDK's
Disk 3 (Samsung SATA-II 1TB)- VMFS volume to hold Data VMDK's
I setup a Windows Server 2008 64-bit VM just like I did with Hyper-V
1 CPU
1024MB RAM
127GB C: Drive - VMDK on VMFS volume on Physical Disk 2
127GB D: Drive - VMDK on VMFS volume on Physical Disk 3
Installed VMWare Tools
Latest MS Patches
Test Procedure:
Copy the contents of the Server 2008 64-bit DVD to C:\Server2008DVD
Reboot (to clear any cache)
Issue the command: "robocopy C:\Server2008DVD D:\Server2008DVD /E" and note the MB/Min value reported at the end.
After about a dozen tests, tweaking, rebooting the VM, trying new VM's, rebooting the ESXi server, trying to let everything stabilize, I am simply not able to get the speed up above 1500MB/Min (25MB/sec).
Again my tests with 2008 Bare Metal and 2008 VM in Hyper-V are more than twice as fast....with Hyper-V matching the bare-metal performance.
Bare Metal 2008x64: 3988MB/Min (66.5MB/sec)
2008x64 VM inside Hyper-V : 3971MB/min (66.1MB/sec)
I'm pretty certain that what I'm seeing is simply ESX's lack of support for consumer-based SATA controllers. It seems like the controller is running in compatibility mode. I had to modify the install ISO and insert my own customized oem.tgz to allow it to load the correct SATA driver for my particular PCI ID. Given enterprise-class hardware, I'm sure the numbers would be much different. But since I"m building a home server, I care about SATA performance.
Anybody have any ideas for increasing the ESXi numbers? Googling around for ESX/ESXi and SATA performance, you see a lot of complaints, but not much in the way of solutions. I get the sense that many of the folks doing this are just happy to get it running at ALL and are just using it as a lab/learning tool and aren't too concerned about performance.
Just so I understand this, are you copying from the actual physical DVD to the C: drive in the guest or is it a mounted iso image? |
Mark Dean
|
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 03/11/2009 : 03:26:32 AM
|
quote:
Originally posted by sixdoubleo
quote:
Originally posted by cj_berlin
Dunno.
On a Proliant ML115, using the onboard Intel SATA controller, I am getting numbers close to what you have quoted above for Hyper-V.
Hey Evgenij,
I just thought I'd check and see if you've made any headway in your ESXi testing?
Hi there,
I bricked one of my lab servers last Friday so it would offer a good opportunity to do some perf testing on it after I have replaced the motherboard.
I won't be doing any Hyper-V benchmarking though, just ESXi vs. bare metal. |
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
|
|
|
cj_berlin
Honorable But Hopeless Addict
Germany
3098 Posts
Status: offline |
Posted - 03/11/2009 : 03:27:55 AM
|
quote:
Originally posted by deandownsouth
Just so I understand this, are you copying from the actual physical DVD to the C: drive in the guest or is it a mounted iso image?
Mark,
I don't think it's important because this copy operation doesn't go into the result that he compares to other platforms.
|
Evgenij Smirnov
MISSION CRITICAL Business IT Services Ltd.
Revolutionising the PC desktop at a fixed rate (in German only):
PCohnePC.DE |
|
|
|
sixdoubleo
Old Timer
USA
738 Posts
Status: offline |
Posted - 03/11/2009 : 10:47:01 AM
|
quote:
Just so I understand this, are you copying from the actual physical DVD to the C: drive in the guest or is it a mounted iso image?
Mark,
I'm not copying from a DVD of any kind...physical or virtual. I have the contents of the 2008 64-bit DVD in a folder on a network share, which I then pull down to the C: drive. Then I reboot (host and guest) to eliminate any caching effects.
Then I perform a timed robocopy of these contents from the C: drive to the D: drive. "robocopy c:\server2008dvd d:\test1 /E".
C: and D: are created so that they are on different physical disks.
Hopefully that makes sense. |
Edited by - sixdoubleo on 03/11/2009 10:47:18 AM |
|
|
| |
Topic |
|
Bare metal vs Hyper-V and ESX disk performance
