VirtioFS slow file transfer speeds #1039

johnsanc314 · 2024-02-04T14:33:33Z

I am not sure if I have something misconfigured or if this is just expected behavior, but I am getting poor file transfer performance. Speeds are 45-60 MB/s whereas samba is around 275 MB/s with my setup. Large directory listings with virtiofs are much faster than samba however. Are these speeds expected? If not, is there anything I can do to improve performance?

To Reproduce
Transfer a single large file (~2 GB+) to or from the host OS using the mounted virtiofs drive.

Expected behavior
Speeds should ideally be comparable to samba

Host:

Distro: Unraid 6.12.6
Kernel version: Linux 6.1.64-Unraid x86_64
QEMU version: 7.2.0
libvirt version: 8.7.0

VM:

Windows 11
VirtioFS, 01/10/2024,100.6.101.58000

The VM XML file is pretty standard per the documentation I could find, noteworthy points below:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='9'>
  <name>Windows 11</name>
  <uuid>2b7d02e7-ce93-6934-5afb-641e9b93ab6e</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windows11"/>
  </metadata>
  <memory unit='KiB'>41943040</memory>
  <currentMemory unit='KiB'>41943040</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='13'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='14'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='15'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='16'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='17'/>
    <vcpupin vcpu='10' cpuset='6'/>
    <vcpupin vcpu='11' cpuset='18'/>
    <vcpupin vcpu='12' cpuset='7'/>
    <vcpupin vcpu='13' cpuset='19'/>
    <vcpupin vcpu='14' cpuset='8'/>
    <vcpupin vcpu='15' cpuset='20'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/2b7d02e7-ce93-6934-5afb-641e9b93ab6e_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/domains/iso/Win11_22H2_English_x64v2.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/domains/iso/virtio-win-0.1.240.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' queue='1024'/>
      <binary path='/usr/libexec/virtiofsd' xattr='on'>
        <cache mode='always'/>
        <sandbox mode='chroot'/>
      </binary>
      <source dir='/mnt/user/Games'/>
      <target dir='Games'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>
    <interface type='bridge'>
      <mac address='52:54:00:1f:a0:a3'/>
      <source bridge='br0'/>
      <target dev='vnet8'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-9-Windows 11/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
      <alias name='tpm0'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x33' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x33' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2f' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x32' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x33' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x33' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev5'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

mackid1993 · 2024-02-05T20:49:56Z

I can confirm, NVME transfers over VirtioFS run about 400 MB/s. Identical transfers in the same environment over SMB 3-5 GB/s. This is on Unraid QEMU 7.2 with the test VirtioFS drivers. @johnsanc314 also tested the latest Rust virtiofsd with similar results.

YanVugenfirer · 2024-02-06T10:11:27Z

Hi All,

This is a known issue: https://issues.redhat.com/browse/RHEL-1212

virtio-fs Windows driver implementation was not optimized for performance

johnsanc314 · 2024-06-15T15:22:35Z

I wanted to resurface this because its more than just "not optimized for performance", myself and others are observing near identical behavior on AMD based systems where a single CPU core will be utilized at 100% during the transfer and also observe slow transfer speeds. What info can we provide to help you diagnose the issue?

Unraid Forum post about AMD issues

YanVugenfirer · 2024-06-16T11:11:32Z

@johnsanc314 Thank you for suggestion to help the diagnosis, but unfortunately we know what is wrong and that the data path is not optimized. Windows Server 2025 support takes most of our time now, hopefully we can have time later on to handle the performance of virtio-fs

YanVugenfirer · 2024-12-03T13:48:44Z

Tracker: https://issues.redhat.com/browse/RHEL-1212

mackid1993 · 2024-12-03T14:17:23Z

Hey sorry, is this anything positive? I don't have a Redhat account so I can't view this link.

…

On Tue, Dec 3, 2024, at 8:49 AM, Yan Vugenfirer wrote: Tracker: https://issues.redhat.com/browse/RHEL-1212 — Reply to this email directly, view it on GitHub <#1039 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO43MYZPNZR7VK3HJTH3BL2DWZFFAVCNFSM6AAAAABS52ZKAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJUGYYDMOBWGI>. You are receiving this because you commented.Message ID: ***@***.***>

YanVugenfirer · 2024-12-03T14:22:48Z

@mackid1993 unfortunately no. Just some housekeeping of backlog and planning for the next year.

johnsanc314 added the Bug Bug label Feb 4, 2024

johnsanc314 assigned YanVugenfirer Feb 4, 2024

github-actions bot added Balloon virtio-fs labels Feb 4, 2024

YanVugenfirer added the Jira_Attached label Dec 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VirtioFS slow file transfer speeds #1039

VirtioFS slow file transfer speeds #1039

johnsanc314 commented Feb 4, 2024

mackid1993 commented Feb 5, 2024

YanVugenfirer commented Feb 6, 2024

johnsanc314 commented Jun 15, 2024

YanVugenfirer commented Jun 16, 2024

YanVugenfirer commented Dec 3, 2024

mackid1993 commented Dec 3, 2024 via email

YanVugenfirer commented Dec 3, 2024

VirtioFS slow file transfer speeds #1039

VirtioFS slow file transfer speeds #1039

Comments

johnsanc314 commented Feb 4, 2024

mackid1993 commented Feb 5, 2024

YanVugenfirer commented Feb 6, 2024

johnsanc314 commented Jun 15, 2024

YanVugenfirer commented Jun 16, 2024

YanVugenfirer commented Dec 3, 2024

mackid1993 commented Dec 3, 2024 via email

YanVugenfirer commented Dec 3, 2024