Facts (please identify any false statements):
I have a 100 Mbps connection between two sites that are 80 ms apart
This is a long fat connection that could benefit from a large TCP window size perhaps up to 100 Mbps * 0.08 sec = 1,000,000 bytes
Both machines are running Windows Server 2012. “Receive window auto tuning level” is normal on both. “Window scaling heuristics” are disabled on both.
I ran “iperf -s” on one side and “iperf -c” on the other. The transfer happened at 5 Mbps. I get the same result going the other direction.
Both sides advertised support for TCP sliding windows in their SYNs.
The receiver requested a TCP window size of 64,512 bytes (0xFC00)
during the entire run with a TCP window scale value of “no shift”
(0x000).The network was able to handle a larger window size (see sequence
diagrams below)The receiver kept the window smaller than the network supports
This connection is happening within an IPSEC VPN. MTU of the tunnel interface is reduced to 1400 bytes in both directions.
Question
- Why is the receiver keeping the window small?
Non-Answers
The network is broken
Linux machines running on the same network open the TCP window to 1.5 megabytes and transmit data at 6 times the bandwidth
Window scaling heuristics are enabled
Window scaling heuristics are disabled (see output of “netsh interface tcp show heuristics” below)
Receive Window Auto-Tuning Level is not normal
Receive Window Auto-Tuning Level is normal (see output of “netsh interface tcp show global” below)
This just doesn’t work well on a virtual machine within ESXi
I get 6 times better performance on a virtual linux machine running on the same host.
Update 1 June 12, 2015 4:30 pm PDT
I modified the test by putting linux on one side of the connection. Sure enough, when linux sends data to Windows Server 2012, Windows offers a too-small TCP receive window (64,512 bytes).
When I send data from Windows to linux, linux offers a large-enough TCP receive window (1,365,120 bytes). However, Windows restricts sends to max ~60,000 bytes in flight.
Update 2 June 13, 2015 3:00 pm PDT
A step closer to root cause. In my setup, neither SO_SNDBUF nor SO_RCVBUF are set (by iperf). These are the send and receive buffers which effectively bound the receive window. When not specifying these values, Windows Server 2012 provide a default value of 64 kB. So the question is now:
Question
- When one is not specified, why isn’t Windows Server 2012 dynamically increasing SO_SNDBUF/SO_RCVBUF to accommodate long fat pipes as described at MSDN?
Non-answers
“netsh winsock show autotuning” is disabled
It is enabled.
Update 3 August 24, 2015 4:00 pm PDT
netsh apparently has been replaced with Set-NetTCPSetting and family. Get-NetTCPSetting combined with Get-NetTCPConnection shows I am operating in the ‘Internet’ regime which offers me these settings:
SettingName : Internet MinRto(ms) : 300 InitialCongestionWindow(MSS) : 4 CongestionProvider : CTCP CwndRestart : False DelayedAckTimeout(ms) : 50 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384
Sender TCP Settings
PS C:\Users\acs> netsh interface tcp show global Querying active state... TCP Global Parameters ---------------------------------------------- Receive-Side Scaling State : enabled Chimney Offload State : disabled NetDMA State : disabled Direct Cache Access (DCA) : disabled Receive Window Auto-Tuning Level : normal Add-On Congestion Control Provider : none ECN Capability : enabled RFC 1323 Timestamps : disabled Initial RTO : 3000 Receive Segment Coalescing State : enabled PS C:\Users\acs> netsh interface tcp show heuristics TCP Window Scaling heuristics Parameters ---------------------------------------------- Window Scaling heuristics : disabled Qualifying Destination Threshold : 3 Profile type unknown : normal Profile type public : normal Profile type private : normal Profile type domain : normal PS C:\Users\acs> Get-NetTCPSetting SettingName : Automatic MinRto(ms) : InitialCongestionWindow(MSS) : CongestionProvider : CwndRestart : DelayedAckTimeout(ms) : MemoryPressureProtection : AutoTuningLevelLocal : AutoTuningLevelGroupPolicy : AutoTuningLevelEffective : EcnCapability : Timestamps : InitialRto(ms) : ScalingHeuristics : DynamicPortRangeStartPort : DynamicPortRangeNumberOfPorts : SettingName : Custom MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Compat MinRto(ms) : 300 InitialCongestionWindow(MSS) : 2 CongestionProvider : Default CwndRestart : False DelayedAckTimeout(ms) : 200 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Datacenter MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Internet MinRto(ms) : 300 InitialCongestionWindow(MSS) : 4 CongestionProvider : CTCP CwndRestart : False DelayedAckTimeout(ms) : 50 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384Sender SYN
No. Time Source Destination Protocol Length Delta Sequence number Acknowledgment number Bytes in flight Calculated window size Info 814 5.036577000 10.10.0.21 10.11.0.1 TCP 66 0.000000000 0 0 64512 49758→5001 [SYN, ECN, CWR] Seq=0 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1 Frame 814: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0 Ethernet II, Src: 00:11:22:33:44:55, Dst: aa:bb:cc:dd:ee:ff Internet Protocol Version 4, Src: 10.10.0.21 (10.10.0.21), Dst: 10.11.0.1 (10.11.0.1) Transmission Control Protocol, Src Port: 49758 (49758), Dst Port: 5001 (5001), Seq: 0, Len: 0 Source Port: 49758 (49758) Destination Port: 5001 (5001) [Stream index: 73] [TCP Segment Len: 0] Sequence number: 0 (relative sequence number) Acknowledgment number: 0 Header Length: 32 bytes .... 0000 1100 0010 = Flags: 0x0c2 (SYN, ECN, CWR) Window size value: 64512 [Calculated window size: 64512] Checksum: 0x1451 [validation disabled] Urgent pointer: 0 Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted Maximum segment size: 1460 bytes No-Operation (NOP) Window scale: 0 (multiply by 1) Kind: Window Scale (3) Length: 3 Shift count: 0 [Multiplier: 1] No-Operation (NOP) No-Operation (NOP) TCP SACK Permitted Option: TrueSender perspective of sequence graph
Receiver TCP Settings
PS C:\Users\acs> netsh interface tcp show global Querying active state... TCP Global Parameters ---------------------------------------------- Receive-Side Scaling State : enabled Chimney Offload State : disabled NetDMA State : disabled Direct Cache Access (DCA) : disabled Receive Window Auto-Tuning Level : normal Add-On Congestion Control Provider : none ECN Capability : enabled RFC 1323 Timestamps : disabled Initial RTO : 3000 Receive Segment Coalescing State : enabled PS C:\Users\acs> netsh interface tcp show heuristics TCP Window Scaling heuristics Parameters ---------------------------------------------- Window Scaling heuristics : disabled Qualifying Destination Threshold : 3 Profile type unknown : normal Profile type public : normal Profile type private : normal Profile type domain : normal PS C:\Users\acs> Get-NetTCPSetting SettingName : Automatic MinRto(ms) : InitialCongestionWindow(MSS) : CongestionProvider : CwndRestart : DelayedAckTimeout(ms) : MemoryPressureProtection : AutoTuningLevelLocal : AutoTuningLevelGroupPolicy : AutoTuningLevelEffective : EcnCapability : Timestamps : InitialRto(ms) : ScalingHeuristics : DynamicPortRangeStartPort : DynamicPortRangeNumberOfPorts : SettingName : Custom MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Compat MinRto(ms) : 300 InitialCongestionWindow(MSS) : 2 CongestionProvider : Default CwndRestart : False DelayedAckTimeout(ms) : 200 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Datacenter MinRto(ms) : 20 InitialCongestionWindow(MSS) : 4 CongestionProvider : DCTCP CwndRestart : True DelayedAckTimeout(ms) : 10 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384 SettingName : Internet MinRto(ms) : 300 InitialCongestionWindow(MSS) : 4 CongestionProvider : CTCP CwndRestart : False DelayedAckTimeout(ms) : 50 MemoryPressureProtection : Enabled AutoTuningLevelLocal : Normal AutoTuningLevelGroupPolicy : NotConfigured AutoTuningLevelEffective : Local EcnCapability : Enabled Timestamps : Disabled InitialRto(ms) : 3000 ScalingHeuristics : Disabled DynamicPortRangeStartPort : 49152 DynamicPortRangeNumberOfPorts : 16384Receiver SYN
No. Time Source Destination Protocol Length Delta Sequence number Acknowledgment number Bytes in flight Calculated window size Info 817 5.110501000 10.11.0.1 10.10.0.21 TCP 70 0.073924000 0 1 64512 5001→49758 [SYN, ACK, ECN] Seq=0 Ack=1 Win=64512 Len=0 MSS=1460 WS=1 SACK_PERM=1 [ETHERNET FRAME CHECK SEQUENCE INCORRECT] Frame 817: 70 bytes on wire (560 bits), 70 bytes captured (560 bits) on interface 0 Ethernet II, Src: aa:bb:cc:dd:ee:ff, Dst: 00:11:22:33:44:55 Internet Protocol Version 4, Src: 10.11.0.1 (10.11.0.1), Dst: 10.10.0.21 (10.10.0.21) Transmission Control Protocol, Src Port: 5001 (5001), Dst Port: 49758 (49758), Seq: 0, Ack: 1, Len: 0 Source Port: 5001 (5001) Destination Port: 49758 (49758) [Stream index: 73] [TCP Segment Len: 0] Sequence number: 0 (relative sequence number) Acknowledgment number: 1 (relative ack number) Header Length: 32 bytes .... 0000 0101 0010 = Flags: 0x052 (SYN, ACK, ECN) Window size value: 64512 [Calculated window size: 64512] Checksum: 0xb5bb [validation disabled] Urgent pointer: 0 Options: (12 bytes), Maximum segment size, No-Operation (NOP), Window scale, No-Operation (NOP), No-Operation (NOP), SACK permitted Maximum segment size: 1460 bytes No-Operation (NOP) Window scale: 0 (multiply by 1) Kind: Window Scale (3) Length: 3 Shift count: 0 [Multiplier: 1] No-Operation (NOP) No-Operation (NOP) TCP SACK Permitted Option: True [SEQ/ACK analysis]Receiver perspective of sequence graph
TCP Window
Answer
I’ve seen this as a driver-specific issue; in my case with QLogic network controllers that were attempting to use TCPChimney. This link describes the TCPChimney functionality added in Windows 2008 – but I’m pretty sure it still applies: https://support.microsoft.com/en-us/kb/951037
I would recommend testing the following, in order; after each test, reboot and see whether the receiver starts increasing the TCP RWIN as expected.
1) Load the latest versions of the drivers for the network adapter on the receiving computer.
1) Disable TCPChimney on the receiving computer
2) Disable all ‘TCP Receive’ offloading. This would be found in Advanced settings of the Network Adapter Properties (the same area where Speed & Duplex would be set)
3) Disable all ‘TCP Send’ offloading (also in the Network Adapter’s Advanced properties)
(
And contrary to the comment “And big TCP window sizes over 65k are bad for servers, as then the memory demand for connections increases. 65k alone might also not make you happy enough. – user303507 Aug 6 ’15 at 11:30”, large TCP Receive Windows are NOT inherently bad for the server. In the case of high-bandwidth, high-latency links (like Satellite relays), large RWIN values are necessary so that we have more TCP data “in the pipe”. Imagine a 600 Mbps connection with 3000 ms latency; the high-bandwidth link would be limited to about 20 KBps ;as only 65 KB of un-acked TCP data could be “in the pipe” at a time.
)
Attribution
Source : Link , Question Author : Chris Stankevitz , Answer Author : Jason Walker
