Failed attempt to tune long distance file transfer across continent between Linux and MacOS [技術]
Conditions
I live in Japan, using a Macbook with MacOS Catalina. A CentOS file server exists at the other end of the Eurasian continent, 9 time zones away. It is maintained by some other organization and I have no control over its behaviour, except by configuring parameters on the client side.
I want to pull 30+ files, >1GB in total, from the file server over HTTPS using curl. It completes quite fast if I use a Linux on Intel NUC that resides in our office. It takes less than 20 minutes. However, it takes more than 6 hours from my Macbook with the same network conditions, so my Macbook is more than 10 times slower than the Linux box with regard to file transfer speed in this specific setup. If I pull a huge file from a CDN site with my Macbook, it is very fast. So the LAN and the local ISP line are fine at least.
To ensure the User-Agent does not affect the transfer speed, I used Linux UA on Mac and vice versa and measured the speed using curl on both platforms. It did not make any difference: Linux client is always 10+ times faster than Macbook client.
Observations
So I captured packets on my Macbook as well as on the Linux in the office and compared them. There are several notable differences in the two packet captures.
1. Macbook starts TCP handshake with ECN and CWR flags turned on. This is the indication that the client wants to establish a TCP connection with ECN (Explicit Congestion Notification). (blog article, Wikipedia)
On the other hand, Linux client initiates TCP handshake without ECN flags.
2. Packets taken on the Linux box have lots of TCP segments with their sizes much larger than the usual segment size (like 12,378 bytes instead of around 1,400 bytes). If I specify "tcp.len > 1448" display filter in Wireshark, it filters out all packets from the Mac pcap. On the other hand, a display filter of "tcp.len > 10000" shows more than 80 packets on the Linux pcap. This is an indication that the file server (Linux) uses TCP segmentation offload against the Linux client, but not against MacOS.
Maybe the server does not like the advertised window size value, 65535, larger than its own, 28960, or maybe ECN is the root cause, or maybe something else. Who knows. This needs experimenting.
Increase mbuf clusters
This article describes a way to increase mbuf clusters on MaOS. Recent versions of MacOS has SIP, and you need to boot to the recovery mode if you want to change boot-args by the nvram command. Then also execute the sysctl commands to tune TCP parameters as described in the page.
Result: no improvements. File transfer using curl seems slower with the changes.
TCP parameters
This article is a bit outdated, but has some TCP tuning tips. I have tried them along with the above technique, but it did not help speedup the transfer either.
That's as far as I could get. Does anyone know how such file transfers can be sped up? I am hoping that there is a way for MacOS client to cause the Linux file server to use TCP segmentation offload. I just cannot find out how.
I live in Japan, using a Macbook with MacOS Catalina. A CentOS file server exists at the other end of the Eurasian continent, 9 time zones away. It is maintained by some other organization and I have no control over its behaviour, except by configuring parameters on the client side.
I want to pull 30+ files, >1GB in total, from the file server over HTTPS using curl. It completes quite fast if I use a Linux on Intel NUC that resides in our office. It takes less than 20 minutes. However, it takes more than 6 hours from my Macbook with the same network conditions, so my Macbook is more than 10 times slower than the Linux box with regard to file transfer speed in this specific setup. If I pull a huge file from a CDN site with my Macbook, it is very fast. So the LAN and the local ISP line are fine at least.
To ensure the User-Agent does not affect the transfer speed, I used Linux UA on Mac and vice versa and measured the speed using curl on both platforms. It did not make any difference: Linux client is always 10+ times faster than Macbook client.
Observations
So I captured packets on my Macbook as well as on the Linux in the office and compared them. There are several notable differences in the two packet captures.
1. Macbook starts TCP handshake with ECN and CWR flags turned on. This is the indication that the client wants to establish a TCP connection with ECN (Explicit Congestion Notification). (blog article, Wikipedia)
On the other hand, Linux client initiates TCP handshake without ECN flags.
2. Packets taken on the Linux box have lots of TCP segments with their sizes much larger than the usual segment size (like 12,378 bytes instead of around 1,400 bytes). If I specify "tcp.len > 1448" display filter in Wireshark, it filters out all packets from the Mac pcap. On the other hand, a display filter of "tcp.len > 10000" shows more than 80 packets on the Linux pcap. This is an indication that the file server (Linux) uses TCP segmentation offload against the Linux client, but not against MacOS.
| Server (far away) | Client 1 | Client 2 | ||
|---|---|---|---|---|
| OS | CentOS | CentOS 6.8 | MacOS Catalina | |
| Application | Apache 2.4.6 | Curl 7.19.7 | Curl 7.64.1 | |
| File Transfer Speed from the Server | - | ~1.2MB/s | <100KB/s | |
| TCP Header | Window Size Value | 28960 | 14600 | 65535 |
| Explicit Congestion Notification | on if asked | off | on | |
| Segmentation Offload (server) | - | on | off | |
| Maximum Segment Size | 1380 for Client 1 1460 for Client 2 | 1460 | 1460 | |
| Window Scale | 7 | 7 | 6 | |
| Selective ACK | permitted | permitted | permitted | |
| Max Ethernet Frame Received by tcpdump | - | 16,482 bytes | 1,514 bytes | |
Maybe the server does not like the advertised window size value, 65535, larger than its own, 28960, or maybe ECN is the root cause, or maybe something else. Who knows. This needs experimenting.
Increase mbuf clusters
This article describes a way to increase mbuf clusters on MaOS. Recent versions of MacOS has SIP, and you need to boot to the recovery mode if you want to change boot-args by the nvram command. Then also execute the sysctl commands to tune TCP parameters as described in the page.
Result: no improvements. File transfer using curl seems slower with the changes.
TCP parameters
This article is a bit outdated, but has some TCP tuning tips. I have tried them along with the above technique, but it did not help speedup the transfer either.
That's as far as I could get. Does anyone know how such file transfers can be sped up? I am hoping that there is a way for MacOS client to cause the Linux file server to use TCP segmentation offload. I just cannot find out how.
コメント 0