[net] FTPClient receive performance issue

[Mikkel Lauritsen]

Hi all,

I am using FTPClient from Commons Net 3.2 to transfer files from a
Windows server (unknown version) through an ADSL connection to a
MacOS client running Java 1.6.0_37.

If I transfer the same files using the command line FTP client I
get a transfer rate of about 2-3 MB/s, but when using FTPClient
everything slows down to a crawl after what appears to be the first
64KB. I have tested against a Linux based FTP server on my local
network, and there performance seems to be fine.

My code basically does

FTPClient client = new FTPClient();
client.connect(hostname);
client.enterLocalPassiveMode();
client.login(username, password));
client.setFileType(FTP.BINARY_FILE_TYPE));
ByteArrayOutputStream baos = new ByteArrayOutputStream(1024 * 1024);
client.retrieveFile("/foo/bar/baz", baos));
client.logout();
client.disconnect();

with listener setup, error handling etc. removed for clarity. The
output from a ProtocolCommandListener is

220 Microsoft FTP Service
USER *******
331 Password required for username.
PASS *******
230 User logged in.
TYPE I
200 Type set to I.
PASV
227 Entering Passive Mode (1,2,3,4,234,188).
RETR /foo/bar/baz
125 Data connection already open; Transfer starting.

Using a CopyStreamListener I can see that I get 64 chunks of 1kB each
in rapid succession, followed by chunks of 320 and 704 bytes, after
which I get delays of several seconds between chunks.

Judging from the fact that the native FTP client works well I'm hoping
that I can somehow configure my way out of the problems, but so far my
trying to poke around with buffer sizes etc. has made no difference.

Does anybody happen to have any suggestions for how to attack this?

Best regards & thanks,
Mikkel Lauritsen

[sebb]

Post by Mikkel Lauritsen
Hi all,
I am using FTPClient from Commons Net 3.2 to transfer files from a
Windows server (unknown version) through an ADSL connection to a
MacOS client running Java 1.6.0_37.
If I transfer the same files using the command line FTP client I
get a transfer rate of about 2-3 MB/s, but when using FTPClient
everything slows down to a crawl after what appears to be the first
64KB. I have tested against a Linux based FTP server on my local
network, and there performance seems to be fine.
My code basically does
FTPClient client = new FTPClient();
client.connect(hostname);
client.enterLocalPassiveMode();
client.login(username, password));
client.setFileType(FTP.BINARY_FILE_TYPE));
ByteArrayOutputStream baos = new ByteArrayOutputStream(1024 * 1024);
client.retrieveFile("/foo/bar/baz", baos));
client.logout();
client.disconnect();
with listener setup, error handling etc. removed for clarity. The
output from a ProtocolCommandListener is
220 Microsoft FTP Service
USER *******
331 Password required for username.
PASS *******
230 User logged in.
TYPE I
200 Type set to I.
PASV
227 Entering Passive Mode (1,2,3,4,234,188).
RETR /foo/bar/baz
125 Data connection already open; Transfer starting.
Using a CopyStreamListener I can see that I get 64 chunks of 1kB each
in rapid succession, followed by chunks of 320 and 704 bytes, after
which I get delays of several seconds between chunks.
Judging from the fact that the native FTP client works well I'm hoping
that I can somehow configure my way out of the problems, but so far my
trying to poke around with buffer sizes etc. has made no difference.
Does anybody happen to have any suggestions for how to attack this?

If you can install a protocol analyser / wire logging application such
as WireShark [1] you could try comparing the traffic to see where the
requests differ.
You may be able to configure the clients so they issue the same
commands, in which case any difference should be down to the client
code (this includes the JVM).

This should also show if the pauses are caused by client or the server/network.
If you run the analyser on the client, of course you cannot
distinguish server pauses from network pauses, so it might be useful
to run the analyser on the server as well (that may not be possible).

I would also try running the same download on a different JVM and OS,
in case the JVM is at fault (not unknown; MacOS Java tends to lag
behind somewhat).
For example, if you have a login to the Linux FTP server you could try
the clients from there.

Change one thing at a time to see where the differences occur.

[1] https://wireshark.org/

Post by Mikkel Lauritsen
Best regards & thanks,
Mikkel Lauritsen
---------------------------------------------------------------------