[net] FTPClient retrieveFile doesn't finish

[Nick Watts]

Hi Victor. If a file is not being completely received from the FTP server,
I'd first try to figure out why that happened. It sounds like a problem
with the FTP server or the file itself.

As to the specific retrieveFile question you asked, I don't think it's fair
to say that it is "hanging".
The retrieveFile method only knows to return or throw an exception when it
receives an FTP status code of some sort. If it never receives a status
code, it will continue to wait indefinitely. Since there's no way to
foretell how long a file will take to download, you wouldn't really want to
set a timeout anyway as you could prematurely cut off a download that is
slow but otherwise proceeding without error.

Given this, you're probably stuck with having to monitor the state of the
connection and the local file output stream yourself. This is not trivial,
as you can imagine. You would need your own timeout configuration, cleanup
code, perhaps a separate thread on which to do the monitoring, etc....

With the programs that I've used Commons Net on, I just skip all that
complexity and let the files take as long as necessary to download. In
other words, for my use cases, it is acceptable for me to have to kill the
Java process and restart if a file is not fully received after an unusual
amount of time.

I can't recall that I've ever actually had one take an unexpected amount of
time but when that does happen, I'll look to the source of the file first
;)

?Hi!
I'm having an issue where sometimes FTPClient.retrieveFile doesn't finish
and it doesn't timeout either.
ftpClient.enterLocalPassiveMode();
ftpClient.setControlKeepAliveTimeout(60);
ftpClient.setControlKeepAliveReplyTimeout(60000);
ftpClient.setDataTimeout(60000);
ftpClient.setBufferSize(8256000);
ftpClient.setReceiveBufferSize(8256000);
ftpClient.setReceieveDataSocketBufferSize(8256000);
The BufferedOutputStream I'm using is defined as follows...
outputStream = new BufferedOutputStream(new FileOutputStream(localFile),
8256000);
What I'm doing is connecting, logging in, transferring a bunch of files
one at a time, logging out, disconnecting. Is there any way I can prevent
retrieveFile from hanging or make it so that retrieveFile times out after a
while so that I can try again? If there is no configurable timeout, what
strategies are people using to interrupt the call to retrieveFile and try
again?
Thoughts?
Thanks!
Victor
This electronic communication and any attachments may contain confidential
and proprietary information of DigitalGlobe, Inc. If you are not the
intended recipient, or an agent or employee responsible for delivering this
communication to the intended recipient, or if you have received this
communication in error, please do not print, copy, retransmit, disseminate
or otherwise use the information. Please indicate to the sender that you
have received this communication in error, and delete the copy you received.
DigitalGlobe reserves the right to monitor any electronic communication
sent or received by its employees, agents or representatives.

[Victor Rodriguez]

Thanks Nick! Setting a timeout is completely reasonable. If X% of file transfers of files of size X have taken time T, it's reasonable to be able to set a timeout of time 2T, for example. In other words, if this transfer is taking twice as long as X% of similarly sized transfers, there must be something wrong. Let's try this again.

I have a system that transfers hundreds or thousands of files per day. Thankfully, I haven't yet gotten into a situation where all of my threads are stuck at the retrieveFile call at the same time, but if that does happen, the production line would stop.

BTW, it's not that retrieveFile doesn't finish. After I wrote that original email, it turns out that it did eventually finish, but after like 24 hours stuck on a single file. Then, my application continued it's remaining file transfers for that job and they all went swimmingly.

I would say that adding the ability to timeout should be very high on the list of future enhancements for the Apache Commons Net FTP API.

Thanks!

Victor

This electronic communication and any attachments may contain confidential and proprietary information of DigitalGlobe, Inc. If you are not the intended recipient, or an agent or employee responsible for delivering this communication to the intended recipient, or if you have received this communication in error, please do not print, copy, retransmit, disseminate or otherwise use the information. Please indicate to the sender that you have received this communication in error, and delete the copy you received.

DigitalGlobe reserves the right to monitor any electronic communication sent or received by its employees, agents or representatives.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@commons.apache.org
For additional commands, e-mail: user-***@commons.apache.org