Java Apache Commons FTPClient 挂起

Apache Commons FTPClient Hanging

提问by IAmYourFaja

We are using the following Apache Commons Net FTP code to connect to an FTP server, poll some directories for files, and if files are found, to retrieve them to the local machine:

我们使用以下 Apache Commons Net FTP 代码连接到 FTP 服务器，轮询某些目录中的文件，如果找到文件，则将它们检索到本地机器：

try {
logger.trace("Attempting to connect to server...");

// Connect to server
FTPClient ftpClient = new FTPClient();
ftpClient.setConnectTimeout(20000);
ftpClient.connect("my-server-host-name");
ftpClient.login("myUser", "myPswd");
ftpClient.changeWorkingDirectory("/loadables/");

// Check for failed connection
if(!FTPReply.isPositiveCompletion(ftpClient.getReplyCode()))
{
    ftpClient.disconnect();
    throw new FTPConnectionClosedException("Unable to connect to FTP server.");
}

// Log success msg
logger.trace("...connection was successful.");

// Change to the loadables/ directory where we poll for files
ftpClient.changeWorkingDirectory("/loadables/");    

// Indicate we're about to poll
logger.trace("About to check loadables/ for files...");

// Poll for files.
FTPFile[] filesList = oFTP.listFiles();
for(FTPFile tmpFile : filesList)
{
    if(tmpFile.isDirectory())
        continue;

    FileOutputStream fileOut = new FileOutputStream(new File("tmp"));
    ftpClient.retrieveFile(tmpFile.getName(), fileOut);
    // ... Doing a bunch of things with output stream
    // to copy the contents of the file down to the local
    // machine. Ommitted for brevity but I assure you this
    // works (except when the WAR decides to hang).
    //
    // This was used because FTPClient doesn't appear to GET
    // whole copies of the files, only FTPFiles which seem like
    // file metadata...
}

// Indicate file fetch completed.
logger.trace("File fetch completed.");

// Disconnect and finish.
if(ftpClient.isConnected())
    ftpClient.disconnect();

logger.trace("Poll completed.");
} catch(Throwable t) {
    logger.trace("Error: " + t.getMessage());
}

We have this scheduled to run every minute, on the minute. When deployed to Tomcat (7.0.19) this code loads up perfectly fine and begins working without a hitch. Every time though, at some point or another, it seems to just hang. By that I mean:

我们计划每分钟运行一次。当部署到 Tomcat (7.0.19) 时，这段代码加载得非常好，并且开始正常工作。但每次，在某些时候，它似乎只是挂起。我的意思是：

• No heap dumps exist
• Tomcat is still running (I can see its pid and can log into the web manager app)
• Inside the manager app, I can see my WAR is still running/started
• catalina.outand my application-specific log show no signs of any exceptions being thrown

• 不存在堆转储
• Tomcat 仍在运行（我可以看到它的 pid 并且可以登录到 Web 管理器应用程序）
• 在管理器应用程序中，我可以看到我的 WAR 仍在运行/启动
• catalina.out并且我的特定于应用程序的日志没有显示任何异常被抛出的迹象

So the JVM is still running. Tomcat is still running, and my deployed WAR is still running, but its just hanging. Sometimes it runs for 2 hours and then hangs; other times it runs for days and then hangs. But when it does hang, it does so between the line that reads About to check loadables/ for files...(which I do see in the logs) and the line that reads File fetch completed.(which I don't see).

所以JVM仍在运行。Tomcat 仍在运行，我部署的 WAR 仍在运行，但只是挂起。有时运行2小时后就挂了；其他时候它会运行几天然后挂起。但是当它挂起时，它会在读取的行About to check loadables/ for files...（我在日志中看到）和读取的行File fetch completed.（我没有看到）之间挂起。

This tells me the hang occurs during the actual polling/fetching of the files, which kind of points me in the same direction as this questionthat I was able to find which concerns itself with FTPClient deadlocking. This has me wondering if these are the same issues (if they are, I'll happily delete this question!). However I don't think believethey're the same (I don't see the same exceptions in my logs).

这告诉我在实际轮询/获取文件期间发生挂起，这使我指向与我能够找到的与 FTPClient 死锁有关的问题相同的方向。这让我想知道这些是否是相同的问题（如果是，我很乐意删除这个问题！）。不过，我不认为认为它们是相同的（我没有看到我日志中的同样的例外）。

A co-worker mentioned it might be a "Passive" vs. "Active" FTP thing. Not really knowing the difference, I am a little confused by the FTPClient fields ACTIVE_REMOTE_DATA_CONNECTION_MODE, PASSIVE_REMOTE_DATA_CONNECTION_MODE, etc. and didn't know what SO thought about that as being a potential issue.

一位同事提到这可能是“被动”与“主动”FTP 的问题。没有真正知道的区别，我由FTPClient领域糊涂一点ACTIVE_REMOTE_DATA_CONNECTION_MODE，PASSIVE_REMOTE_DATA_CONNECTION_MODE等等，不知道为什么情绪这么想过，作为一个潜在的问题。

Since I'm catching Throwables as a last resort here, I would have expected to see somethingin the logs if something is going wrong. Ergo, I feel like this is a definite hang issue.

由于我在Throwable这里将 s 作为最后的手段，如果出现问题，我本来希望在日志中看到一些东西。因此，我觉得这是一个明确的挂起问题。

Any ideas? Unfortunately I don't know enough about FTP internals here to make a firm diagnosis. Could this be something server-side? Related to the FTP server?

有任何想法吗？不幸的是，我对 FTP 内部结构的了解不够，无法做出明确的诊断。这可能是服务器端的东西吗？与FTP服务器有关吗？

采纳答案by tjg184

This could be a number of things, but your friend's suggestion would be worthwhile.

这可能是很多事情，但你朋友的建议是值得的。

Try ftpClient.enterLocalPassiveMode();to see if it helps.

试试看ftpClient.enterLocalPassiveMode();是否有帮助。

I would also suggest to put the disconnect in the finallyblockso that it never leaves a connection out there.

我还建议将断开连接放在finally块中，以便它永远不会在那里留下连接。

回答by molavec

Yesterday, I didn't sleep but I think I solved the problem.

昨天，我没有睡觉，但我想我解决了这个问题。

You can increase the buffer size with FTPClient.setBufferSize();

您可以使用 FTPClient.setBufferSize() 增加缓冲区大小；

   /**
 * Download encrypted and configuration files.
 * 
 * @throws SocketException
 * @throws IOException
 */
public void downloadDataFiles(String destDir) throws SocketException,
        IOException {

    String filename;
    this.ftpClient.connect(ftpServer);
    this.ftpClient.login(ftpUser, ftpPass);

    /* CHECK NEXT 4 Methods (included the commented) 
    *  they were very useful for me!
    *  and icreases the buffer apparently solve the problem!!
    */
    //  ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
    log.debug("Buffer Size:" + ftpClient.getBufferSize());
    this.ftpClient.setBufferSize(1024 * 1024);
    log.debug("Buffer Size:" + ftpClient.getBufferSize());


    /*  
     *  get Files to download
     */
    this.ftpClient.enterLocalPassiveMode();
    this.ftpClient.setAutodetectUTF8(true);
            //this.ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
    this.ftpClient.enterLocalPassiveMode();
    FTPFile[] ftpFiles = ftpClient
            .listFiles(DefaultValuesGenerator.LINPAC_ENC_DIRPATH);

    /*
     * Download files
     */
    for (FTPFile ftpFile : ftpFiles) {

        // Check if FTPFile is a regular file           
        if (ftpFile.getType() == FTPFile.FILE_TYPE) {
            try{

            filename = ftpFile.getName();

            // Download file from FTP server and save
            fos = new FileOutputStream(destDir + filename);

            //I don't know what useful are these methods in this step
            // I just put it for try
            this.ftpClient.enterLocalPassiveMode();
            this.ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
            this.ftpClient.setAutodetectUTF8(true);
            this.ftpClient.enterLocalPassiveMode();

            ftpClient.retrieveFile(
                    DefaultValuesGenerator.LINPAC_ENC_DIRPATH + filename,
                    fos
                    );

            }finally{
                fos.flush();
                fos.close();                }
        }
    }
    if (fos != null) {
        fos.close();
    }
}

I hope that this code could be usefull for someone!

我希望这段代码对某人有用！

回答by John Blomberg

I had to include the following after login in order to call s.listFiles and transfer without it 'hanging' and eventually failing:

我必须在登录后包含以下内容才能调用 s.listFiles 并在不“挂起”并最终失败的情况下进行传输：

s.login(username, password);
s.execPBSZ(0);
s.execPROT("P");

回答by Tm Zengerle

I had this same issue when trying to perform a listfiles from a Linux machine to a IIS server. The code worked great from my developer workstation, but would hang when running on the server specifically due to a firewall gumming up the mix.

我在尝试从 Linux 机器到 IIS 服务器执行列表文件时遇到了同样的问题。该代码在我的开发人员工作站上运行良好，但在服务器上运行时会挂起，特别是由于防火墙阻碍了混合。

Must do these things in order and will require you to extend FTPSClient 3.5

必须按顺序执行这些操作，并且需要您扩展 FTPSClient 3.5

1. connect (implicit = true, SSLContext = TLS)
2. check isPositiveCompletion
3. authenticate (of course)
4. execPBSZ(0)
5. execPROT("P")
6. set boolean to indicate Skip Passive IP (custom FTPSClient class)
7. set the save connection IP address (custom FTPSClient class)
8. setUseEPSVwithIPv4(false)
9. enterLocalPassiveMode() or enterRemotePassiveMode()
10. initiateListParsing() or any list command a.) At this point the openDataConnectionwill be executed, be sure to save the port being used here b.) The PASV command is executed c.) The _parsePassiveModeReply is executed, here you will open the socket with the IP address you used for connecting and the saved port.
11. disconnect (always)

1. 连接（隐式 = 真，SSLContext = TLS）
2. 检查 isPositiveCompletion
3. 认证（当然）
4. execPBSZ(0)
5. execPROT("P")
6. 设置布尔值以指示跳过被动 IP（自定义 FTPSClient 类）
7. 设置保存连接IP地址（自定义FTPSClient类）
8. setUseEPSVwithIPv4(false)
9. enterLocalPassiveMode() 或 enterRemotePassiveMode()
10. initialListParsing() 或任何列表命令 a.) 此时将执行openDataConnection，请务必保存此处使用的端口 b.) 执行 PASV 命令 c.) 执行 _parsePassiveModeReply，此处将打开套接字使用您用于连接的 IP 地址和保存的端口。
11. 断开连接（总是）

More INFO: My issue is specific to a firewall between the Linux machine and IIS server.
The root of my problem is that in passive mode the IP address used to open the socket when doing a data connection is different that the one used to do the initial connection. So due to two issues (see below) with APACHE commons-net 3.5 it was incredibly difficult to figure out. My solution: Extend FTPSClient so that I could override methods _parsePassiveModeReply & openDataConnection. My parsePassiveModeReply is really just saving the port from the reply since the reply indicates what port is being used. My openDataConnection method is using the saved port and the original IP used during connection.

更多信息：我的问题特定于 Linux 机器和 IIS 服务器之间的防火墙。
我的问题的根源在于，在被动模式下，进行数据连接时用于打开套接字的 IP 地址与用于进行初始连接的 IP 地址不同。因此，由于 APACHE commons-net 3.5 的两个问题（见下文），很难弄清楚。我的解决方案：扩展FTPSClient这样我就可以重写方法_parsePassiveModeReply＆openDataConnection。我的 parsePassiveModeReply 实际上只是从回复中保存端口，因为回复表明正在使用哪个端口。我的 openDataConnection 方法是使用保存的端口和连接期间使用的原始 IP。

Problems with APACHE FTPCLient 3.5

APACHE FTPCLient 3.5 的问题

1. Data Connection does not time out (hangs) so its is not apparent what the problem is.
2. The FTPSClient class does not skip passive IP addresses. Setting passiveNatWorkaround to true doesn't work as I expected or maybe it doesn't skip the IP at all.

1. 数据连接不会超时（挂起），因此问题所在并不明显。
2. FTPSClient 类不会跳过被动 IP 地址。将passiveNatWorkaround 设置为true 并不能像我预期的那样工作，或者它根本不会跳过IP。

Things to pay attention to:

需要注意的事项：

• When going through a firewall you must have access to the port range defined by IIS (see configuring Microsoft IIS firewall).
• You should also ensure you have any appropriate certificates in your keystore or the cert specified at runtime.

• Add the following to your class, this is so very helpful to know what FTP commands are being executed.

     ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
  

• Check the FTP server logs as it will tell you what is being performed and possible why you are having problems. You should always see a data channel opened before executing a list. Compare the results of your application to that of what a successful curl command performs.
• Reply codes as they will indicate where a problem is occurring.

• Use the curl command to verify you have connectivity, The following is a good start and if all is well will list the contents in the root directory.

  curl -3 ftps://[user id]:[password][ftp server ip]:990/ -1 -v --disable-epsv --ftp-skip-pasv-ip --ftp-ssl --insecure
  

• 通过防火墙时，您必须有权访问 IIS 定义的端口范围（请参阅配置 Microsoft IIS 防火墙）。
• 您还应该确保您的密钥库或运行时指定的证书中有任何适当的证书。

• 将以下内容添加到您的类中，这对于了解正在执行哪些 FTP 命令非常有帮助。

     ftpClient.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out), true));
  

• 检查 FTP 服务器日志，因为它会告诉您正在执行的操作以及您遇到问题的可能原因。在执行列表之前，您应该始终看到打开的数据通道。将应用程序的结果与成功的 curl 命令执行的结果进行比较。
• 回复代码，因为它们将指示发生问题的位置。

• 使用 curl 命令验证您是否已连接，以下是一个好的开始，如果一切顺利，将列出根目录中的内容。

  curl -3 ftps://[user id]:[password][ftp server ip]:990/ -1 -v --disable-epsv --ftp-skip-pasv-ip --ftp-ssl --insecure
  

FTPSClient extended (SAMPLE CODE)

FTPSClient 扩展（示例代码）

import java.io.IOException;
import java.net.Inet6Address;
import java.net.InetSocketAddress;
import java.net.Socket;

import javax.net.ssl.SSLContext;

import org.apache.commons.net.MalformedServerReplyException;
import org.apache.commons.net.ftp.FTPReply;
import org.apache.commons.net.ftp.FTPSClient;

/**
 * TODO Document Me!
 */
public class PassiveFTPSClient extends FTPSClient {
    private String passiveSkipToHost;
    private int passiveSkipToPort;
    private boolean skipPassiveIP;


    /** Pattern for PASV mode responses. Groups: (n,n,n,n),(n),(n) */
    private static final java.util.regex.Pattern PARMS_PAT;    
    static {
    PARMS_PAT = java.util.regex.Pattern.compile(
            "(\d{1,3},\d{1,3},\d{1,3},\d{1,3}),(\d{1,3}),(\d{1,3})");
       }
    /**
     * @param b
     * @param sslContext
     */
    public PassiveFTPSClient(boolean b, SSLContext sslContext) {
    super(b, sslContext);
    }

    protected void _parsePassiveModeReply(String reply) throws MalformedServerReplyException {
    if (isSkipPassiveIP()) {
        System.out.println( "================> _parsePassiveModeReply"  + getPassiveSkipToHost());
        java.util.regex.Matcher m = PARMS_PAT.matcher(reply);
        if (!m.find()) {
        throw new MalformedServerReplyException(
            "Could not parse passive host information.\nServer Reply: " + reply);
        }
        try {
        int oct1 = Integer.parseInt(m.group(2));
        int oct2 = Integer.parseInt(m.group(3));
        passiveSkipToPort = (oct1 << 8) | oct2;
        }
        catch (NumberFormatException e) {
        throw new MalformedServerReplyException(
            "Could not parse passive port information.\nServer Reply: " + reply);
        }            
        //do nothing
    } else {
        super._parsePassiveModeReply(reply);
    }
    }

    protected Socket _openDataConnection_(String command, String arg) throws IOException {
    System.out.println( "================> _openDataConnection_"  + getPassiveSkipToHost());
    System.out.println( "================> _openDataConnection_ isSkipPassiveIP: " + isSkipPassiveIP());        
    if (!isSkipPassiveIP()) {
        return super._openDataConnection_(command, arg);
    }
    System.out.println( "================> getDataConnectionMode: " + getDataConnectionMode());
    if (getDataConnectionMode() != ACTIVE_LOCAL_DATA_CONNECTION_MODE &&
        getDataConnectionMode() != PASSIVE_LOCAL_DATA_CONNECTION_MODE) {
        return null;
    }

    final boolean isInet6Address = getRemoteAddress() instanceof Inet6Address;

    Socket socket;
    if (getDataConnectionMode() == ACTIVE_LOCAL_DATA_CONNECTION_MODE) {
        return super._openDataConnection_(command, arg);

    }
    else
    { // We must be in PASSIVE_LOCAL_DATA_CONNECTION_MODE

        // Try EPSV command first on IPv6 - and IPv4 if enabled.
        // When using IPv4 with NAT it has the advantage
        // to work with more rare configurations.
        // E.g. if FTP server has a static PASV address (external network)
        // and the client is coming from another internal network.
        // In that case the data connection after PASV command would fail,
        // while EPSV would make the client succeed by taking just the port.
        boolean attemptEPSV = isUseEPSVwithIPv4() || isInet6Address;
        if (attemptEPSV && epsv() == FTPReply.ENTERING_EPSV_MODE)
        {

        System.out.println( "================> _parseExtendedPassiveModeReply a: ");                
        _parseExtendedPassiveModeReply(_replyLines.get(0));
        }
        else
        {
        if (isInet6Address) {
            return null; // Must use EPSV for IPV6
        }
        // If EPSV failed on IPV4, revert to PASV
        if (pasv() != FTPReply.ENTERING_PASSIVE_MODE) {
            return null;
        }
        System.out.println( "================> _parseExtendedPassiveModeReply b: ");
        _parsePassiveModeReply(_replyLines.get(0));
        }
        // hardcode fore testing
        //__passiveHost = "10.180.255.181";
        socket = _socketFactory_.createSocket();
        if (getReceiveDataSocketBufferSize() > 0) {
        socket.setReceiveBufferSize(getReceiveDataSocketBufferSize());
        }
        if (getSendDataSocketBufferSize()  > 0) {
        socket.setSendBufferSize(getSendDataSocketBufferSize() );
        }
        if (getPassiveLocalIPAddress() != null) {
        System.out.println( "================> socket.bind: " + getPassiveSkipToHost());
        socket.bind(new InetSocketAddress(getPassiveSkipToHost(), 0));
        }

        // For now, let's just use the data timeout value for waiting for
        // the data connection.  It may be desirable to let this be a
        // separately configurable value.  In any case, we really want
        // to allow preventing the accept from blocking indefinitely.
        //     if (__dataTimeout >= 0) {
        //         socket.setSoTimeout(__dataTimeout);
        //     }

        System.out.println( "================> socket connect: " + getPassiveSkipToHost() + ":" + passiveSkipToPort);
        socket.connect(new InetSocketAddress(getPassiveSkipToHost(), passiveSkipToPort), connectTimeout);
        if ((getRestartOffset() > 0) && !restart(getRestartOffset()))
        {
        socket.close();
        return null;
        }

        if (!FTPReply.isPositivePreliminary(sendCommand(command, arg)))
        {
        socket.close();
        return null;
        }
    }

    if (isRemoteVerificationEnabled() && !verifyRemote(socket))
    {
        socket.close();

        throw new IOException(
            "Host attempting data connection " + socket.getInetAddress().getHostAddress() +
            " is not same as server " + getRemoteAddress().getHostAddress());
    }

    return socket;
        }

    /**
    * Enable or disable passive mode NAT workaround.
    * If enabled, a site-local PASV mode reply address will be replaced with the
    * remote host address to which the PASV mode request was sent
    * (unless that is also a site local address).
    * This gets around the problem that some NAT boxes may change the
    * reply.
    *
    * The default is true, i.e. site-local replies are replaced.
    * @param enabled true to enable replacing internal IP's in passive
    * mode.
    */
    public void setSkipPassiveIP(boolean enabled) {
    super.setPassiveNatWorkaround(enabled);
    this.skipPassiveIP = enabled;
    System.out.println( "================> skipPassiveIP: " + skipPassiveIP);
    }
    /**
     * Return the skipPassiveIP.
     * @return the skipPassiveIP
     */
    public boolean isSkipPassiveIP() {
    return skipPassiveIP;
    }
    /**
     * Return the passiveSkipToHost.
     * @return the passiveSkipToHost
     */
    public String getPassiveSkipToHost() {
    return passiveSkipToHost;
    }

    /**
     * Set the passiveSkipToHost.
     * @param passiveSkipToHost the passiveSkipToHost to set
     */
    public void setPassiveSkipToHost(String passiveSkipToHost) {
    this.passiveSkipToHost = passiveSkipToHost;
    System.out.println( "================> setPassiveSkipToHost: " + passiveSkipToHost);
    }

}