bash 使用 cURL 下载目录中的所有文件

downloading all the files in a directory with cURL

提问by l--''''''---------''''''''''''

I am using cURL to try to download all files in a certain directory.

我正在使用 cURL 尝试下载某个目录中的所有文件。

here's what my list of files looks like:

这是我的文件列表的样子：

I have tried to do in bash script: iiumlabs.[].csv.pgpand iiumlabs*and I guess curl is not big on wildcards.

我曾尝试在bash脚本做的事：iiumlabs.[].csv.pgp和iiumlabs*，我想袅袅不上通配符大。

curl -u login:pass ftp.myftpsite.com/iiumlabs* -O

question:how do i download this directory of files using cURL?

问题：如何使用 cURL 下载此文件目录？

采纳答案by Micha? Górny

OK, considering that you are using Windows, the most simple way to do that is to use the standard ftptool bundled with it. I base the following solution on Windows XP, hoping it'll work as well (or with minor modifications) on other versions.

好的，考虑到您使用的是 Windows，最简单的方法是使用ftp与它捆绑在一起的标准工具。我将以下解决方案基于 Windows XP，希望它在其他版本上也能正常工作（或稍作修改）。

First of all, you need to create a batch (script) file for the ftpprogram, containing instructions for it. Name it as you want, and put into it:

首先，您需要为程序创建一个批处理（脚本）文件ftp，其中包含它的指令。随意命名，然后放入：

curl -u login:pass ftp.myftpsite.com/iiumlabs* -O

open ftp.myftpsite.com
login
pass
mget *
quit

The first line opens a connection to the ftp server at ftp.myftpsite.com. The two following lines specify the login, and the password which ftp will ask for (replace loginand passwith just the login and password, without any keywords). Then, you use mget *to get all files. Instead of the *, you can use any wildcard. Finally, you use quitto close the ftpprogram without interactive prompt.

第一行在 处打开到 ftp 服务器的连接ftp.myftpsite.com。以下两行指定了登录名和 ftp 要求的密码（仅用登录名和密码替换login和pass，不带任何关键字）。然后，您使用mget *获取所有文件。*您可以使用任何通配符代替。最后，您使用quit在ftp没有交互提示的情况下关闭程序。

If you needed to enter some directory first, add a cdcommand before mget. It should be pretty straightforward.

如果您需要先进入某个目录，请cd在mget. 它应该非常简单。

Finally, write that file and run ftplike this:

最后，编写该文件并ftp像这样运行：

ftp -i -s:yourscript

where -idisables interactivity (asking before downloading files), and -sspecifies path to the script you created.

where-i禁用交互性（在下载文件之前询问），并-s指定您创建的脚本的路径。

---

---

Sadly, file transfer over SSH is not natively supported in Windows. But for that case, you'd probably want to use PuTTytools anyway. The one of particular interest for this case would be pscpwhich is practically the PuTTy counter-part of the openssh scpcommand.

遗憾的是，Windows 本身并不支持通过 SSH 传输文件。但对于这种情况，您可能无论如何都想使用PuTTy工具。对于这种情况，特别感兴趣的是pscp它实际上是 opensshscp命令的 PuTTy 对应部分。

The syntax is similar to copycommand, and it supports wildcards:

语法类似于copycommand，它支持通配符：

pscp -batch [email protected]:iiumlabs* .

If you authenticate using a key file, you should pass it using -i path-to-key-file. If you use password, -pw pass. It can also reuse sessions saved using PuTTy, using the load -load your-session-nameargument.

如果您使用密钥文件进行身份验证，则应使用-i path-to-key-file. 如果使用密码，-pw pass. 它还可以使用 load-load your-session-name参数重用使用 PuTTy 保存的会话。

回答by kkeller

If you're not bound to curl, you might want to use wgetin recursive mode but restricting it to one level of recursion, try the following;

如果您不绑定 curl，您可能希望在递归模式下使用wget，但将其限制为一级递归，请尝试以下操作；

wget --no-verbose --no-parent --recursive --level=1\
--no-directories --user=login --password=pass ftp://ftp.myftpsite.com/

• --no-parent: Do not ever ascend to the parent directory when retrieving recursively.
• --level=depth: Specify recursion maximum depth level depth. The default maximum depth is five layers.
• --no-directories: Do not create a hierarchy of directories when retrieving recursively.

• --no-parent: 递归检索时不要上升到父目录。
• --level=depth：指定递归最大深度级别深度。默认最大深度为五层。
• --no-directories: 递归检索时不要创建目录层次结构。

回答by Robi

What about something like this:

这样的事情怎么样：

for /f %%f in ('curl -s -l -u user:pass ftp://ftp.myftpsite.com/') do curl -O -u user:pass ftp://ftp.myftpsite.com/%%f

回答by M.Selman SEZG?N

You can use script like this for mac:

您可以在 mac 上使用这样的脚本：

for f in $(curl -s -l -u user:pass ftp://your_ftp_server_ip/folder/) 
 do curl -O -u user:pass ftp://your_ftp_server_ip/folder/$f 
done

回答by kiwixz

Here is how I did to download quickly with cURL (I'm not sure how many files it can download though) :

这是我使用 cURL 快速下载的方法（我不确定它可以下载多少个文件）：

setlocal EnableDelayedExpansion

cd where\to\download

set STR=
for /f "skip=2 delims=" %%F in ('P:\curl -l -u user:password ftp://ftp.example.com/directory/anotherone/') do set STR=-O "ftp://ftp.example.com/directory/anotherone/%%F" !STR!
path\to\curl.exe -v -u user:password !STR!

Why skip=2? To get ride of .and ..

为什么跳过=2？乘坐.和..

Why delims=? To support names with spaces

为什么delims=？支持带空格的名称

回答by Prasanth

Oh, I have just the thing you need!

哦，我有你需要的东西！

$host = "ftp://example.com/dir/";
$savePath = "downloadedFiles";
if($check = isFtpUp($host)){

    echo $ip." -is alive<br />";

    $check = trim($check);
    $files = explode("\n",$check);

    foreach($files as $n=>$file){
        $file = trim($file);
        if($file !== "." || $file !== ".."){
            if(!saveFtpFile($file, $host.$file, $savePath)){
                // downloading failed. possible reason: $file is a folder name.
                // echo "Error downloading file.<br />";
            }else{
                echo "File: ".$file." - saved!<br />";
            }
        }else{
            // do nothing
        }
    }
}else{
    echo $ip." - is down.<br />";
}

and functions isFtpUpand saveFtpFileare as follows:

和功能isFtpUp与saveFtpFile如下：

function isFtpUp($host){
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $host);
curl_setopt($ch, CURLOPT_USERPWD, "anonymous:[email protected]");
curl_setopt($ch, CURLOPT_FTPLISTONLY, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 3);

$result = curl_exec($ch);

return $result;

}

function saveFtpFile( $targetFile = null, $sourceFile = null, $savePath){

// function settings
set_time_limit(60);
$timeout = 60;
$ftpuser = "anonymous";
$ftppassword = "[email protected]";
$savePath = "downloadedFiles"; // should exist!
$curl = curl_init();
$file = @fopen ($savePath.'/'.$targetFile, 'w');

if(!$file){
    return false;
}

curl_setopt($curl, CURLOPT_URL, $sourceFile);
curl_setopt($curl, CURLOPT_USERPWD, $ftpuser.':'.$ftppassword);

// curl settings

// curl_setopt($curl, CURLOPT_FAILONERROR, 1);
// curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, $timeout);
curl_setopt($curl, CURLOPT_FILE, $file);

$result = curl_exec($curl);


if(!$result){
    return false;
}

curl_close($curl);
fclose($file);

return $result;
}

EDIT:

编辑：

it's a php script. save it as a .php file, put it on your webserver, change $ip to address(need not be ip) of ftp server you want to download files from, create a directory named downloadedFiles on the same directory as this file.

这是一个php脚本。将其另存为 .php 文件，将其放在您的网络服务器上，将 $ip 更改为要从中下载文件的 ftp 服务器的地址（不必是 ip），在与此文件相同的目录下创建一个名为 downloadFiles 的目录。