git 如何在 GitLab 中一次克隆一个组的所有项目？

How to clone all projects of a group at once in GitLab?

提问by Doon

In my GitLab repository, I have a group with 20 projects. I want to clone all projects at once. Is that possible?

在我的 GitLab 存储库中，我有一个包含 20 个项目的组。我想一次克隆所有项目。那可能吗？

采纳答案by VonC

Not really, unless:

不是真的，除非：

• you have a 21st project which references the other 20 as submodules.
  (in which case a clone followed by a git submodule update --initwould be enough to get all 20 projects cloned and checked out)

• or you somehow list the projects you have access (GitLab API for projects), and loop on that result to clone each one (meaning that can be scripted, and then executed as "one" command)

• 您有一个第 21 个项目，它引用其他 20 个作为子模块。
  （在这种情况下，一个克隆后跟一个git submodule update --init就足以克隆和检出所有 20 个项目）

• 或者您以某种方式列出您有权访问的项目（项目的 GitLab API），并循环该结果以克隆每个项目（意味着可以编写脚本，然后作为“一个”命令执行）

---

---

Since 2015, Jay Gabezmentions in the comments(August 2019) the tool gabrie30/ghorg

自 2015 年以来，Jay Gabez在评论（2019 年 8 月）中提到了该工具gabrie30/ghorg

ghorgallows you to quickly clone all of an orgs, or users repos into a single directory.

ghorg允许您快速将所有组织或用户存储库克隆到单个目录中。

Usage:

用法：

$ ghorg clone someorg
$ ghorg clone someuser --clone-type=user --protocol=ssh --branch=develop
$ ghorg clone gitlab-org --scm=gitlab --namespace=gitlab-org/security-products
$ ghorg clone --help

---

---

Also (2020): https://github.com/ezbz/gitlabber

另外（2020）：https: //github.com/ezbz/gitlabber

usage: gitlabber [-h] [-t token] [-u url] [--debug] [-p]
                [--print-format {json,yaml,tree}] [-i csv] [-x csv]
                [--version]
                [dest]

Gitlabber - clones or pulls entire groups/projects tree from gitlab

回答by Hot Diggity

Here's an example in Python 3:

这是 Python 3 中的一个示例：

from urllib.request import urlopen
import json
import subprocess, shlex

allProjects     = urlopen("http://[yourServer:port]/api/v4/projects?private_token=[yourPrivateTokenFromUserProfile]&per_page=100000")
allProjectsDict = json.loads(allProjects.read().decode())
for thisProject in allProjectsDict: 
    try:
        thisProjectURL  = thisProject['ssh_url_to_repo']
        command     = shlex.split('git clone %s' % thisProjectURL)
        resultCode  = subprocess.Popen(command)

    except Exception as e:
        print("Error on %s: %s" % (thisProjectURL, e.strerror))

回答by Dinesh Balasubramanian

One liner with curl, jq, tr

一个带卷曲、jq、tr 的衬垫

for repo in $(curl "https://<your-host>/api/v4/groups/<group_id>?private_token=<your_private_token>" | jq .projects[].ssh_url_to_repo | tr -d '"'); do git clone $repo; done;

回答by Benedikt

There is a tool called myrepos, which manages multiple version controls repositories. Updating all repositories simply requires one command:

有一个名为myrepos的工具，它管理多个版本控制存储库。更新所有存储库只需要一个命令：

mr update

In order to register all gitlab projects to mr, here is a small python script. It requires the package python-gitlabinstalled:

为了将所有gitlab项目注册到mr，这里有一个小的python脚本。它需要安装python-gitlab包：

import os
from subprocess import call
from gitlab import Gitlab

# Register a connection to a gitlab instance, using its URL and a user private token
gl = Gitlab('http://192.168.123.107', 'JVNSESs8EwWRx5yDxM5q')
groupsToSkip = ['aGroupYouDontWantToBeAdded']

gl.auth() # Connect to get the current user

gitBasePathRelative = "git/"
gitBasePathRelativeAbsolut = os.path.expanduser("~/" + gitBasePathRelative)
os.makedirs(gitBasePathRelativeAbsolut,exist_ok=True)

for p in gl.Project():
    if not any(p.namespace.path in s for s in groupsToSkip):
        pathToFolder = gitBasePathRelative + p.namespace.name + "/" + p.name
        commandArray = ["mr", "config", pathToFolder, "checkout=git clone '" + p.ssh_url_to_repo + "' '" + p.name + "'"]
        call(commandArray)

os.chdir(gitBasePathRelativeAbsolut)

call(["mr", "update"])

回答by Jonas Gr?ger

I built a script (curl, git, jq required) just for that. We use it and it works just fine: https://gist.github.com/JonasGroeger/1b5155e461036b557d0fb4b3307e1e75

我为此构建了一个脚本（需要 curl、git、jq）。我们使用它，它工作得很好：https: //gist.github.com/JonasGroeger/1b5155e461036b557d0fb4b3307e1e75

To find out your namespace, its best to check the API quick:

要找出您的命名空间，最好快速检查 API：

curl "https://domain.com/api/v3/projects?private_token=$GITLAB_PRIVATE_TOKEN"

There, use "namespace.name" as NAMESPACEfor your group.

在那里，使用“namespace.name”作为NAMESPACE您的组。

The script essentially does:

该脚本主要执行以下操作：

1. Get all Projects that match your PROJECT_SEARCH_PARAM

2. Get their pathand ssh_url_to_repo

   2.1. If the directory pathexists, cd into it and call git pull

   2.2. If the directory pathdoes not exist, call git clone

1. 获取与您匹配的所有项目 PROJECT_SEARCH_PARAM

2. 得到他们的path和ssh_url_to_repo

   2.1. 如果目录path存在， cd 进入它并调用git pull

   2.2. 如果目录path不存在，调用git clone

回答by pbuchheit

Here is another example of a bash script to clone all the repos in a group. The only dependency you need to install is jq (https://stedolan.github.io/jq/). Simply place the script into the directory you want to clone your projects into. Then run it as follows:

这是克隆组中所有存储库的 bash 脚本的另一个示例。您需要安装的唯一依赖项是 jq ( https://stedolan.github.io/jq/)。只需将脚本放入您要将项目克隆到的目录中。然后按如下方式运行它：

./myscript <group name> <private token> <gitlab url>

i.e.

IE

./myscript group1 abc123tyn234 http://yourserver.git.com

./myscript group1 abc123tyn234 http://yourserver.git.com

Script:

脚本：

#!/bin/bash
if command -v jq >/dev/null 2>&1; then
  echo "jq parser found";
else
  echo "this script requires the 'jq' json parser (https://stedolan.github.io/jq/).";
  exit 1;
fi

if [ -z "" ]
  then
    echo "a group name arg is required"
    exit 1;
fi

if [ -z "" ]
  then
    echo "an auth token arg is required. See /profile/account"
    exit 1;
fi

if [ -z "" ]
  then
    echo "a gitlab URL is required."
    exit 1;
fi

TOKEN="";
URL="/api/v3"
PREFIX="ssh_url_to_repo";

echo "Cloning all git projects in group ";

GROUP_ID=$(curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups?search= | jq '.[].id')
echo "group id was $GROUP_ID";
curl --header "PRIVATE-TOKEN: $TOKEN" $URL/groups/$GROUP_ID/projects?per_page=100 | jq --arg p "$PREFIX" '.[] | .[$p]' | xargs -L1 git clone

回答by Moshohayeb

If you are okay with some shell sorecery this will clone all the repos grouped by their group-id (you need jq and parallel)

如果您对某些 shell sorecery 没问题，这将克隆所有按组 ID 分组的存储库（您需要 jq 和并行）

seq 3                                                                           \
| parallel curl -s "'https://[gitlabUrl]/api/v4/projects?page={}&per_page=100&private_token=[privateToken]'
                     | jq '.[] | .ssh_url_to_repo, .name, .namespace.path'" \
| tr -d '"'                                                                 \
| awk '{ printf "%s ", 
import requests
import subprocess, shlex
import os

print('Starting getrepos process..')

key = '12345678901234567890' # your gitlab key
base_url = 'https://your.gitlab.url/api/v4/projects?simple=true&per_page=10&private_token='
url = base_url + key

base_dir = os.getcwd()

while True:
    print('\n\nRetrieving from ' + url)
    response = requests.get(url, verify = False)
    projects = response.json()

    for project in projects:
        project_name = project['name']
        project_path = project['namespace']['full_path']
        project_url = project['ssh_url_to_repo']

        os.chdir(base_dir)
        print('\nProcessing %s...' % project_name)

        try:
            print('Moving into directory: %s' % project_path)
            os.makedirs(project_path, exist_ok = True)
            os.chdir(project_path)
            cmd = shlex.split('git clone --mirror %s' % project_url)
            subprocess.run(cmd)
        except Exception as e:
            print('Error: ' + e.strerror)

    if 'next' not in response.links:
        break

    url = response.links['next']['url'].replace('127.0.0.1:9999', 'your.gitlab.url')


print('\nDone')
; if (NR % 3 == 0) print " " }'                    \
| parallel --colsep ' ' 'mkdir -p {2} && git clone {1} {3}/{2}'

回答by forresthopkinsa

An updated Python 3 script that accomplishes this really effectively using Gitlab's latest api and proper pagination:

一个更新的 Python 3 脚本，它使用 Gitlab 的最新 api 和适当的分页非常有效地完成了这一点：

#!/usr/bin/python3
import os
import sys
import gitlab
import subprocess

glab = gitlab.Gitlab(f'https://{sys.argv[1]}', f'{sys.argv[3]}')
groups = glab.groups.list()
groupname = sys.argv[2]
for group in groups:
    if group.name == groupname:
        projects = group.projects.list(all=True)

for repo in projects:
    command = f'git clone {repo.ssh_url_to_repo}'
    process = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
    output, _ = process.communicate()
    process.wait()

Requires the requestslibrary (for navigating to the page links).

需要请求库（用于导航到页面链接）。

回答by Miroshnichenko Dmitriy

Yep it's possible, here is the code.

是的，这是可能的，这是代码。

prerequisites:

先决条件：

pip install python-gitlab

pip 安装 python-gitlab

for pag in {1..3} // number of pages projects has span {per page 20 projects so if you have 50 projects loop should be 1..3}
do
curl -s http://gitlink/api/v4/groups/{groupName}/projects?page=$pag > url.txt
grep -o '"ssh_url_to_repo": *"[^"]*"' url.txt | grep -o '"[^"]*"$' | while read -r line ; do
l1=${line%?}
l2=${l1:1}
echo "$l2"
git clone $l2
done
done

Example:

例子：

• create .py file (ex. gitlab-downloader.py)
• copy-paste code from above
• on Linux OS (or OSX) do chmod +x on the script file (ex. chmod +x gitlab-downloader.py)
• run it with 3 params: Gitlab hostname, groupname, your Personal Access Token(see https://gitlab.exmaple.com/profile/personal_access_tokens)

• 创建 .py 文件（例如 gitlab-downloader.py）
• 从上面复制粘贴代码
• 在 Linux 操作系统（或 OSX）上对脚本文件执行 chmod +x（例如 chmod +x gitlab-downloader.py）
• 使用 3 个参数运行它：Gitlab 主机名、组名、您的个人访问令牌（请参阅https://gitlab.exmaple.com/profile/personal_access_tokens）

回答by Vivek

I have written the script to pull the complete code base from gitlab for particular group.

我已经编写了脚本来从 gitlab 中为特定组提取完整的代码库。

##代码##