bash 每个命令的“参数列表太长”

"Argument list too long" for every command

提问by Mr.Weathers

Occasionally, when I have a program that generates large arrays I get this bug where every command throws the error "Argument list too long" even if I just type:

有时，当我有一个生成大型数组的程序时，我会遇到这个错误，即使我只输入：

$ cp
-bash: /bin/cp: Argument list too long
$

I can't use ls, or even open a new file with vim:

我无法使用ls，甚至无法使用以下命令打开新文件vim：

$ vim test.txt
-bash: /usr/bin/vim: Argument list too long
$

I tried using "wait" for all bg processes to finish, but no change. It seems to happen inconsistently, but when it does, the only fix is to restart the shell.

我尝试使用“等待”让所有 bg 进程完成，但没有改变。它似乎不一致地发生，但是当它发生时，唯一的解决方法是重新启动 shell。

Any ideas what might be going on?

任何想法可能会发生什么？

Update: I did some further testing and i got the error to be repeatable. It happens when a recursively defined array reaches 85 elements in length. The first command which throws the error is a bcthat doesnt even depend on the array! and then from there on out, almost every other command throws the same error.

更新：我做了一些进一步的测试，我发现错误是可重复的。当递归定义的数组长度达到 85 个元素时，就会发生这种情况。抛出错误的第一个命令是一个bc甚至不依赖于数组的命令！然后从那时起，几乎所有其他命令都会抛出相同的错误。

Update: The program I'm using has many bash scripts working together, but I've determined the problem always arises in this one:

更新：我正在使用的程序有许多 bash 脚本一起工作，但我确定问题总是出现在这个：

function MPMDrun_prop()
{
PARDIR=
COMPDIR=
runSTR=
NUMNODES=
ForceRun=

if [ $# -le 3 ] ; then
echo "USAGE: MPMDrun_prop $PARDIR $COMPDIR $runSTR $NUMNODES $ForceRun"
fi
echo "in MPMDrun_Prop"
. $PARDIR/ParameterScan.inp
. $MCTDHBDIR/Scripts/get_NumberOfJobs.sh

if [ "$MPMD" != "T" ]; then
  MPMDnodes=1
fi

## If no runscripts in the $PARDIR, copy one and strip of the line which runs the program
if [ -z "$(ls $PARDIR/run*.sh 2> /dev/null)"  ] ; then

 if [ "$forhost" == "maia" ]; then
   cp $MCTDHBDIR/../PBS_Scripts/run-example-maia.sh $PARDIR/run.tmp
   sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
   jobtime=86400
 elif [ "$forhost" == "hermit" ]; then
   cp $MCTDHBDIR/../PBS_Scripts/run-example-hermit.sh $PARDIR/run.tmp
   sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
   jobtime=86400
 elif [ "$forhost" == "hornet" ]; then
   cp $MCTDHBDIR/../PBS_Scripts/run-example-hornet.sh $PARDIR/run.tmp
   sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
   jobtime=86400
 elif [ "$forhost" == "bwgrid" ]; then
   cp $MCTDHBDIR/../PBS_Scripts/run-example-BWGRID.sh $PARDIR/run.tmp
   sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
   jobtime=86400
 fi
 sed 's|nodes=[0-9]*|nodes=0|' < $PARDIR/run.sh > $PARDIR/run.tmp
 sed 's|#PBS -N.*|#PBS -N MONSTER_'$MonsterName'|' < $PARDIR/run.tmp > $PARDIR/run.sh_

 rm $PARDIR/run.sh
 rm $PARDIR/run.tmp
 chmod 755 $PARDIR/run.sh_
 echo ". $MCTDHBDIR/Scripts/RunFlagSleeper.sh" >> $PARDIR/run.sh_
 ## Include check_convergence.sh for mixed relax/prop compatibility
 echo ". $MCTDHBDIR/Scripts/check_convergence.sh" >> $PARDIR/run.sh_
 echo "RunFlagSleeper $jobtime " >> $PARDIR/run.sh_
 echo "(" >> $PARDIR/run.sh_
 cp $PARDIR/run.sh_ $PARDIR/run1.sh
fi
### Add $runSTR to the most recent runscript
### find runscript$N.sh (run1.sh, run 2.sh, etc)  that has numnodes less than $MPMDnodes
for qq in $(ls $PARDIR/run[0-9]*.sh | sort -g ); do
  NodesInRun=$(cat $qq | grep -o "nodes *= *[0-9]*" | grep -o "[0-9]*")
  if [ "$NodesInRun" -lt "$MPMDnodes"  ]; then
   ## The number of nodes already specified in the runscript doesnt exceed the maximum, so add on another job
   NewNodes=$(echo "$NodesInRun+$NUMNODES" | bc)
   ## Start each aprun command in its own subshell
   ## wait for 24 hrs after aprun, to guarantee that no subshell finishes before the job is done
   sed 's|nodes=[0-9]*|nodes='$NewNodes'|' < $qq > $qq-1
   sed 's|\(RunFlagSleeper .*\)| '$COMPDIR'|' <$qq-1 >$qq
   rm $qq-1
     echo "  (" >> $qq
     ## Sleeps for $jobtime - 5 mins, then removes runflag. in case aprun doesnt finish in $jobtime 
     echo "    cd $COMPDIR" >> $qq
     echo "    $runSTR" >> $qq
     ## remove runflag after aprun command has finished
     echo "    rm $COMPDIR/RunFlag" >> $qq
#     echo "sleep $jobtime" >> $qq-1
     echo "  ) &" >> $qq
#   mv $qq-1 $qq
   ## put a flag in the computation directory so it isnt computed multiple times
   touch $COMPDIR/RunFlag

   if [[ "$NewNodes" -ge "$MPMDnodes" || "$ForceRun" == "T"  ]]; then
    ## This last process made the nodecount exceed the maximum, or there is a ForceRun flag passed
    ## So now, exceute the runscript and start another
    echo "  wait" >> $qq
    echo ") &" >> $qq
    echo "PID=$!" >> $qq
    echo "wait $PID" >> $qq
    ## Ensure the queue has room for the next job, if not, wait for it
   Njobs=$(get_NumberOfJobs $runhost)
   while [ "$Njobs" -ge "$maxjobs" ]; do
    echo "Njobs=$Njobs and maxjobs=$maxjobs"
    echo "Waiting 30 minutes for que to clear"
    sleep 1800
   done
    echo "qsub $qq"
#    qsub $qq
    RunCount=$(echo $qq | grep -o 'run[0-9]*.sh' | grep -o '[0-9]*')
    let "RunCount++"
    cp $PARDIR/run.sh_ $PARDIR/run$RunCount.sh
   fi
  fi
done
} 

The error typically starts at the 80-90'th call of this function at the first cpor bc. I've commented ALL array manipulations, so there is zero chance this is caused by the array being too large. The environment stays at ~100-200 Kb so that isn't the problem either.

错误通常从第 80-90 次调用此函数的第一个cp或 开始bc。我已经评论了所有数组操作，因此这是由数组太大引起的可能性为零。环境保持在 ~100-200 Kb，所以这也不是问题。

回答by rici

That error message is a bit misleading. It should say something like "Argument list and environment use too much space".

该错误消息有点误导。它应该说类似“参数列表和环境使用太多空间”之类的内容。

The environment consists of all the environment variables you have exported, plus the environment your shell was started with. Normally, the environment should only be a few kilobytes, but there is nothing stopping you from exporting a million-byte string, and if you do that, you'll use up all the space allowed.

环境包含您导出的所有环境变量，以及启动 shell 的环境。通常，环境应该只有几千字节，但是没有什么可以阻止您export输入百万字节的字符串，如果这样做，您将用完允许的所有空间。

It's not totally obvious how much space the system allows for arguments + environment. You should be able to query the limit with getconf ARG_MAX, and with Gnu xargsyou can get more information from xargs --show-limits </dev/null(in both cases, assuming you haven't exceeded the limit :) ), but sometimes the actual space available will turn out to be less than what is indicated.

系统允许参数+环境有多少空间并不完全明显。您应该能够使用 查询限制getconf ARG_MAX，并且使用 Gnuxargs您可以获得更多信息xargs --show-limits </dev/null（在这两种情况下，假设您没有超过限制:)），但有时实际可用空间会小于什么表示。

In any event, it's not a good idea to try to stuff megabytes into the environment. If you're tempted to do that, put the data in a temporary file instead, and just export the name of the file.

无论如何，尝试将兆字节塞入环境并不是一个好主意。如果您想这样做，请将数据放在临时文件中，然后导出文件名。

回答by Dr. Debasish Jana

Since you stated that when you have a program that generates large arrays you get this bug where every command throws the error "Argument list too long". So, I presume that last command you executed is causing problem for next command. My suggestion is that don't use large argument list for any command. This could cause an overflow in the environment causing problems even for next command. Instead of large arg list, use a file having list of data and use the file redirected for input as in:

因为你说过当你有一个生成大数组的程序时，你会得到这个错误，其中每个命令都会抛出错误“参数列表太长”。因此，我认为您执行的最后一个命令会导致下一个命令出现问题。我的建议是不要对任何命令使用大参数列表。这可能会导致环境溢出，即使对于下一个命令也会导致问题。使用具有数据列表的文件而不是大型 arg 列表，并使用重定向的文件进行输入，如下所示：

command < inputfile