Java 如何找到Spark的安装目录？

How to find Spark's installation directory?

提问by Anil

I want to change spark-env.sh. How can I find the installation directory in ubuntu??

我想改变spark-env.sh。如何在ubuntu中找到安装目录？？

I looked in the UI but didn't find anything.

我查看了用户界面，但没有找到任何东西。

whereis spark 

result : spark:

结果 ： spark:

Here's the log for locate command locate spark

这是 locate 命令的日志 locate spark

/home/sys6002/.netbeans/8.0.2/apache-tomcat-8.0.15.0_base/temp/spark-ba1ea995-b959-43f4-ab6c-7d9f1ee5fcc1/blockmgr-db3a931b-7f1a-423e-b5da-b75a958a1909/11
/home/sys6002/.netbeans/8.0.2/apache-tomcat-8.0.15.0_base/temp/spark-ba1ea995-b959-43f4-ab6c-7d9f1ee5fcc1/blockmgr-db3a931b-7f1a-423e-b5da-b75a958a1909/13
/home/sys6002/.netbeans/8.0.2/apache-tomcat-8.0.15.0_base/temp/spark-ba1ea995-b959-43f4-ab6c-7d9f1ee5fcc1/httpd-16b4313e-72dc-4665-b4ac-df491869386d/files
/home/sys6002/.netbeans/8.0.2/apache-tomcat-8.0.15.0_base/temp/spark-ba1ea995-b959-43f4-ab6c-7d9f1ee5fcc1/httpd-16b4313e-72dc-4665-b4ac-df491869386d/jars
/home/sys6002/Desktop/diff spark hadoop.png
/home/sys6002/Desktop/sparkmain
/home/sys6002/Downloads/learning-spark-master.zip
/home/sys6002/Downloads/mongo-spark-master
/home/sys6002/Downloads/spark-1.5.1
/home/sys6002/Downloads/spark-1.5.1-bin-hadoop2.6
/home/sys6002/Downloads/spark-1.5.1-bin-hadoop2.6 (2)
/home/sys6002/Downloads/spark-1.5.1-bin-hadoop2.6.tgz
/home/sys6002/Downloads/spark-1.5.1-bin-without-hadoop
/home/sys6002/Downloads/spark-cassandra-connector-master
/home/sys6002/Downloads/spark-core_2.9.3-0.8.0-incubati
home/sys6002/anaconda3/pkgs/odo-0.3.2-np19py34_0/lib/python3.4/site-packages/odo/backends/tests/__pycache__/test_sparksql.cpython-34.pyc
/home/sys6002/spark-example/a.txt
/home/sys6002/spark-example/a.txt~
/home/sys6002/spark-example/pom.xml
/home/sys6002/spark-example/pom.xml~
/home/sys6002/spark-example/src
/home/sys6002/spark-example/src/main
/home/sys6002/spark-example/src/test
/home/sys6002/spark-example/src/main/java
/home/sys6002/spark-example/src/main/java/com
/home/sys6002/spark-example/src/main/java/com/geekcap
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld/App.java
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld/WordCount.java~
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld/sparkexample
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld/sparkexample/WordCount.java
/home/sys6002/spark-example/src/main/java/com/geekcap/javaworld/sparkexample/WordCount.java~

/home/sys6002/spark-example/src/test/java/com/geekcap/javaworld/AppTest.java
/usr/share/app-install/desktop/lightspark:lightspark.desktop
/usr/share/app-install/desktop/sparkleshare:sparkleshare-invite-opener.desktop
/usr/share/app-install/desktop/sparkleshare:sparkleshare.desktop

回答by Avihoo Mamka

You can try these two commands:

你可以试试这两个命令：

1. locate spark

2. whereis spark

1. locate spark

2. whereis spark

locate- For each given pattern, locate searches one or more databases of file names and displays the file names that contain the pattern. Patterns can contain shell-style metacharacters: '', '?', and '[]'. The metacharacters do not treat '/' or '.' specially. Therefore, a pattern 'foobar' can match a file name that contains 'foo3/bar', and a pattern 'duck' can match a file name that contains 'lake/.ducky'. Patterns that contain metacharacters should be quoted to protect them from expansion by the shell.

whereislocates source/binary and manuals sections for specified files. The supplied names are first stripped of leading pathname components and any (single) trailing extension of the form .ext, for example, .c. Prefixes of s. resulting from use of source code control are also dealt with. whereis then attempts to locate the desired program in a list of standard Linux places.

locate- 对于每个给定的模式，locate 搜索一个或多个文件名数据库并显示包含该模式的文件名。模式可以包含 shell 样式的元字符：' '、'?' 和 '[]'。元字符不处理 '/' 或 '.' 特别。因此，模式“foobar”可以匹配包含“foo3/bar”的文件名，模式“ duck”可以匹配包含“lake/.ducky”的文件名。应引用包含元字符的模式以防止它们被 shell 扩展。

whereis定位指定文件的源/二进制和手册部分。所提供的名称首先去除前导路径名组件和任何 .ext 形式的（单个）尾随扩展名，例如 .c。s 的前缀。使用源代码控制产生的结果也得到处理。然后 whereis 尝试在标准 Linux 位置列表中定位所需的程序。

回答by Jacek Laskowski

Why do you need the home directory of Spark at all? You don't necessarily have to have a local installation of Spark to run Spark applications. You do need to have a local installation of Spark package to have Spark Standalone or to distribute it over YARN or Mesos clusters, but it doesn't seem to be your case.

为什么你需要 Spark 的主目录？您不必在本地安装 Spark 即可运行 Spark 应用程序。您确实需要在本地安装 Spark 包才能拥有 Spark Standalone 或通过 YARN 或 Mesos 集群分发它，但您的情况似乎并非如此。

You need jars and the Spark defaults will do the rest.

您需要 jars，剩下的由 Spark 默认设置完成。

You can see the Web UI since it's up by default when you create a SparkContext instance.

您可以看到 Web UI，因为它在您创建 SparkContext 实例时默认处于启动状态。

All in all, Spark is in the project you're developing as a bunch of jars, and when your Spark application starts, it creates a SparkContext that depending on your master URL will connect to the runtime environment.

总而言之，Spark 在您正在开发的项目中是一堆 jar，当您的 Spark 应用程序启动时，它会创建一个 SparkContext，根据您的主 URL 将连接到运行时环境。

Environment variables in spark-env.shcan be easily set using SparkContext.

spark-env.sh可以使用 轻松设置 中的环境变量SparkContext。

回答by Anil

Thanks to Jacek, I found that Spark is not installed in my machine and I was just using the spark libararies in my project.

感谢 Jacek，我发现我的机器上没有安装 Spark，我只是在我的项目中使用了 spark 库。

To add more,to Run spark application in java project, You don't necessarily need apache spark, scalaand pyspark.

要添加更多内容，要在 java 项目中运行 spark 应用程序，您不一定需要apache spark,scala和pyspark.

Including jars will make your job done

包括罐子会让你的工作完成

回答by Maddenker

This also worked for me:

这也对我有用：

cd $SPARK_HOME

cd $SPARK_HOME

It will take you to the directory where Spark was installed, if that environment variable is set.

如果设置了该环境变量，它将带您到安装 Spark 的目录。

回答by moctarjallo

You should search for spark-shellinstead

你应该寻找spark-shell替代

whereis spark-shell

Typically this would resolve to a path like this /opt/spark-2.3.1-bin-hadoop2.7

通常这会解析为这样的路径 /opt/spark-2.3.1-bin-hadoop2.7

回答by Jin Lim

If you want to delete local data from Spark application in Mac

如果您想从 Mac 中的 Spark 应用程序中删除本地数据

Go to

去

~/Library/Containers/

Then find folder has name got sparksomething

然后找到文件夹名称得到了spark一些东西

回答by NucFlash

If you can't find the directory of where Spark is, you can override the default directory by setting the environment variable SPARK_CONF_DIRto point to a directory of your liking. Then Spark will use that directory to locate spark-defaults.conf, spark-env.sh, etc. See https://spark.apache.org/docs/latest/configuration.html#overriding-configuration-directory

如果找不到 Spark 所在的目录，可以通过将环境变量设置SPARK_CONF_DIR为指向您喜欢的目录来覆盖默认目录。然后星火将使用该目录查找spark-defaults.conf，spark-env.sh等见https://spark.apache.org/docs/latest/configuration.html#overriding-configuration-directory

回答by Carter Shanklin

Run

跑

echo 'sc.getConf.get("spark.home")' | spark-shell

After a moment your Spark home will be printed, you'll see something like this:

片刻之后，您的 Spark 主页将被打印出来，您将看到如下内容：

scala> sc.getConf.get("spark.home")
res0: String = /usr/local/lib/python3.7/site-packages/pyspark

So in this case my Spark Home is /usr/local/lib/python3.7/site-packages/pyspark

所以在这种情况下，我的 Spark Home 是 /usr/local/lib/python3.7/site-packages/pyspark