Python 如何在 Apache Spark 预构建版本中添加任何新库，如 spark-csv
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/30757439/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How to add any new library like spark-csv in Apache Spark prebuilt version
提问by Abhishek Choudhary
I have build the Spark-csvand able to use the same from pyspark shell using the following command
我已经构建了Spark-csv并且能够使用以下命令从 pyspark shell 使用相同的
bin/spark-shell --packages com.databricks:spark-csv_2.10:1.0.3
error getting
错误获取
>>> df_cat.save("k.csv","com.databricks.spark.csv")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/pyspark/sql/dataframe.py", line 209, in save
self._jdf.save(source, jmode, joptions)
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
File "/Users/abhishekchoudhary/bigdata/cdh5.2.0/spark-1.3.1/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError
Where should I place the jar file in my spark pre-built setup so that I will be able to access spark-csvfrom python editor directly as well.
我应该将 jar 文件放在我的 Spark 预构建设置中的什么位置，以便我也可以spark-csv直接从 python 编辑器访问。
采纳答案by Yannick Marcon
At the time I used spark-csv, I also had to download commons-csvjar (not sure it is still relevant). Both jars where in the spark distribution folder.
在我使用 spark-csv 时，我还必须下载commons-csvjar（不确定它是否仍然相关）。两个 jars 都在 spark 分发文件夹中。
I downloaded the jars as follow:
wget http://search.maven.org/remotecontent?filepath=org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar -O commons-csv-1.1.jar<br/> wget http://search.maven.org/remotecontent?filepath=com/databricks/spark-csv_2.10/1.0.0/spark-csv_2.10-1.0.0.jar -O spark-csv_2.10-1.0.0.jarthen started the python spark shell with the arguments:
./bin/pyspark --jars "spark-csv_2.10-1.0.0.jar,commons-csv-1.1.jar"and read a spark dataframe from a csv file:
from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.load(source="com.databricks.spark.csv", path = "/path/to/you/file.csv") df.show()
我按如下方式下载了 jars：
wget http://search.maven.org/remotecontent?filepath=org/apache/commons/commons-csv/1.1/commons-csv-1.1.jar -O commons-csv-1.1.jar<br/> wget http://search.maven.org/remotecontent?filepath=com/databricks/spark-csv_2.10/1.0.0/spark-csv_2.10-1.0.0.jar -O spark-csv_2.10-1.0.0.jar然后使用参数启动python spark shell：
./bin/pyspark --jars "spark-csv_2.10-1.0.0.jar,commons-csv-1.1.jar"并从 csv 文件中读取 spark 数据帧：
from pyspark.sql import SQLContext sqlContext = SQLContext(sc) df = sqlContext.load(source="com.databricks.spark.csv", path = "/path/to/you/file.csv") df.show()
回答by Jimmy
Instead of placing the jars in any specific folder a simple fix would be to start the pyspark shell with the following arguments:
不是将 jars 放在任何特定文件夹中，一个简单的解决方法是使用以下参数启动 pyspark shell：
bin/pyspark --packages com.databricks:spark-csv_2.10:1.0.3
This will automatically load the required spark-csv jars.
这将自动加载所需的 spark-csv jar。
Then do the following to read the csv file:
然后执行以下操作以读取 csv 文件：
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')
df.show()
回答by kentt
Another option is to add the following to your spark-defaults.conf:
另一种选择是将以下内容添加到 spark-defaults.conf 中：
spark.jars.packages com.databricks:spark-csv_2.11:1.2.0
回答by user3815062
Below command helped me -: With Scala 2.10 version
下面的命令帮助了我 -：使用 Scala 2.10 版本
/opt/mapr/spark/spark-1.5.2/bin/spark-shell --master local[*] --packages com.databricks:spark-csv_2.10:1.4.0
Has below dependencies -:
具有以下依赖项 -：
com.databricks#spark-csv_2.10;1.4.0!spark-csv_2.10.jar (2043ms)
org.apache.commons#commons-csv;1.1!commons-csv.jar (419ms)
com.univocity#univocity-parsers;1.5.1!univocity-parsers.jar (1481ms)
回答by Kamil Sindi
Assuming the session/context hasn't been created yet:
假设尚未创建会话/上下文：
import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--packages com.databricks:spark-csv_2.10:1.3.0 pyspark-shell'
回答by Shubham Sinha
first find out the path of the spark. for example for pyspark
首先找出火花的路径。例如对于 pyspark
which pyspark
it will return you the path for example like this- /home/ubuntu/bin/pyspark
它会返回你的路径，例如这样 - /home/ubuntu/bin/pyspark
then run this command by change the path as per your spark path general-: path --packages com.databricks:spark-csv_2.10:1.0.3
然后通过根据您的火花路径一般更改路径来运行此命令： path --packages com.databricks:spark-csv_2.10:1.0.3
/home/ubuntu/bin/pyspark --packages com.databricks:spark-csv_2.10:1.0.3
