使用 pandas 中的 read_csv 时为特定列设置数据类型
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/50642777/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Set data type for specific column when using read_csv from pandas
提问by Xitrum
I have a large csv file (~10GB), with around 4000 columns. I know that most of data i will expect is int8, so i set:
我有一个大的 csv 文件（~10GB），大约有 4000 列。我知道我期望的大部分数据是 int8，所以我设置：
pandas.read_csv('file.dat', sep=',', engine='c', header=None,
na_filter=False, dtype=np.int8, low_memory=False)
Thing is, the final column (4000th position) is int32, is there away can i tell read_csv that use int8 by default, and at column 4000th, use int 32?
问题是，最后一列（第 4000 个位置）是 int32，我可以告诉 read_csv 默认使用 int8，在第 4000 列使用 int 32 吗？
Thank you
谢谢
采纳答案by Anton vBR
If you are certain of the number you could recreate the dictionary like this:
如果您确定数字，您可以像这样重新创建字典：
dtype = dict(zip(range(4000),['int8' for _ in range(3999)] + ['int32']))
Considering that this works:
考虑到这有效：
import pandas as pd
import numpy as np
?
data = '''\
1,2,3
4,5,6'''
?
fileobj = pd.compat.StringIO(data)
df = pd.read_csv(fileobj, dtype={0:'int8',1:'int8',2:'int32'}, header=None)
?
print(df.dtypes)
Returns:
返回：
0 int8
1 int8
2 int32
dtype: object
From the docs:
从文档：
dtype : Type name or dict of column -> type, default None
Data type for data or columns. E.g. {‘a': np.float64, ‘b': np.int32} Use str or object to preserve and not interpret dtype. If converters are specified, they will be applied INSTEAD of dtype conversion.
dtype : 类型名称或列的字典 -> 类型，默认无
数据或列的数据类型。例如 {'a': np.float64, 'b': np.int32} 使用 str 或 object 来保留而不是解释 dtype。如果指定了转换器，它们将被应用于 dtype 转换的 INSTEAD。
回答by James
Since you have no header, the column names are the integer order in which they occur, i.e. the first column is df[0]. To programmatically set the last column to be int32, you can read the first line of the file to get the width of the dataframe, then construct a dictionary of the integer types you want to use with the number of the columns as the keys.
由于您没有标题，列名是它们出现的整数顺序，即第一列是df[0]。要以编程方式将最后一列设置为int32，您可以读取文件的第一行以获取数据框的宽度，然后以列数作为键构造您要使用的整数类型的字典。
import numpy as np
import pandas as pd
with open('file.dat') as fp:
width = len(fp.readline().strip().split(','))
dtypes = {i: np.int8 for i in range(width)}
# update the last column's dtype
dtypes[width-1] = np.int32
# reset the read position of the file pointer
fp.seek(0)
df = pd.read_csv(fp, sep=',', engine='c', header=None,
na_filter=False, dtype=dtypes, low_memory=False)
