Python Pandas to_sql，如何创建带有主键的表？

Python Pandas to_sql, how to create a table with a primary key?

提问by patapouf_ai

I would like to create a MySQL table with Pandas' to_sql function which has a primary key (it is usually kind of good to have a primary key in a mysql table) as so:

我想用 Pandas 的 to_sql 函数创建一个 MySQL 表，它有一个主键（在 mysql 表中有一个主键通常很好），如下所示：

group_export.to_sql(con = db, name = config.table_group_export, if_exists = 'replace', flavor = 'mysql', index = False)

but this creates a table without any primary key, (or even without any index).

但这会创建一个没有任何主键（甚至没有任何索引）的表。

The documentation mentions the parameter 'index_label' which combined with the 'index' parameter could be used to create an index but doesn't mention any option for primary keys.

该文档提到参数“index_label”与“index”参数结合可用于创建索引，但没有提及主键的任何选项。

Documentation

文档

采纳答案by krvkir

Disclaimer: this answer is more experimental then practical, but maybe worth mention.

免责声明：这个答案更具实验性而不是实用，但也许值得一提。

I found that class pandas.io.sql.SQLTablehas named argument keyand if you assign it the name of the field then this field becomes the primary key:

我发现该类pandas.io.sql.SQLTable具有命名参数key，如果您为其分配字段名称，则该字段将成为主键：

Unfortunately you can't just transfer this argument from DataFrame.to_sql()function. To use it you should:

不幸的是，你不能只从DataFrame.to_sql()函数中传递这个参数。要使用它，您应该：

1. create pandas.io.SQLDatabaseinstance

   engine = sa.create_engine('postgresql:///somedb')
   pandas_sql = pd.io.sql.pandasSQL_builder(engine, schema=None, flavor=None)
   

2. define function analoguous to pandas.io.SQLDatabase.to_sql()but with additional *kwargsargument which is passed to pandas.io.SQLTableobject created inside it (i've just copied original to_sql()method and added *kwargs):

   def to_sql_k(self, frame, name, if_exists='fail', index=True,
              index_label=None, schema=None, chunksize=None, dtype=None, **kwargs):
       if dtype is not None:
           from sqlalchemy.types import to_instance, TypeEngine
           for col, my_type in dtype.items():
               if not isinstance(to_instance(my_type), TypeEngine):
                   raise ValueError('The type of %s is not a SQLAlchemy '
                                    'type ' % col)
   
       table = pd.io.sql.SQLTable(name, self, frame=frame, index=index,
                        if_exists=if_exists, index_label=index_label,
                        schema=schema, dtype=dtype, **kwargs)
       table.create()
       table.insert(chunksize)
   

3. call this function with your SQLDatabaseinstance and the dataframe you want to save

   to_sql_k(pandas_sql, df2save, 'tmp',
           index=True, index_label='id', keys='id', if_exists='replace')
   

1. 创建pandas.io.SQLDatabase实例

   engine = sa.create_engine('postgresql:///somedb')
   pandas_sql = pd.io.sql.pandasSQL_builder(engine, schema=None, flavor=None)
   

2. 定义类似于pandas.io.SQLDatabase.to_sql()但带有附加*kwargs参数的函数，该参数传递给pandas.io.SQLTable在其中创建的对象（我刚刚复制了原始to_sql()方法并添加了*kwargs）：

   def to_sql_k(self, frame, name, if_exists='fail', index=True,
              index_label=None, schema=None, chunksize=None, dtype=None, **kwargs):
       if dtype is not None:
           from sqlalchemy.types import to_instance, TypeEngine
           for col, my_type in dtype.items():
               if not isinstance(to_instance(my_type), TypeEngine):
                   raise ValueError('The type of %s is not a SQLAlchemy '
                                    'type ' % col)
   
       table = pd.io.sql.SQLTable(name, self, frame=frame, index=index,
                        if_exists=if_exists, index_label=index_label,
                        schema=schema, dtype=dtype, **kwargs)
       table.create()
       table.insert(chunksize)
   

3. 使用您的SQLDatabase实例和要保存的数据框调用此函数

   to_sql_k(pandas_sql, df2save, 'tmp',
           index=True, index_label='id', keys='id', if_exists='replace')
   

And we get something like

我们得到类似的东西

CREATE TABLE public.tmp
(
  id bigint NOT NULL DEFAULT nextval('tmp_id_seq'::regclass),
...
)

in the database.

在数据库中。

PS You can of course monkey-patch DataFrame, io.SQLDatabaseand io.to_sql()functions to use this workaround with convenience.

当然猴子补丁的PS可以DataFrame，io.SQLDatabase并且io.to_sql()功能使用此解决办法提供便利。

回答by howMuchCheeseIsTooMuchCheese

automap_basefrom sqlalchemy.ext.automap(tableNamesDict is a dict with only the Pandas tables):

automap_basefrom sqlalchemy.ext.automap（tableNamesDict 是一个只有 Pandas 表的字典）：

metadata = MetaData()
metadata.reflect(db.engine, only=tableNamesDict.values())
Base = automap_base(metadata=metadata)
Base.prepare()

Which would have worked perfectly, except for one problem, automap requires the tables to have a primary key. Ok, no problem, I'm sure Pandas to_sqlhas a way to indicate the primary key... nope. This is where it gets a little hacky:

这本来可以完美运行，除了一个问题，automap 要求表具有主键。好的，没问题，我确定 Pandasto_sql有办法指示主键……不。这是它变得有点hacky的地方：

for df in dfs.keys():
    cols = dfs[df].columns
    cols = [str(col) for col in cols if 'id' in col.lower()]
    schema = pd.io.sql.get_schema(dfs[df],df, con=db.engine, keys=cols)
    db.engine.execute('DROP TABLE ' + df + ';')
    db.engine.execute(schema)
    dfs[df].to_sql(df,con=db.engine, index=False, if_exists='append')

I iterate thru the dictof DataFrames, get a list of the columns to use for the primary key (i.e. those containing id), use get_schemato create the empty tables then append the DataFrameto the table.

我遍历dictof DataFrames，获取用于主键的列列表（即那些包含id），用于get_schema创建空表，然后将 附加DataFrame到表中。

Now that you have the models, you can explicitly name and use them (i.e. User = Base.classes.user) with session.queryor create a dict of all the classes with something like this:

现在你有了模型，你可以显式地命名和使用它们（即User = Base.classes.user）session.query或创建一个所有类的字典，如下所示：

alchemyClassDict = {}
for t in Base.classes.keys():
    alchemyClassDict[t] = Base.classes[t]

And query with:

并查询：

res = db.session.query(alchemyClassDict['user']).first()

回答by tomp

Simply add the primary key after uploading the table with pandas.

上传带有pandas的表后，只需添加主键即可。

group_export.to_sql(con=engine, name=example_table, if_exists='replace', 
                    flavor='mysql', index=False)

with engine.connect() as con:
    con.execute('ALTER TABLE `example_table` ADD PRIMARY KEY (`ID_column`);')