Python 使用 Django 1.7 加载初始数据和数据迁移

Loading initial data with Django 1.7 and data migrations

提问by Micka?l

I recently switched from Django 1.6 to 1.7, and I began using migrations (I never used South).

我最近从 Django 1.6 切换到 1.7，并开始使用迁移（我从未使用过 South）。

Before 1.7, I used to load initial data with a fixture/initial_data.jsonfile, which was loaded with the python manage.py syncdbcommand (when creating the database).

在1.7之前，我曾经用一个fixture/initial_data.json文件加载初始数据，该文件是用python manage.py syncdb命令加载的（创建数据库时）。

Now, I started using migrations, and this behavior is deprecated :

现在，我开始使用迁移，这种行为已被弃用：

If an application uses migrations, there is no automatic loading of fixtures. Since migrations will be required for applications in Django 2.0, this behavior is considered deprecated. If you want to load initial data for an app, consider doing it in a data migration. (https://docs.djangoproject.com/en/1.7/howto/initial-data/#automatically-loading-initial-data-fixtures)

如果应用程序使用迁移，则不会自动加载夹具。由于 Django 2.0 中的应用程序需要迁移，因此此行为被视为已弃用。如果要为应用加载初始数据，请考虑在数据迁移中进行。（https://docs.djangoproject.com/en/1.7/howto/initial-data/#automatically-loading-initial-data-fixtures）

The official documentationdoes not have a clear example on how to do it, so my question is :

在官方文件并没有对如何做一个明显的例子，所以我的问题是：

What is the best way to import such initial data using data migrations :

使用数据迁移导入此类初始数据的最佳方法是什么：

1. Write Python code with multiple calls to mymodel.create(...),
2. Use or write a Django function (like calling loaddata) to load data from a JSON fixture file.

1. 编写多次调用的 Python 代码mymodel.create(...)，
2. 使用或编写 Django 函数（如调用loaddata）从 JSON 固定文件加载数据。

I prefer the second option.

我更喜欢第二种选择。

I don't want to use South, as Django seems to be able to do it natively now.

我不想使用 South，因为 Django 现在似乎可以在本地使用它。

采纳答案by n__o

Update: See @GwynBleidD's comment below for the problems this solution can cause, and see @Rockallite's answer below for an approach that's more durable to future model changes.

更新：请参阅下面@GwynBleidD 的评论，了解此解决方案可能导致的问题，并参阅下面@Rockallite 的回答，了解对未来模型更改更持久的方法。

---

---

Assuming you have a fixture file in <yourapp>/fixtures/initial_data.json

假设你有一个夹具文件 <yourapp>/fixtures/initial_data.json

1. Create your empty migration:

   In Django 1.7:

   python manage.py makemigrations --empty <yourapp>
   

   In Django 1.8+, you can provide a name:

   python manage.py makemigrations --empty <yourapp> --name load_intial_data
   

2. Edit your migration file <yourapp>/migrations/0002_auto_xxx.py

   2.1. Custom implementation, inspired by Django' loaddata(initial answer):

   import os
   from sys import path
   from django.core import serializers
   
   fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
   fixture_filename = 'initial_data.json'
   
   def load_fixture(apps, schema_editor):
       fixture_file = os.path.join(fixture_dir, fixture_filename)
   
       fixture = open(fixture_file, 'rb')
       objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
       for obj in objects:
           obj.save()
       fixture.close()
   
   def unload_fixture(apps, schema_editor):
       "Brutally deleting all entries for this model..."
   
       MyModel = apps.get_model("yourapp", "ModelName")
       MyModel.objects.all().delete()
   
   class Migration(migrations.Migration):  
   
       dependencies = [
           ('yourapp', '0001_initial'),
       ]
   
       operations = [
           migrations.RunPython(load_fixture, reverse_code=unload_fixture),
       ]
   

   2.2. A simpler solution for load_fixture(per @juliocesar's suggestion):

   from django.core.management import call_command
   
   fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
   fixture_filename = 'initial_data.json'
   
   def load_fixture(apps, schema_editor):
       fixture_file = os.path.join(fixture_dir, fixture_filename)
       call_command('loaddata', fixture_file) 
   

   Useful if you want to use a custom directory.

   2.3. Simplest:calling loaddatawith app_labelwill load fixtures from the <yourapp>'s fixturesdir automatically :

   from django.core.management import call_command
   
   fixture = 'initial_data'
   
   def load_fixture(apps, schema_editor):
       call_command('loaddata', fixture, app_label='yourapp') 
   

   If you don't specify app_label, loaddata will try to load fixturefilename from allapps fixtures directories (which you probably don't want).

3. Run it

   python manage.py migrate <yourapp>
   

1. 创建您的空迁移：

   在 Django 1.7 中：

   python manage.py makemigrations --empty <yourapp>
   

   在 Django 1.8+ 中，您可以提供一个名称：

   python manage.py makemigrations --empty <yourapp> --name load_intial_data
   

2. 编辑您的迁移文件 <yourapp>/migrations/0002_auto_xxx.py

   2.1. 自定义实现，灵感来自 Django' loaddata（初始答案）：

   import os
   from sys import path
   from django.core import serializers
   
   fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
   fixture_filename = 'initial_data.json'
   
   def load_fixture(apps, schema_editor):
       fixture_file = os.path.join(fixture_dir, fixture_filename)
   
       fixture = open(fixture_file, 'rb')
       objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
       for obj in objects:
           obj.save()
       fixture.close()
   
   def unload_fixture(apps, schema_editor):
       "Brutally deleting all entries for this model..."
   
       MyModel = apps.get_model("yourapp", "ModelName")
       MyModel.objects.all().delete()
   
   class Migration(migrations.Migration):  
   
       dependencies = [
           ('yourapp', '0001_initial'),
       ]
   
       operations = [
           migrations.RunPython(load_fixture, reverse_code=unload_fixture),
       ]
   

   2.2. 一个更简单的解决方案load_fixture（根据@juliocesar 的建议）：

   from django.core.management import call_command
   
   fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
   fixture_filename = 'initial_data.json'
   
   def load_fixture(apps, schema_editor):
       fixture_file = os.path.join(fixture_dir, fixture_filename)
       call_command('loaddata', fixture_file) 
   

   如果您想使用自定义目录，则很有用。

   2.3. 最简单：调用loaddatawithapp_label将自动从<yourapp>的fixtures目录加载设备：

   from django.core.management import call_command
   
   fixture = 'initial_data'
   
   def load_fixture(apps, schema_editor):
       call_command('loaddata', fixture, app_label='yourapp') 
   

   如果您不指定app_label，loaddata 将尝试fixture从所有应用程序装置目录（您可能不想要）加载文件名。

3. 运行

   python manage.py migrate <yourapp>
   

回答by Griffosx

In my opinion fixtures are a bit bad. If your database changes frequently, keeping them up-to-date will came a nightmare soon. Actually, it's not only my opinion, in the book "Two Scoops of Django" it's explained much better.

在我看来，固定装置有点糟糕。如果您的数据库频繁更改，保持它们最新将很快成为一场噩梦。实际上，这不仅是我的观点，在“Django 的两勺”一书中，它的解释要好得多。

Instead I'll write a Python file to provide initial setup. If you need something more I'll suggest you look at Factory boy.

相反，我将编写一个 Python 文件来提供初始设置。如果你需要更多的东西，我建议你看看工厂男孩。

If you need to migrate some data you should use data migrations.

如果您需要迁移某些数据，您应该使用数据迁移。

There's also "Burn Your Fixtures, Use Model Factories"about using fixtures.

还有关于使用设备的“烧掉你的设备，使用模型工厂”。

回答by FlogFR

In order to give your database some initial data, write a data migration.In the data migration, use the RunPythonfunction to load your data.

为了给你的数据库一些初始数据，写一个数据迁移。在数据迁移中，使用RunPython函数加载您的数据。

Don't write any loaddata command as this way is deprecated.

不要编写任何 loaddata 命令，因为这种方式已被弃用。

Your data migrations will be run only once. The migrations are an ordered sequence of migrations. When the 003_xxxx.py migrations is run, django migrations writes in the database that this app is migrated until this one (003), and will run the following migrations only.

您的数据迁移将只运行一次。迁移是有序的迁移序列。当运行 003_xxxx.py 迁移时，django 迁移会在数据库中写入此应用已迁移到此（003），并且只会运行以下迁移。

回答by alexhayes

Inspired by some of the comments (namely n__o's) and the fact that I have a lot of initial_data.*files spread out over multiple apps I decided to create a Django app that would facilitate the creation of these data migrations.

受到一些评论（即 n__o 的）的启发，以及我有很多initial_data.*文件分布在多个应用程序中的事实，我决定创建一个 Django 应用程序来促进这些数据迁移的创建。

Using django-migration-fixtureyou can simply run the following management command and it will search through all your INSTALLED_APPSfor initial_data.*files and turn them into data migrations.

使用django-migration-fixture，您只需运行以下管理命令，它将搜索您INSTALLED_APPS的所有initial_data.*文件并将它们转换为数据迁移。

./manage.py create_initial_data_fixtures
Migrations for 'eggs':
  0002_auto_20150107_0817.py:
Migrations for 'sausage':
  Ignoring 'initial_data.yaml' - migration already exists.
Migrations for 'foo':
  Ignoring 'initial_data.yaml' - not migrated.

See django-migration-fixturefor install/usage instructions.

有关安装/使用说明，请参阅django-migration-fixture。

回答by leifdenby

The solutions presented above didn't work for me unfortunately. I found that every time I change my models I have to update my fixtures. Ideally I would instead write data migrations to modify created data and fixture-loaded data similarly.

不幸的是，上面提出的解决方案对我不起作用。我发现每次更改模型时都必须更新灯具。理想情况下，我会改为编写数据迁移来修改创建的数据和夹具加载的数据。

To facilitate this I wrote a quick functionwhich will look in the fixturesdirectory of the current app and load a fixture. Put this function into a migration in the point of the model history that matches the fields in the migration.

为方便起见，我编写了一个快速函数，它将在fixtures当前应用程序的目录中查找并加载一个装置。将此函数放入与迁移中的字段匹配的模型历史记录点的迁移中。

回答by Rockallite

Short version

精简版

You should NOTuse loaddatamanagement command directly in a data migration.

您应不使用loaddata的数据迁移直接管理命令。

# Bad example for a data migration
from django.db import migrations
from django.core.management import call_command


def load_fixture(apps, schema_editor):
    # No, it's wrong. DON'T DO THIS!
    call_command('loaddata', 'your_data.json', app_label='yourapp')


class Migration(migrations.Migration):
    dependencies = [
        # Dependencies to other migrations
    ]

    operations = [
        migrations.RunPython(load_fixture),
    ]

Long version

长版

loaddatautilizes django.core.serializers.python.Deserializerwhich uses the most up-to-date models to deserialize historical data in a migration. That's incorrect behavior.

loaddata利用django.core.serializers.python.Deserializer它使用最新的模型来反序列化迁移中的历史数据。这是不正确的行为。

For example, supposed that there is a data migration which utilizes loaddatamanagement command to load data from a fixture, and it's already applied on your development environment.

例如，假设有一个数据迁移，它使用loaddata管理命令从夹具加载数据，并且它已经应用到您的开发环境中。

Later, you decide to add a new requiredfield to the corresponding model, so you do it and make a new migration against your updated model (and possibly provide a one-off value to the new field when ./manage.py makemigrationsprompts you).

稍后，您决定向相应的模型添加一个新的必填字段，因此您执行此操作并对更新后的模型进行新迁移（并可能在./manage.py makemigrations提示时为新字段提供一次性值）。

You run the next migration, and all is well.

您运行下一次迁移，一切都很好。

Finally, you're done developing your Django application, and you deploy it on the production server. Now it's time for you to run the whole migrations from scratch on the production environment.

最后，您已完成 Django 应用程序的开发，并将其部署在生产服务器上。现在是您在生产环境中从头开始运行整个迁移的时候了。

However, the data migration fails. That's because the deserialized model from loaddatacommand, which represents the current code, can't be saved with empty data for the new requiredfield you added. The original fixture lacks necessary data for it!

但是，数据迁移失败。这是因为来自loaddatacommand的反序列化模型（代表当前代码）无法与您添加的新必填字段的空数据一起保存。原始夹具缺少必要的数据！

But even if you update the fixture with required data for the new field, the data migration still fails. When the data migration is running, the nextmigration which adds the corresponding column to the database, is not applied yet. You can't save data to a column which does not exist!

但是即使您使用新字段所需的数据更新夹具，数据迁移仍然失败。当数据迁移正在运行时，将相应列添加到数据库的下一次迁移尚未应用。您无法将数据保存到不存在的列中！

Conclusion:in a data migration, the loaddatacommand introduces potential inconsistency between the model and the database. You should definitely NOTuse it directly in a data migration.

结论：在数据迁移中，该loaddata命令会在模型和数据库之间引入潜在的不一致。您绝对不应该在数据迁移中直接使用它。

The Solution

解决方案

loaddatacommand relies on django.core.serializers.python._get_modelfunction to get the corresponding model from a fixture, which will return the most up-to-date version of a model. We need to monkey-patch it so it gets the historical model.

loaddata命令依赖于django.core.serializers.python._get_model从夹具中获取相应模型的函数，这将返回模型的最新版本。我们需要对它进行猴子修补，以便获得历史模型。

(The following code works for Django 1.8.x)

（以下代码适用于 Django 1.8.x）

# Good example for a data migration
from django.db import migrations
from django.core.serializers import base, python
from django.core.management import call_command


def load_fixture(apps, schema_editor):
    # Save the old _get_model() function
    old_get_model = python._get_model

    # Define new _get_model() function here, which utilizes the apps argument to
    # get the historical version of a model. This piece of code is directly stolen
    # from django.core.serializers.python._get_model, unchanged. However, here it
    # has a different context, specifically, the apps variable.
    def _get_model(model_identifier):
        try:
            return apps.get_model(model_identifier)
        except (LookupError, TypeError):
            raise base.DeserializationError("Invalid model identifier: '%s'" % model_identifier)

    # Replace the _get_model() function on the module, so loaddata can utilize it.
    python._get_model = _get_model

    try:
        # Call loaddata command
        call_command('loaddata', 'your_data.json', app_label='yourapp')
    finally:
        # Restore old _get_model() function
        python._get_model = old_get_model


class Migration(migrations.Migration):
    dependencies = [
        # Dependencies to other migrations
    ]

    operations = [
        migrations.RunPython(load_fixture),
    ]

回答by Antony Fuentes Artavia

On Django 2.1, I wanted to load some models (Like country names for example) with initial data.

在 Django 2.1 上，我想用初始数据加载一些模型（例如国家/地区名称）。

But I wanted this to happen automatically right after the execution of initial migrations.

但是我希望在执行初始迁移后立即自动发生这种情况。

So I thought that it would be great to have an sql/folder inside each application that required initial data to be loaded.

所以我认为sql/在每个需要加载初始数据的应用程序中都有一个文件夹会很棒。

Then within that sql/folder I would have .sqlfiles with the required DMLs to load the initial data into the corresponding models, for example:

然后在该sql/文件夹中，我将拥有.sql包含所需 DML 的文件以将初始数据加载到相应的模型中，例如：

INSERT INTO appName_modelName(fieldName)
VALUES
    ("country 1"),
    ("country 2"),
    ("country 3"),
    ("country 4");

To be more descriptive, this is how an app containing an sql/folder would look:

为了更具描述性，以下是包含sql/文件夹的应用程序的外观：

Also I found some cases where I needed the sqlscripts to be executed in a specific order. So I decided to prefix the file names with a consecutive number as seen in the image above.

我还发现了一些需要sql以特定顺序执行脚本的情况。所以我决定在文件名前面加上一个连续的数字，如上图所示。

Then I needed a way to load any SQLsavailable inside any application folder automatically by doing python manage.py migrate.

然后我需要一种方法来自动加载SQLs任何应用程序文件夹中的任何可用文件python manage.py migrate。

So I created another application named initial_data_migrationsand then I added this app to the list of INSTALLED_APPSin settings.pyfile. Then I created a migrationsfolder inside and added a file called run_sql_scripts.py(Which actually is a custom migration). As seen in the image below:

所以我创建了另一个名为的应用程序initial_data_migrations，然后将此应用程序添加到文件列表INSTALLED_APPS中settings.py。然后我在migrations里面创建了一个文件夹并添加了一个名为run_sql_scripts.py（这实际上是一个自定义迁移）的文件。如下图所示：

I created run_sql_scripts.pyso that it takes care of running all sqlscripts available within each application. This one is then fired when someone runs python manage.py migrate. This custom migrationalso adds the involved applications as dependencies, that way it attempts to run the sqlstatements only after the required applications have executed their 0001_initial.pymigrations (We don't want to attempt running a SQL statement against a non-existent table).

我创建run_sql_scripts.py以便它负责运行sql每个应用程序中可用的所有脚本。当有人跑时，这个人就会被解雇python manage.py migrate。此自定义migration还将所涉及的应用程序添加为依赖项，这样它sql只会在所需的应用程序执行0001_initial.py迁移后才尝试运行语句（我们不想尝试针对不存在的表运行 SQL 语句）。

Here is the source of that script:

这是该脚本的来源：

import os
import itertools

from django.db import migrations
from YourDjangoProjectName.settings import BASE_DIR, INSTALLED_APPS

SQL_FOLDER = "/sql/"

APP_SQL_FOLDERS = [
    (os.path.join(BASE_DIR, app + SQL_FOLDER), app) for app in INSTALLED_APPS
    if os.path.isdir(os.path.join(BASE_DIR, app + SQL_FOLDER))
]

SQL_FILES = [
    sorted([path + file for file in os.listdir(path) if file.lower().endswith('.sql')])
    for path, app in APP_SQL_FOLDERS
]


def load_file(path):
    with open(path, 'r') as f:
        return f.read()


class Migration(migrations.Migration):

    dependencies = [
        (app, '__first__') for path, app in APP_SQL_FOLDERS
    ]

    operations = [
        migrations.RunSQL(load_file(f)) for f in list(itertools.chain.from_iterable(SQL_FILES))
    ]

I hope someone finds this helpful, it worked just fine for me!. If you have any questions please let me know.

我希望有人觉得这有帮助，它对我来说很好用！如果您有任何问题，请告诉我。

NOTE: This might not be the best solution since I'm just getting started with django, however still wanted to share this "How-to" with you all since I didn't find much information while googling about this.

注意：这可能不是最好的解决方案，因为我刚刚开始使用 django，但是仍然想与大家分享这个“操作方法”，因为我在谷歌搜索时没有找到太多信息。