mongodb 修改和重放MongoDB oplog

Modify and replay MongoDB oplog

提问by michaeltwofish

Is it possible to modify the MongoDB oplog and replay it?

是否可以修改 MongoDB oplog 并重放它？

A bug caused an update to be applied to more documents than it was supposed to be, overwriting some data. Data was recovered from backup and reintegrated, so nothing was actually lost, but I was wondering if there was a way to modify the oplog to remove or modify the offending update and replay it.

一个错误导致更新应用于比预期更多的文档，覆盖了一些数据。数据从备份中恢复并重新集成，所以实际上没有丢失任何东西，但我想知道是否有办法修改 oplog 以删除或修改有问题的更新并重放它。

I don't have in depth knowledge of MongoDB internals, so informative answers along the lines of, "you don't understand how it works, it's like this" will also be considered for acceptance.

我对 MongoDB 内部没有深入的了解，因此也将考虑接受诸如“您不了解它是如何工作的，它是这样的”之类的信息丰富的答案。

回答by Asya Kamsky

One of the big issues in application or human error data corruption is that the offending write to the primary will immediately be replicated to the secondary.

应用程序或人为错误数据损坏的一大问题是对主服务器的违规写入将立即复制到辅助服务器。

This is one of the reasons that users take advantage of "slaveDelay" - an option to run one of your secondary nodes with a fixed time delay (of course that only helps you if you discover the error or bug during the time period that's shorter than the delay on that secondary).

这是用户利用“slaveDelay”的原因之一 - 一种以固定时间延迟运行您的辅助节点之一的选项（当然，只有在您发现错误或错误的时间段内比该次要的延迟）。

In case you don't have such a set-up, you have to rely on a backup to recreate the state of the records you need to restore to their pre-bug state.

如果您没有这样的设置，则必须依靠备份来重新创建需要恢复到错误前状态的记录的状态。

Perform all the operations on a separate stand-alone copy of your data - only after verifying that everything was properly recreated should you then move the corrected data into your production system.

在单独的独立数据副本上执行所有操作 - 只有在验证所有内容都已正确重新创建之后，您才能将更正后的数据移动到您的生产系统中。

What is required to be able to do this is a recent copy of the backup (let's say the backup is X hours old) and the oplog on your cluster must hold more than X hours worth of data. I didn't specify which node's oplog because (a) every member of the replica set has the same contents in the oplog and (b) it ispossible that your oplog size is different on different node members, in which case you want to check the "largest" one.

能够做到这一点所需要的是备份的最新副本（假设备份是 X 小时前的），并且集群上的 oplog 必须保存超过 X 小时的数据。因为（a）每一个副本集的成员在OPLOG及（b）是相同的内容我没有指定哪个节点的OPLOG是可能是您的OPLOG大小是不同的节点成员不同，在这种情况下，要检查“最大”的那个。

So let's say your most recent backup is 52 hours old, but luckily you have an oplog that holds 75 hours worth of data (yay).

因此，假设您最近的备份是 52 小时前的，但幸运的是，您有一个保存 75 小时数据的 oplog（是的）。

You already realized that all of your nodes (primary and secondaries) have the "bad" data, so what you would do is restore this most recent backup into a new mongod. This is where you will restore these records to what they were right before the offending update - and then you can just move them into the current primary from where they will get replicated to all the secondaries.

您已经意识到您的所有节点（主节点和辅助节点）都有“坏”数据，因此您要做的是将这个最新的备份恢复到一个新的 mongod 中。在这里，您可以将这些记录恢复到有问题的更新之前的状态 - 然后您可以将它们移动到当前主服务器中，从那里它们将被复制到所有辅助服务器。

While restoring your backup, create a mongodump of your oplog collection via this command:

在恢复备份时，通过以下命令创建 oplog 集合的 mongodump：

mongodump -d local -c oplog.rs -o oplogD

mongodump -d local -c oplog.rs -o oplogD

Move the oplog to its own directory renaming it to oplog.bson:

将 oplog 移动到它自己的目录，将其重命名为 oplog.bson：

mkdir oplogR
mv oplogD/local/oplog.rs.bson oplogR/oplog.bson

Now you need to find the "offending" operation. You can dump out the oplog to human readable form, using the bsondumpcommand on oplogR/oplog.bson file (and then use grep or what-not to find the "bad" update). Alternatively you can query against the original oplog in the replica set via use localand db.oplog.rs.find()commands in the shell.

现在您需要找到“违规”操作。您可以使用bsondumpoplogR/oplog.bson 文件上的命令将 oplog 转储为人类可读的形式（然后使用 grep 或 what-not 来查找“坏”更新）。或者，您可以通过shell 中的use local和db.oplog.rs.find()命令查询副本集中的原始 oplog 。

Your goal is to find this entry and note its tsfield.

您的目标是找到此条目并记下其ts字段。

It might look like this:

它可能看起来像这样：

"ts" : Timestamp( 1361497305, 2789 )

"ts" : Timestamp( 1361497305, 2789 )

Note that the mongorestorecommand has two options, one called --oplogReplayand the other called oplogLimit. You will now replay this oplog on the restored stand-alone server BUT you will stop before this offending update operation.

请注意，该mongorestore命令有两个选项，一个称为--oplogReplay，另一个称为oplogLimit。您现在将在恢复的独立服务器上重放此 oplog，但您将在此违规更新操作之前停止。

The command would be (host and port are where your newly restored backup is):

命令将是（主机和端口是您新恢复的备份所在的位置）：

mongorestore -h host --port NNNN --oplogReplay --oplogLimit 1361497305:2789 oplogR

mongorestore -h host --port NNNN --oplogReplay --oplogLimit 1361497305:2789 oplogR

This will restore each operation from the oplog.bson file in oplogR directory stopping right before the entry with ts value Timestamp(1361497305, 2789).

这将从 oplogR 目录中的 oplog.bson 文件中恢复每个操作，在具有 ts 值 Timestamp(1361497305, 2789) 的条目之前停止。

Recall that the reason you were doing this on a separate instance is so you can verify the restore and replay created correct data - once you have verified it then you can write the restored records to the appropriate place in the real primary (and allow replication propagate the corrected records to the secondaries).

回想一下，您在单独的实例上执行此操作的原因是，您可以验证还原和重放创建的正确数据 - 一旦验证了它，您就可以将还原的记录写入真实主服务器中的适当位置（并允许复制传播更正后的记录）。