Ruby-on-rails 登录到数据库而不是日志文件

Log to database instead of log files

提问by chrisrbailey

I'm interested in sending all Rails application logging to a database (MySQL or MongoDB) either in addition to or instead of to a log file. There are a few reasons, most of which are concerned about log file analysis. We already use Google Analytics, but there are a variety of things we want to do that aren't as workable in Analytics.

我有兴趣将所有 Rails 应用程序日志发送到数据库（MySQL 或 MongoDB），除了日志文件之外，或者代替日志文件。有几个原因，其中大部分与日志文件分析有关。我们已经在使用 Google Analytics，但是我们想做的很多事情在 Analytics 中并不可行。

Furthermore, I'd like to do "real time" investigation of issues by looking at logs. Sifting through a log file is a tedious way to do that, and I'd like to do better searching and filtering than a log file (easily) allows for.

此外，我想通过查看日志对问题进行“实时”调查。筛选日志文件是一种繁琐的方法，我希望进行比日志文件（轻松）允许的更好的搜索和过滤。

Finally, I often want to examine something closer to site visitor behavior: tracing the path through the site for example, so that I can see what the last page was that a user was looking at before an error occurred. Given we have multiple app servers, the separate log files make this a real pain. If all the data were in a database, I could then easily see the proper sequence of pages for a given visitor. I know that Syslog would be one way to solve this particular thing (single log file/repository), but I want to combine that with better searching abilities that I associate with database searches.

最后，我经常想检查一些更接近站点访问者行为的东西：例如，跟踪通过站点的路径，以便我可以看到用户在发生错误之前查看的最后一页是什么。鉴于我们有多个应用服务器，单独的日志文件让这变得非常痛苦。如果所有数据都在一个数据库中，那么我就可以轻松查看给定访问者的正确页面顺序。我知道 Syslog 将是解决这个特定问题（单个日志文件/存储库）的一种方法，但我想将它与我与数据库搜索相关联的更好的搜索能力结合起来。

I'm wondering what folks recommend to solve this. Do you directly log to a database, or do you dump log files into a DB (but what's your approach for that so that it's essentially realtime/as up to date as the logfile itself)?

我想知道人们推荐什么来解决这个问题。您是直接登录到数据库，还是将日志文件转储到数据库中（但是您的方法是什么，以便它基本上是实时的/与日志文件本身一样最新）？

I am currently determining at what level I'd like this logging, because another thing I looked at is writing a small Rack filter that would log all requests. This would miss all the extra output that the normal Rails logging dumps out (all the SQL and output on cache hits and misses, etc.), but it would achieve a big part of my goal, and seems to have the advantage of not disturbing anything else in the system.

我目前正在确定我想要这个日志记录的级别，因为我看到的另一件事是编写一个小的 Rack 过滤器来记录所有请求。这会错过正常 Rails 日志转储出来的所有额外输出（所有 SQL 和缓存命中和未命中的输出等），但它会实现我的目标的很大一部分，并且似乎具有不打扰的优势系统中的任何其他内容。

Anyway, I am not looking for one right answer, more of a discussion and information on what anyone else might be doing in this same light.

无论如何，我不是在寻找一个正确的答案，更多的是关于其他人在同样的情况下可能会做什么的讨论和信息。

回答by newtonapple

My company have been logging some structured traffic info straight into a MySQL log database. This database is replicated downstream to another database. All analytics run off the final database replication. Our site sustain quite a bit of traffic. So far, it doesn't seem to have any major problems. However, our IT department have some growing concerns regarding to the scalability of the current setup and is suggesting that we offload the log info onto "proper" log-files. The log-files will then be reinserted back into the same downstream database tables. Which brings me to this question. :)

我的公司一直在将一些结构化的流量信息直接记录到 MySQL 日志数据库中。该数据库被下游复制到另一个数据库。所有分析都在最终的数据库复制之后运行。我们的网站维持着相当多的流量。到目前为止，它似乎没有任何重大问题。但是，我们的 IT 部门对当前设置的可扩展性越来越担心，并建议我们将日志信息卸载到“正确的”日志文件中。然后日志文件将重新插入到相同的下游数据库表中。这让我想到了这个问题。:)

Here are some of pros and cons that I see regarding to the subject of log-files vs log-db (relational):

以下是我看到的关于日志文件与日志数据库（关系）主题的一些优缺点：

• log-files are fast, reliable, and scalable (At least I have heard Yahoo! makes heavy uses of log files for their click tracking analytics).
• log-files are easy for sys-admin to maintain.
• log-files can be very flexible since you can write almost anything to it.
• log-files requires heavy parsing and potentially a map-reduced type of setup for data-extraction.
• log-db structures are a lot closer to your application, making some feature's turn around time a lot shorter. This can be a blessing or a curse. Probably a curse in the long run since you'll most likely end up with a highly coupled application and analytic code base.
• log-db can reduce logging noises and redundancies since log-files are insert only where as log-db gives you the ability to do update and associated-insert (normalization if you dare).
• log-db can be fast and scalable too if you go with database partitioning and/or multi-log databases (rejoin data via downstream replications)

• 日志文件快速、可靠且可扩展（至少我听说雅虎大量使用日志文件进行点击跟踪分析）。
• 日志文件很容易让系统管理员维护。
• 日志文件可以非常灵活，因为您几乎可以向其中写入任何内容。
• 日志文件需要大量解析，并且可能需要减少地图类型的数据提取设置。
• log-db 结构更接近您的应用程序，从而使某些功能的周转时间更短。这可能是一种祝福，也可能是一种诅咒。从长远来看可能是一个诅咒，因为您很可能最终会得到一个高度耦合的应用程序和分析代码库。
• log-db 可以减少日志记录噪音和冗余，因为日志文件仅在 log-db 使您能够进行更新和相关插入（如果您敢的话，可以进行标准化）时插入。
• 如果您使用数据库分区和/或多日志数据库（通过下游复制重新加入数据），log-db 也可以快速且可扩展

I think some stress tests on the log database are needed in my situation. This way at least I know how much headroom I have.

我认为在我的情况下需要对日志数据库进行一些压力测试。这样至少我知道我有多少净空。

Recently, I've been looking into some key-value / document-based databases like Redis, Tokyo Cabinet, and MongoDB. These fast inserting databases can potentially be the sweet spot since they provide persistence, high (write) throughputs, and querying capabilities to varying degrees. They can make the data-extraction process much simpler than parsing and map-reducing through gigs of log files.

最近，我一直在研究一些基于键值/文档的数据库，例如 Redis、Tokyo Cabinet 和 MongoDB。这些快速插入的数据库可能是最佳选择，因为它们提供了不同程度的持久性、高（写入）吞吐量和查询功能。它们可以使数据提取过程比通过日志文件的演出进行解析和减少映射简单得多。

In the long run, I believe it is crucial to have a robust analytics data warehouse. Freeing application data from analytic data and vice versa can be a big WIN.

从长远来看，我认为拥有强大的分析数据仓库至关重要。从分析数据中释放应用程序数据，反之亦然，这可能是一个巨大的胜利。

---

---

Lastly, I would just like to point out there are many similar / closely related questions here on StackOverflow in case you want to broaden your discussion.

最后，我想指出 StackOverflow 上有许多类似/密切相关的问题，以防您想扩大讨论范围。

• Storage of many log files
• Is writing server log files to a database a good idea?
• Using a SQL Server for application logging. Pros/Cons?
• Fast Search in Logs
• Separate production database for logging
• You Log to Your DB, Where Do You Log When Your DB is Down?

• 存储许多日志文件
• 将服务器日志文件写入数据库是个好主意吗？
• 使用 SQL Server 进行应用程序日志记录。优点缺点？
• 在日志中快速搜索
• 用于日志记录的单独生产数据库
• 您登录到您的数据库，当您的数据库关闭时，您在哪里登录？

---

---

Edit:

编辑：

rsysloglooks very interesting. It gives you the ability to write directly to MySQL. If you are using Ruby, you should have a look at the logging gem. It provides multi-target logging capabilities. It's really nice.

rsyslog看起来很有趣。它使您能够直接写入 MySQL。如果您使用 Ruby，您应该查看日志记录 gem。它提供多目标日志记录功能。这太好了。

回答by Simone Carletti

If you want to change the default logging behavior, simply create a custom logger object that respond to all the Rails logger method:

如果要更改默认日志记录行为，只需创建一个响应所有 Rails 记录器方法的自定义记录器对象：

• add
• debug, warn, error, info, fatal, unknown

• 添加
• 调试、警告、错误、信息、致命、未知

http://github.com/rails/rails/blob/9d7aae710384fb5f04129c35b86c5ea5fb9d83a9/activesupport/lib/active_support/buffered_logger.rb

http://github.com/rails/rails/blob/9d7aae710384fb5f04129c35b86c5ea5fb9d83a9/activesupport/lib/active_support/buffered_logger.rb

Because it's yourlogger, you can decide to implement your personal logic. You can write to the database, to the standard output of whenever you want.

因为它是您的记录器，您可以决定实现您的个人逻辑。您可以随时写入数据库，写入标准输出。

Then, replace the default logger for every base class you want to customize.

然后，为您要自定义的每个基类替换默认记录器。

ActiveRecord::Base.logger = YouLogger.new

You can easily create an initializer file called logger.rb and write there all your custom configurations. In this way, the logger will be immediately replaced on Rails startup.

您可以轻松创建一个名为 logger.rb 的初始化文件，并在其中写入所有自定义配置。这样，记录器将在 Rails 启动时立即被替换。

回答by atmorell

I use the rails "exception logger", to log all problems to my database while my site is in production mode. It will give you a nice interface where you can check for problems. If you want to see what your visitors are doing in realtime then take a look at woopra

当我的站点处于生产模式时，我使用 rails "exception logger"将所有问题记录到我的数据库中。它将为您提供一个很好的界面，您可以在其中检查问题。如果您想实时查看访问者在做什么，请查看woopra

回答by Nishad

Chris,

克里斯，

I think Dima's comment is important here. Are you satisfied with (1) having an access log in a DB (in real time), or (2) are you more interested in Rails/app-specific logging?

我认为 Dima 的评论在这里很重要。您是否对 (1) 在数据库中拥有访问日志（实时）感到满意，或者 (2) 您是否对 Rails/特定于应用程序的日志记录更感兴趣？

For (1), with Apache (at least), you can log to a database using piped logging.

对于 (1)，使用 Apache（至少），您可以使用管道日志记录到数据库。

http://httpd.apache.org/docs/1.3/logs.html#piped

http://httpd.apache.org/docs/1.3/logs.html#piped

I wrote a program that runs in the background waiting for input, which it parses and logs to a Postgres DB. My httpd.conf file pipes to this program with a CustomLog directive.

我写了一个在后台运行等待输入的程序，它解析并记录到 Postgres 数据库。我的 httpd.conf 文件通过 CustomLog 指令连接到这个程序。

This is relatively simple to set up, and gives you all the obvious advantages of being able to analyze your logs in a DB. It works very well for me, especially for tracing what a user was doing just before an error. However, you have to protect against sql injection, buffer overflows, and other security issues in the logging program.

这设置起来相对简单，并且为您提供了能够在数据库中分析日志的所有明显优势。它对我来说非常有效，特别是用于跟踪用户在发生错误之前正在做什么。但是，您必须在日志记录程序中防止 sql 注入、缓冲区溢出和其他安全问题。

For (2), I am not a Rails developer so I can only talk about general approaches. If you want to log environment vars, or application data, or very selective bits of information, you could consider writing a web server module. Depending on your exact needs, you could also get by with some combination of conditional logging directives and filtering in the logging program.

对于 (2)，我不是 Rails 开发人员，所以我只能谈谈一般方法。如果您想记录环境变量、应用程序数据或非常有选择性的信息位，您可以考虑编写一个 Web 服务器模块。根据您的确切需求，您还可以在日志程序中使用条件日志指令和过滤的某种组合。

It really comes down to whether you need a Rails-specific solution or a more general web-server-wide solution.

这实际上取决于您是需要特定于 Rails 的解决方案还是更通用的 Web 服务器范围的解决方案。

回答by VP.

as no answer was accepted until now, i will give my contribution

由于直到现在都没有接受答案，我会做出我的贡献

i did develop a plugin to rsylog to save the logs not in files but at mongodb

我确实为 rsylog 开发了一个插件来将日志保存在文件中而不是在 mongodb 中

the whole source code, from rsyslog + plugin is here https://github.com/vpereira/rsyslogd-mongo

来自 rsyslog + 插件的整个源代码在这里https://github.com/vpereira/rsyslogd-mongo

to compile it, you should just run ./configure --help and see the available options.

要编译它，您应该只运行 ./configure --help 并查看可用选项。

回答by Bob Aman

Having made the mistake of logging to a database recently myself, I feel I can offer one extremely good reason why you should not do this: Transactions. Let's say you start a transaction, log a bunch of stuff during the course of the transaction, and ultimate you end up with an error condition. You log the error condition, and oh hey. ROLLBACK. Suddenly, everything you just logged is gone and you have no idea what happened or why.

最近我自己犯了登录数据库的错误，我觉得我可以提供一个非常好的理由为什么你不应该这样做：事务。假设你开始一个事务，在事务过程中记录一堆东西，最终你会遇到一个错误情况。你记录错误情况，哦，嘿。ROLLBACK. 突然，您刚刚记录的所有内容都消失了，您不知道发生了什么或为什么。

And particularly in the context of Rails, where really useful libraries like AASM will wrap a whole bunch of stuff in a transaction, you can end up with transactions in places you didn't think you would, which also makes the problem very hard to debug.

尤其是在 Rails 的上下文中，像 AASM 这样真正有用的库会将一大堆东西包装在一个事务中，你最终可能会在你认为不会的地方处理事务，这也使得问题很难调试.

In my case, the reason I logged things to the database was that I needed context-sensitive logs. Essentially I needed to be able to look up all log entries related to a specific database model. However, the right answer is to put those logs in some separate location that's a better fit for log data (and which, in my case, happens to be query-able).

就我而言，我将事情记录到数据库的原因是我需要上下文敏感的日志。本质上，我需要能够查找与特定数据库模型相关的所有日志条目。但是，正确的答案是将这些日志放在更适合日志数据的某个单独位置（在我的情况下，恰好是可查询的）。