php 你如何实现一个好的亵渎过滤器？

How do you implement a good profanity filter?

提问by Ben Throop

Many of us need to deal with user input, search queries, and situations where the input text can potentially contain profanity or undesirable language. Oftentimes this needs to be filtered out.

我们中的许多人需要处理用户输入、搜索查询以及输入文本可能包含亵渎或不良语言的情况。通常，这需要过滤掉。

Where can one find a good list of swear words in various languages and dialects?

哪里可以找到各种语言和方言的脏话清单？

Are there APIs available to sources that contain good lists? Or maybe an API that simply says "yes this is clean" or "no this is dirty" with some parameters?

是否有可用于包含良好列表的源的 API？或者也许一个 API 只是简单地说“是的，这很干净”或“不，这很脏”带有一些参数？

What are some good methods for catching folks trying to trick the system, like a$$, azz, or a55?

有什么好的方法可以抓住试图欺骗系统的人，例如 a$$、azz 或 a55？

Bonus points if you offer solutions for PHP. :)

如果您提供 PHP 解决方案，则加分。:)

Edit: Response to answers that say simply avoid the programmatic issue:

编辑：对只是避免编程问题的答案的回应：

I think there is a place for this kind of filter when, for instance, a user can use public image search to find pictures that get added to a sensitive community pool. If they can search for "penis", then they will likely get many pictures of, yep. If we don't want pictures of that, then preventing the word as a search term is a good gatekeeper, though admittedly not a foolproof method. Getting the list of words in the first place is the real question.

我认为有这种过滤器的地方，例如，当用户可以使用公共图像搜索来查找添加到敏感社区池的图片时。如果他们可以搜索“阴茎”，那么他们可能会得到很多照片，是的。如果我们不想要它的图片，那么阻止这个词作为搜索词是一个很好的看门人，尽管不可否认，这不是一个万无一失的方法。首先获取单词列表是真正的问题。

So I'm really referring to a way to figure out of a single token is dirty or not and then simply disallow it. I'd not bother preventing a sentiment like the totally hilarious "long necked giraffe" reference. Nothing you can do there. :)

所以我真的指的是一种确定单个令牌是否脏的方法，然后简单地禁止它。我不会费心阻止像完全热闹的“长颈鹿”参考这样的情绪。在那里你无能为力。:)

采纳答案by HanClinto

Obscenity Filters: Bad Idea, or Incredibly Intercoursing Bad Idea?

淫秽过滤器：坏主意，还是令人难以置信的交流坏主意？

Also, one can't forget The Untold History of Toontown's SpeedChat, where even using a "safe-word whitelist" resulted in a 14 year old quickly circumventing it with: "I want to stick my long-necked Giraffe up your fluffy white bunny."

此外，人们不能忘记Toontown 的 SpeedChat 的不为人知的历史，即使使用“安全词白名单”也会导致一个 14 岁的孩子迅速绕过它： “我想把我的长颈鹿放在你毛茸茸的白色兔子身上.”

Bottom line: Ultimately, for any system that you implement, there is absolutely no substitute for human review (whether peer or otherwise). Feel free to implement a rudimentary tool to get rid of the drive-by's, but for the determined troll, you absolutely must have a non-algorithm-based approach.

底线：最终，对于您实施的任何系统，绝对无法替代人工（无论是同行还是其他人）。随意实施一个基本的工具来摆脱偷渡的问题，但对于坚定的巨魔，你绝对必须有一种非基于算法的方法。

A system that removes anonymity and introduces accountability (something that Stack Overflow does well) is helpful also, particularly in order to help combat John Gabriel's G.I.F.T.

一个消除匿名并引入问责制的系统（Stack Overflow 做得很好）也很有帮助，特别是为了帮助打击John Gabriel 的 GIFT

You also asked where you can get profanity lists to get you started -- one open-source project to check out is Dansguardian-- check out the source code for their default profanity lists. There is also an additional third party Phrase Listthat you can download for the proxy that may be a helpful gleaning point for you.

您还询问了在哪里可以获得脏话列表以帮助您入门——一个需要检查的开源项目是Dansguardian—— 查看他们默认脏话列表的源代码。您还可以为代理下载一个额外的第三方短语列表，这对您来说可能是一个有用的收集点。

Edit in response the question edit:Thanks for the clarification on what you're trying to do. In that case, if you're just trying to do a simple word filter, there are two ways you can do it. One is to create a single long regexp with all of the banned phrases that you want to censor, and merely do a regex find/replace with it. A regex like:

编辑以回答问题编辑：感谢您对您正在尝试做的事情的澄清。在这种情况下，如果您只是想进行简单的单词过滤，有两种方法可以做到。一种是使用您要的所有被禁止的短语创建一个长正则表达式，并仅用它进行正则表达式查找/替换。一个正则表达式，如：

$filterRegex = "(boogers|snot|poop|shucks|argh)"

and run it on your input string using preg_match()to wholesale test for a hit,

并使用preg_match()在您的输入字符串上运行它以批量测试命中，

or preg_replace()to blank them out.

或preg_replace()将它们清空。

You can also load those functions up with arrays rather than a single long regex, and for long word lists, it may be more manageable. See the preg_replace()for some good examples as to how arrays can be used flexibly.

您还可以使用数组而不是单个长正则表达式加载这些函数，对于长单词列表，它可能更易于管理。有关如何灵活使用数组的一些很好的示例，请参阅preg_replace()。

For additional PHP programming examples, see this page for a somewhat advanced generic classfor word filtering that *'s out the center letters from censored words, and this previous Stack Overflow questionthat also has a PHP example (the main valuable part in there is the SQL-based filtered word approach -- the leet-speak compensator can be dispensed with if you find it unnecessary).

有关其他 PHP 编程示例，请参阅此页面，了解用于单词过滤的某种高级通用类，该类从的单词中去掉中心字母，以及之前的 Stack Overflow 问题也有一个 PHP 示例（其中主要有价值的部分是基于 SQL 的过滤词方法——如果你觉得没有必要，可以省去 leet-speak 补偿器）。

You also added: "Getting the list of words in the first place is the real question." -- in addition to some of the previous Dansgaurdian links, you may find this handy .zipof 458 words to be helpful.

您还补充说：“首先获取单词列表才是真正的问题。”——除了之前的一些 Dansgaurdian 链接之外，您可能会发现这个包含 458 个单词的方便的 .zip很有帮助。

回答by nickhar

Whilst I know that this question is fairly old, but it's a commonly occurring question...

虽然我知道这个问题已经很老了，但这是一个普遍存在的问题......

There is both a reason and a distinct need for profanity filters (see Wikipedia entry here), but they often fall short of being 100% accurate for very distinct reasons; Contextand accuracy.

脏话过滤器是有原因的，也有明显的需求（请参阅此处的维基百科条目），但由于非常明显的原因，它们通常无法达到 100% 准确；上下文和准确性。

It depends (wholly) on what you're trying to achieve - at it's most basic, you're probably trying to cover the "seven dirty words" and then some... Some businesses need to filter the most basic of profanity: basic swear words, URLs or even personal information and so on, but others need to prevent illicit account naming (Xbox live is an example) or far more...

这（完全）取决于您要实现的目标 - 在最基本的情况下，您可能试图涵盖“七个肮脏的词”，然后是一些......一些企业需要过滤最基本的亵渎：基本脏话、URL 甚至个人信息等等，但其他人需要防止非法帐户命名（Xbox live 就是一个例子）或更多......

User generated content doesn't just contain potential swear words, it can also contain offensive references to:

用户生成的内容不仅包含潜在的脏话，还可能包含对以下内容的冒犯性引用：

• Sexual acts
• Sexual orientation
• Religion
• Ethnicity
• Etc...

• 性行为
• 性取向
• 宗教
• 种族
• 等等...

And potentially, in multiple languages. Shutterstock has developed basic dirty-words listsin 10 languages to date, but it's still basic and very much oriented towards their 'tagging' needs. There are a number of other lists available on the web.

并且可能有多种语言。迄今为止，Shutterstock 已经开发了 10 种语言的基本脏词列表，但它仍然是基本的，并且非常面向他们的“标记”需求。网络上还有许多其他列表。

I agree with the accepted answer that it's not a defined science and aslanguage is a continually evolving challengebut one where a 90% catch rate is better than 0%. It depends purely on your goals - what you're trying to achieve, the level of support you have and how important it is to remove profanities of different types.

我同意公认的答案，即它不是一门明确的科学，因为语言是一项不断发展的挑战，但 90% 的捕获率优于 0%。这完全取决于您的目标——您想要达到的目标、您获得的支持水平以及删除不同类型的脏话的重要性。

In building a filter, you need to consider the following elements and how they relate to your project:

在构建过滤器时，您需要考虑以下元素以及它们与您的项目的关系：

• Words/phrases
• Acronyms (FOAD/LMFAO etc)
• False positives(words, places and names like 'mishit', 'scunthorpe' and 'titsworth')
• URLs (porn sites are an obvious target)
• Personal information (email, address, phone etc - if applicable)
• Language choice (usually English by default)
• Moderation (how, if at all, you can interact with user generated content and what you can do with it)

• 单词/短语
• 首字母缩略词（FOAD/LMFAO 等）
• 误报（单词、地点和名称，如“mishit”、“scunthorpe”和“titsworth”）
• URL（网站是一个明显的目标）
• 个人信息（电子邮件、地址、电话等 - 如果适用）
• 语言选择（通常默认为英语）
• 审核（如果有的话，您如何与用户生成的内容进行交互以及您可以用它做什么）

You can easily build a profanity filter that captures 90%+ of profanities, but you'll never hit 100%. It's just not possible. The closer you want to get to 100%, the harder it becomes... Having built a complex profanity engine in the past that dealt with more than 500K realtime messages per day, I'd offer the following advice:

您可以轻松构建一个可捕获 90% 以上的脏话的脏话过滤器，但您永远不会达到 100%。这是不可能的。你越想接近 100%，它就越难......在过去构建了一个复杂的亵渎引擎，每天处理超过 500K 的实时消息，我会提供以下建议：

A basic filter would involve:

基本过滤器将涉及：

• Building a list of applicable profanities
• Developing a method of dealing with derivations of profanities

• 建立适用的脏话清单
• 开发一种处理脏话派生词的方法

A moderately complex filer would involve, (In addition to a basic filter):

一个中等复杂的过滤器将涉及，（除了基本过滤器）：

• Using complex pattern matching to deal with extended derivations (using advanced regex)
• Dealing with Leetspeak(l33t)
• Dealing with false positives

• 使用复杂模式匹配来处理扩展派生（使用高级正则表达式）
• 处理Leetspeak(l33t)
• 处理误报

A complex filter would involve a number of the following (In addition to a moderate filter):

复杂的过滤器将涉及以下多项内容（除中等过滤器外）：

• Whitelistsand blacklists
• Naive bayesian inferencefiltering of phrases/terms
• Soundexfunctions (where a word sounds like another)
• Levenshtein distance
• Stemming
• Human moderators to help guide a filtering engine to learn by example or where matches aren't accurate enough without guidance (a self/continually-improving system)
• Perhaps some form of AI engine

• 白名单和黑名单
• 短语/术语的朴素贝叶斯推理过滤
• Soundex函数（一个词听起来像另一个词）
• 莱文斯坦距离
• 词干
• 人类版主帮助指导过滤引擎通过示例学习或在没有指导的情况下匹配不够准确（自我/持续改进系统）
• 也许某种形式的人工智能引擎

回答by Matt Passell

I don't know of any good libraries for this, but whatever you do, make sure that you err in the direction of letting stuff through. I've dealt with systems that wouldn't allow me to use "mpassell" as a username, because it contains "ass" as a substring. That's a great way to alienate users!

我不知道有什么好的库可以做到这一点，但无论你做什么，都要确保你在让东西通过的方向上犯错。我处理过的系统不允许我使用“mpassell”作为用户名，因为它包含“ass”作为子字符串。这是疏远用户的好方法！

回答by Matthew

During a job interview of mine, the company CTO who was interviewing me tried out a word/web game I wrote in Java. Out of a word list of the entire Oxford English dictionary, what was the first word that came up to be guessed?

在我的求职面试中，面试我的公司 CTO 试用了我用 Java 编写的文字/网页游戏。在整个牛津英语词典的词表中，第一个猜到的词是什么？

Of course, the most foul word in the English language.

当然是英语里最脏的词了。

Somehow, I still got the job offer, but I then tracked down a profanity word list (not unlike this one) and wrote a quick script to generate a new dictionary without all of the bad words (without even having to look at the list).

不知何故，我仍然得到了工作机会，但我随后找到了一个脏话列表（与这个不同）并编写了一个快速脚本来生成一个没有所有坏词的新词典（甚至不必查看列表） .

For your particular case, I think comparing the search to real words sounds like the way to go with a word list like that. The alternative styles/punctuation require a bit more work, but I doubt users will use that often enough to be an issue.

对于您的特定情况，我认为将搜索与真实单词进行比较听起来像是使用这样的单词列表的方式。替代样式/标点符号需要更多的工作，但我怀疑用户是否会经常使用它来成为问题。

回答by Steven A. Lowe

a profanity filtering system will never be perfect, even if the programmer is cocksure and keeps abreast of all nude developments

脏话过滤系统永远不会是完美的，即使程序员很自信并且跟上所有裸体的发展

that said, any list of 'naughty words' is likely to perform as well as any other list, since the underlying problem is language understandingwhich is pretty much intractable with current technology

也就是说，任何“顽皮词”列表的表现都可能与任何其他列表一样好，因为潜在的问题是语言理解，这对于当前的技术来说非常棘手

so, the only practical solution is twofold:

因此，唯一实用的解决方案是双重的：

1. be prepared to update your dictionary frequently
2. hire a human editor to correct false positives (e.g. "clbuttic" instead of "classic") and false negatives (oops! missed one!)

1. 准备好经常更新你的字典
2. 聘请人工编辑来纠正误报（例如“clbuttic”而不是“经典”）和漏报（哎呀！错过了一个！）

回答by Axel

The only way to prevent offensive user input is to prevent all user input.

防止冒犯性用户输入的唯一方法是防止所有用户输入。

If you insist on allowing user input and need moderation, then incorporate human moderators.

如果您坚持允许用户输入并需要审核，那么请加入人工审核员。

回答by Tim Cavanaugh

Have a look at CDYNE's Profanity Filter Web Service

看看CDYNE 的亵渎过滤器 Web 服务

Testing URL

测试网址

回答by Dave Sherohman

Regarding your "trick the system" subquestion, you can handle that by normalizing both the "bad word" list and the user-entered text before doing your search. e.g., Use a series of regexes (or trif PHP has it) to convert [z$5]to "s", [4@]to "a", etc., then compare the normalized "bad word" list against the normalized text. Note that the normalization could potentially lead to additional false positives, although I can't think of any actual cases at the moment.

关于您的“欺骗系统”子问题，您可以在进行搜索之前通过规范化“坏词”列表和用户输入的文本来处理该问题。例如，使用一系列正则表达式（如果 PHP 有，则使用tr）将[z$5]转换为“s”，将[4@] 转换为“a”等，然后将规范化的“坏词”列表与规范化的“坏词”列表进行比较文本。请注意，标准化可能会导致额外的误报，尽管我目前想不出任何实际情况。

The larger challenge is to come up with something that will let people quote "The pen ismightier than the sword" while blocking "p e n i s".

更大的挑战是想出一些让人们在阻止“阴茎”的同时引用“笔比剑更强大”的东西。

回答by Sam

Beware of localization issues: what is a swearword in one language might be a perfectly normal word in another.

当心本地化问题：在一种语言中的脏话在另一种语言中可能是一个完全正常的词。

One current example of this: ebay uses a dictionary approach to filter "bad words" from feedback. If you try to enter the german translation of "this was a perfect transaction" ("das war eine perfekte Transaktion"), ebay will reject the feedback due to bad words.

目前的一个例子是：ebay 使用字典方法从反馈中过滤“坏词”。如果您尝试输入“这是一次完美的交易”（“das war eine perfekte Transaktion”）的德文翻译，ebay 会因措辞不当而拒绝反馈。

Why? Because the german word for "was" is "war", and "war" is in ebay dictionary of "bad words".

为什么？因为“was”的德语单词是“war”，而“war”在ebay词典中的“bad words”。

So beware of localisation issues.

所以要注意本地化问题。

回答by scunliffe

If you can do something like Digg/Stackoverflow where the users can downvote/mark obscene content... do so.

如果您可以执行 Digg/Stackoverflow 之类的操作，用户可以在其中投票/标记淫秽内容……这样做。

Then all you need to do is review the "naughty" users, and block them if they break the rules.

然后你需要做的就是“顽皮”的用户，如果他们违反规则，就阻止他们。