The U.S. Patent and Trademark Office has granted Microsoft a new patent that reveals details of a computer-implemented method for detecting “toxic” content. The patent — a “toxic” content test with explanatory characteristics — was granted to Microsoft on October 1st, with Zhang Xiaoran, Emilia Stoica and Clayton Holz listed as inventors.
The technique probabilly identifies a “toxic” keyword identifier as an indication of “toxic” content.
Here’s a background. In the organization, human resources personnel conduct frequent surveys to solicit and obtain input from employees to assess and improve the health of the company. Reviews may include identifiers for potentially undesirable content in the workplace, including rudeness, disrespect, threats, obscenity, insults, and comments that contain identity-based hatred. Such organizations will appreciate solutions that help them quickly identify these “toxic” comments in the tens or hundreds of thousands of comments submitted by employees. If the survey responses show “toxicity”, the HR department can mine the responses for further action.
Computer-implemented techniques for accurate and interpretable “toxic” content detection are disclosed. The technique includes using probability “toxic” keyword identifiers to probabilly identify keywords that indicate “toxic” content. In one implementation, the “toxic” keyword is based on comparing the frequency of the terms in a set of example “toxic” interpersonal electronic communications with the frequency of the terms in a set of example “non-toxic” interpersonal electronic communications. If a keyword has a term frequency several times more frequently in a set of “toxic” instances than its term frequency in a set of “non-toxic” instances, the keyword is determined to indicate “toxic” content. In this way, you can determine a collection of multiple keywords that indicate “toxic” content. Survey reviews containing keywords identified as “toxic” are then marked as potentially “toxic” in the user interface for review.
Technology companies have filed multiple patents, but they often don’t appear in consumer-facing products, so any speculation about the application of the patented technology is speculation. Microsoft could also expand the use of the technology to analyze “toxic” behavior in chat and video recordings at Microsoft Teams.