Beykpour said one of the steps the company takes to reduce toxicity is to de-rank abusive replies using machine learning: He added that Twitter optimizes replies that are more likely to get reactions or replies. However, it tweaks its algorithm to de-rank replies that are reaction-worthy, yet abusive. When Thompson asked him about how the company tries to control system so it doesn’t incentivize toxicity, Beykpour said the social network trains its AI models rigorously to understand its rules and regulations: The last line is quite intriguing, and is likely at the heart of many a controversy surrounding Twitter. Users who get banned often complain that Twitter’s moderation wasn’t adequately nuanced to understand the context of the tweets that got them in trouble. On the flip side, some accounts aren’t banned when they tweet controversial or abusive content. Basically we’re trying to predict the tweets that are likely to violate our rules. And that’s just one form of what people might consider abusive, because something that you might consider abusive may not be against our policies, and that’s where it gets tricky. When Thompson jokingly asked if Twitter planned to give abusers a ‘red tick’ or roll out a toxicity score to de-incentivize them, Beykpour waved it off, and said the company is experimenting with more subtle features in its beta app, such as hiding like counts and retweet counts. Twitter’s challenge in terms of training its AI and moderation team is to consider the ever-changing social and political context of different geographies. Some terms or statements that were normalized a few years ago, might be abusive in the current context. So, the company needs to review and refine its policy constantly. The whole interview is full of interesting tidbits about how Twitter is thinking about the future of its platform, including open-sourcing it. Find it on Wired here.
