How we improved our AI spam detector from 64% to 97% precision

Last week, we spent a few hours reviewing 289 posts and comments that our new AI spam detector had flagged in Framer’s Community. By the time we finished, we’d found 92 legitimate posts and comments that the AI thought were spam.

Some were creators announcing their own templates. Others were people replying to questions in their own threads. One was a creator giving away a template on X, which the AI had confidently classified as a scam.

The detector was doing a pretty good job at catching actual spam, but it was also getting a lot of things wrong. And if we’d enabled automatic moderation at that point, some of those mistakes would have directly affected real creators.

A few days and several prompt iterations later, we’d gone from 64% precision to 97%.

First, some context

Since we opened up Framer’s Community earlier this year, anyone can publish posts, comments, templates, plugins, and components without waiting for manual approval.

That’s been a big change for how we handle moderation. Instead of reviewing everything before it goes live, we now rely on automated systems and human moderators to identify problems after publication.

Spam is one of those problems. Sometimes it’s obvious: someone posting the same promotional message dozens of times. Other times, it’s a little harder to distinguish from normal Community activity.

Tom, one of our engineers, built an AI-powered detector to help us identify spam in posts and comments. On October 23, we enabled it in shadow mode, meaning it would analyze content and flag anything suspicious without actually hiding or removing it.

The idea was to see how well it performed on real traffic before letting it take action.

Over the following five days, it analyzed 2,160 posts and comments and flagged 289 of them. We went through every single flag to understand what it was catching and, more importantly, what it was getting wrong.

Where the detector went wrong

Of those 289 flags, 163 were genuine spam, 92 were false positives, and 34 were somewhere in between.

The detector was excellent at identifying spam in comments on other people’s posts, with 99% precision. But on top-level posts, precision dropped to 38%. A system designed to protect the Community was at risk of hiding perfectly legitimate content.

Once we started looking at the mistakes, a few patterns stood out.

For example, whenever someone publishes a template or component on Framer, an automatic post is created in the Community. Creators will often write their own announcement afterwards, sharing what they’ve built and why.

To a human, those are clearly two different things. One is an automatically generated resource post, and the other is the creator introducing their work.

To the AI, they looked like duplicates.

It had flagged 35 legitimate launch posts for this reason alone.

Something similar was happening in comment threads. A creator might publish a post about their latest project, then reply to someone with a link to the demo. The detector would sometimes classify that as someone using another person’s post to promote their own work.

Except it was their own post.

All 20 examples we reviewed of creators replying in their own threads had been flagged incorrectly.

There were also some stranger cases. One creator was running a giveaway for their own template on X, and the detector decided it was a scam. Another post had been edited seven times, mostly to change whitespace, and the same content was classified as legitimate six times and as a duplicate once.

None of these were particularly difficult cases for a human moderator. But the model was missing context that we take for granted.

Teaching the model how our Community works

We started rewriting the prompt, using the cases we’d reviewed to understand where its instructions were falling short.

The biggest change was making the rules much more specific to how Framer’s Community actually works.

For duplicate detection, we explained that automatically generated resource posts should never count as a creator reposting their own work. We also tightened the definition of a duplicate. It had to be the same thing, published again within 72 hours, without meaningful new information.

Sharing the same website, signature, or creator wasn’t enough.

For comments, we made sure the model understood the relationship between the author of a post and the people replying to it. Sharing a demo link in your own thread is normal. Doing the same thing repeatedly across dozens of unrelated posts is a different story.

Author history also helped with spam the detector had been missing entirely.

We had accounts posting short, generic compliments across the Community, sometimes around a hundred times a day. Individually, messages like “Bold neon energy, nice” didn’t look particularly suspicious. But when you looked at the author’s history, the pattern was hard to miss.

So we adjusted the instructions to consider that history when evaluating repetitive comments.

We also uncovered a technical issue along the way. The model was sometimes getting date calculations wrong, claiming that posts published four days apart were less than 72 hours apart. Rather than asking the AI to work that out, we moved the calculation into code and passed it the actual time difference. Not everything needs to be solved by the model.

From 64% to 97%

After iterating on the prompt, we ran the same labelled cases through the updated detector to see how the results compared.

Precision went from 64% to 97%. Of the 92 false positives from the original shadow-mode audit, 72 were clearly legitimate and 20 were borderline. Only around four or five of those 72 were still being flagged.

The 35 launch posts were no longer flagged as duplicates, and the model was also catching more genuine spam than before.

The 97% figure comes from replaying cases we’d already used to improve the prompt, though, so it’s not an independent measure of how the detector will perform on everything it sees in the future. That’s why we also tested the updated prompt against fresh production traffic in read-only mode.

The share of top-level posts flagged as spam dropped from 29% to around 6%, while precision on those posts was approximately 95%. It was also catching 75% of the spam that moderators had independently reported.

Still, the results were encouraging. The detector didn’t get there by becoming less sensitive, since it was also catching spam that had previously slipped through.

What happens next

The detector is still being evaluated before we let it automatically hide content.

The plan is to have it take action on suspicious posts and comments, while keeping a human involved in the communication that follows. When content is hidden, the message explaining the decision to the creator will be prepared as a draft in Intercom, so a moderator can review it before anything is sent.

We don’t think automating detection means we have to automate every part of moderation. Even with a much better prompt, there will always be cases that need a moderator’s judgment.

What I learned

I’ve spent a lot of time moderating Framer’s Community, so I thought I had a pretty good understanding of what spam looks like.

What I hadn’t appreciated as much was how much context goes into making those decisions.

When I see someone linking to their own template in a comment, I instinctively look at where they’re commenting, what the original post is about, whether they’re the author, and whether they’ve been doing the same thing elsewhere.

I don’t consciously go through a checklist every time. After doing this for a while, a lot of those decisions become intuitive.

But an AI model doesn’t have that understanding unless you give it the right context.

That’s probably been the most interesting part of working on this. Tom built the system that makes the detection possible, and my contribution has been taking what we’ve learned from moderating the Community and turning it into instructions the model can actually use.

The biggest improvements came from looking at real mistakes, understanding why they happened, and being much more precise about what we wanted the model to do.