The Filter That Cut Out the Best
I build automation for other people's businesses, but this case is about my own. I have a pipeline: it reads the project feed on a freelance marketplace, picks the relevant ones, prepares a draft proposal and puts a card into the database. I open it, edit it, send it.
For a month and a half it ran and brought me junk.
How the system learned to say "no"
The screening logic was honest and easy to follow. Every time the system brought me junk, I gave a verdict: "not mine." The verdict became a rule. The rules kept piling up.
By August the filter was close to a hundred lines of stop words. Copywriting, page layout, logos, scraping, cold calling, courses, gamedev, video editing: every line was backed by a real rejection, and I could defend each one. The system had diligently learned everything I told it.
The problem is that I only ever told it about rejections. I corrected it when it was wrong and stayed silent when it was right. In a month and a half it learned a hundred ways to say "no" and not a single new way to say "yes."
The moment it became visible
In all that time, exactly one of my proposals led to a meeting with a client. I sat down to work out what made it different from the rest, and found that my own filter would have cut it twice: once for the word "marketing" in the category name, and again for "client acquisition" in the description.
The only thing that had worked, the system considered junk. And it was consistent: I had written both stop lines myself, based on my own past verdicts.
My second hypothesis fell apart along the way. I thought our drafts were losing because they were too long. I measured: my winning proposal, that same one, was 3800 characters. The machine drafts ranged from 2400 to 4000. Mine was longer. Length wasn't the issue.
Labels that were in plain sight
Then came the funny part. The marketplace itself tags every project with a category and subcategory: "Programming / Chatbot development", "Business automation / Document management". That tag comes in the feed, in every project, in the same kind of field as the title and description.
My script read the title, the description and the date. It never read the category field.
For a month and a half I was guessing the project's profile with regular expressions over the text, while ready-made labels from the platform sat right next to it. Even more: there was a whole "AI - artificial intelligence" category with 29 subcategories: agents, bots, RAG, prompt engineering. My core area, all of it. The filter was completely blind to it.
This is probably the main lesson of the case: before building your own classification, check whether the source already provides one. I spent a month and a half poorly reproducing data that was sitting in the same response.
But "just the category" doesn't work either
The temptation was obvious: throw out the hundred lines of stop words, take a whitelist of categories and live in peace. So I tried it, and looked at what it would do on the live feed.
Out of 60 projects, 8 passed on category. Six of those eight turned out to be junk. "Looking for a pastry technologist" was filed under "Business automation / Customer support". "Order acquisition manager" was under "Business analytics". Vote boosting and invitation mailings were under "Messengers".
The reason is simple: the client picks the category from a dropdown in ten seconds and doesn't care. The platform's labels are exactly as accurate as the person who set them was careful.
The answer wasn't "either/or"
Both approaches are bad on their own and good together, if each gets its own job.
The category is responsible for recall: it decides whether a project falls within my range of topics at all. Here the platform's labels beat any of my regexes, because they don't depend on the words the client used to describe the task.
The stop list is responsible for precision, but only inside an already allowed category: it removes things that don't count as work, like vote boosting, full-time vacancies posing as projects, private investigation.
The key point is that the stop list didn't disappear, it changed roles. Before, it decided whether something was my topic, and it got it wrong, because you can't reliably infer a topic from individual words. Now it answers a much simpler question: is this work at all. Words answer that question well.
I checked it against my own labels: on 19 projects that I had at various times judged to be mine, the stop list doesn't fire once. So the precision came for free: it takes nothing away from recall.
The numbers
The marketplace publishes about 58 projects a day. Before, five cards reached me per run, and more than half of them were junk. Now, out of 60 projects in the feed, 8 remain after the category step and 1 after the residual filter. My historical rate is exactly the same: roughly one relevant project a day.
Borderline categories, where the profile is close but I'm not sure, come separately with a "for decision" flag and no draft. A draft costs money, and I don't want to spend it on a guess: if I say "yes," it gets generated.
There are now 92 regression tests. The categories in them are real: for the ones that pass, they come from projects I actually took on; for the rejected ones, from the live feed.
Put it to work
What to check in practice
If you have any automated screening, of projects, resumes, leads or incoming email, these are the four checks I would do first.
- Unpack the source's full response and list the fields you don't read. For a month and a half I was guessing the project profile with regexes while the platform's ready-made labels sat in that very response.
- Take your last ten rejections and read them yourself. The report shows what passed; nobody sees what was filtered out, and that's where my one proposal that led to a meeting was.
- Count how many "no" rules you have accumulated and how many saved "yes" examples. If the first outnumber the second by an order of magnitude, you have trained a distrust machine without noticing.
- Separate the two questions your filter is lumping together: "is this my topic" and "is this work at all." The source's labels answer the first well, stop words answer the second. One mechanism can't handle both.
How to tell it worked
The right signal isn't more items passing, but being able to name the reason for every rejection and being fine with it.
Two more:
- on historical examples that you yourself judged to be yours, the filter doesn't fire once (for me, 19 out of 19)
- borderline cases come separately and without expensive actions: no draft, no model call, just a "for decision" flag.
What I took away from this case
A system trained only on rejections learns only to reject. If you correct your automation when it's wrong and stay silent when it's right, you are building a distrust machine. Positive examples need to be collected as carefully as negative ones, or your quality metric becomes one-sided and you won't notice.
Check what's already in the data before inferring it from text. Ready-made labels from the source are almost always better than your heuristic, and almost always sit in a field you haven't unpacked.
Someone else's labels are exactly as accurate as the person who set them was careful. A user-chosen category is a strong signal about topic and a weak signal about quality. Don't confuse those two roles.
Don't throw out the old solution; ask which question it answers well. My stop list was bad as a topic classifier and good as a "this isn't work at all" detector. The same hundred lines, a different job.
And one last thing, not about filters this time. The only way to see that a system is cutting out the best is to look at what it cut. The report shows you what passed. It's the rejections you need to review.