I Built a Tool to Detect Reciprocal Growth. Then It Flagged My Own Article.

The Truth About My Substack Growth — Part 3 of 5

The first version of my growth classifier made a reasonable mistake.

It labelled my own article about subscribe-for-subscribe as reciprocal content.

The article was not asking anyone to subscribe back.

It was analysing why reciprocal growth had distorted my early results.

But the classifier saw the language inside it:

• subscribe-for-subscribe

• follow-for-follow

• reciprocal support

• subscribing back

It recognised the subject.

It completely missed the intent.

That false positive exposed a larger problem in the way growth tools interpret performance.

A metric can be correct while the lesson drawn from it is wrong.

The problem I was actually trying to solve

In Part 1, I analysed 105 Notes.

Two networking Notes accounted for 56 of the 62 subscribers Substack attributed to individual Notes.

One was explicit:

Follow = Follow
Subscribe = Subscribe
Let’s start.

The other told writers I was reading, subscribing and supporting them, then invited them to connect.

The results were real.

The first Note recorded 479 impressions, 39 replies and 44 attributed subscribers.

The second recorded 355 impressions, 14 replies and 12 attributed subscribers.

But neither result proved that readers wanted my next article.

Then, in Part 2, I found that my subscriber count continued rising while later article open rates clustered between 18% and 23%.

That still did not prove that reciprocal subscribers caused the decline.

It showed something narrower—and more useful:

Subscriber acquisition and reader attention were no longer telling the same story.

I’m building this classifier into Subshack, the Chrome extension I’m creating for Substack writers.

So I wanted Subshack to answer a harder question than:

Which Notes produced the biggest numbers?

I wanted it to ask:

What mechanism produced those numbers—and should that mechanism influence my editorial recommendations?

The first version failed for an important reason

A basic classifier can look for obvious exchange language:

• “subscribe and I’ll subscribe back”

• “follow for follow”

• “drop your Substack”

• “support me and I’ll support you”

Those patterns are useful.

They are also insufficient.

An article criticising subscribe-for-subscribe may contain exactly the same words as a Note promoting it.

One describes the mechanism.

The other activates it.

My first classifier treated both as equivalent.

That meant it could make two opposite mistakes:

1. Exclude genuinely editorial analysis because it mentioned reciprocal growth.

2. Treat reciprocal performance as evidence that the underlying topic created genuine demand.

The problem was not the raw data.

The problem was allowing a weak interpretation of that data to control the recommendations.

I stopped treating classification as a moral judgement

“Real growth” and “fake growth” make dramatic labels.

They are not precise enough for an analytics product.

A reciprocal subscriber is still a person.

A community interaction can become a genuine relationship.

A subscriber acquired through an editorial Note may still never open another article.

So the purpose of classification is not to declare one person valuable and another worthless.

It is to separate how an outcome happened from the outcome itself.

I now use four categories:

Organic content

The Note primarily shares an idea, observation, story, argument or useful resource without requesting an exchange.

Community or networking content

The Note is designed to meet people, participate in a creator community or start a conversation, but it does not explicitly promise reciprocal action.

Reciprocal content

The Note contains an explicit or strongly implied exchange: subscribe back, follow back, support in return or another action tied to reciprocity.

Unknown

The available evidence is not strong enough to classify the mechanism honestly.

Unknown is not a failure.

It is better than forcing certainty into ambiguous data.

How I changed the classifier

I did not solve the problem by asking an AI model to guess harder.

I changed the order of decisions.

1. Explicit exchange rules come first

Clear promises such as “subscribe and I’ll subscribe back” can be classified deterministically.

The same input should produce the same result every time.

2. Analytical context acts as a safeguard

If the content is reviewing, criticising or reporting on reciprocal growth, the presence of those keywords is not enough to classify it as reciprocal.

The system must distinguish participation from analysis.

3. Ambiguous content stays uncertain

Semantic classification is used only when the deterministic evidence is incomplete.

If the signals still conflict, the result remains unknown instead of being forced into the nearest category.

4. Manual correction always wins

No automated system understands every context.

If I correct a classification, that override persists. The same Note should not be misclassified again during the next analysis.

5. Recommendations require enough organic evidence

Subshack does not make a strong editorial recommendation from one lucky Note.

It requires at least five organic Notes before presenting a pattern as something worth repeating.

Below that threshold, the honest answer is:

There is not enough evidence yet.

The rule that changed the product

The simplest version of the rule is:

Raw truth stays intact. Recommendations get filtered.

Subshack never deletes the impressions, replies, subscribers or other activity produced by reciprocal Notes.

Doing that would rewrite history.

Instead, the dashboard shows two different views:

• All activity answers what happened.

• Organic analysis helps decide what editorial content to repeat.

In the later dashboard snapshot, 106 Notes had synced—one more than the original 105-Note audit.

The classifier separated them into:

• 92 organic Notes

• 8 community or networking Notes

• 6 reciprocal Notes

The 14 community and reciprocal Notes still remained in the raw analytics.

They were simply excluded from recommendations about what editorial content worked.

That distinction produced a less flattering but more useful organic view:

• 1.3 subscribers per 1,000 impressions

• 4 subscribers gained

• 3,072 impressions

• 45% discovery rate

Those numbers do not erase my early growth.

They stop two exceptional reciprocal Notes from teaching the software that I should publish more reciprocal Notes.

What the classifier still cannot know

Classification adds context.

It does not create perfect attribution.

Subshack still cannot know with certainty:

• why every person subscribed

• whether a community interaction was sincere

• whether an organic subscriber will become a regular reader

• whether a reciprocal connection will eventually become genuine

• how much algorithmic distribution affected a Note

The categories describe the most likely mechanism visible in the content.

They do not read people’s minds.

That limitation matters because clean labels can create false confidence.

The purpose of the system is not to remove uncertainty.

It is to stop obvious distortions from silently becoming advice.

A small classification test

Consider these three Notes:

“Subscribe and I’ll subscribe back. Drop your link below.”

“I tested subscribe-for-subscribe for seven days. Here is what happened to my open rate.”

“Looking for other SaaS writers to exchange feedback and learn from.”

The first is reciprocal.

The second is editorial analysis.

The third is harder.

It may be community networking.

It may become reciprocal depending on what “exchange” means in practice.

That is exactly why I need an unknown state, reviewable evidence and manual correction—not just a confident label.

How would you classify the third Note?

I would genuinely like to see where other writers draw the line.

The uncomfortable product lesson

I originally built analytics to find what worked so I could repeat it.

That sounds sensible.

But “repeat what worked” becomes dangerous when the software cannot distinguish:

• attention from intent

• transactions from amplification

• community participation from editorial demand

• acquiring a subscriber from creating a reader

The biggest number is not automatically the strongest signal.

Sometimes it is simply the easiest mechanism to activate.

My first classifier made the same mistake I had made personally.

It saw the surface result and moved too quickly toward a conclusion.

Correcting that mistake changed Subshack from a performance dashboard into something more useful:

a system that preserves what happened while remaining sceptical about what it means.

If your analytics showed that your best-performing content was driven by reciprocity, would you still want it recommending more of the same?

I’m building this classification system into Subshack. If you want to test it as I build it, join the Founding Beta.

Next in Part 4: I tested four distribution channels outside Substack. Together, they produced one subscriber.