Half My Old Articles' Comment Counts Include One Reply I Wrote Myself

The author of the wire material runs a scheduled routine to write and publish technical articles on the DEV.to platform. On November 4th, a change in the publishing rules was supposed to improve the quality of the output. To verify if the change worked, the author pulled every published article's statistics through the DEV.to API and split them into before and after cutoff dates.

Upon initial analysis, the average comments per article decreased significantly from 2.18 before the change to 0.61 after the change. This led the author to investigate further, as they suspected the drop might be due to their own comments and replies to old articles being counted as part of the new statistics. They pulled the comment tree for one of the spike articles and found that there was only one top-level comment from a real reader and one reply from the author himself dated November 4th.

After analyzing more articles and automating the check, the author discovered that at least 58 out of the 141 before-period articles had their comment counts inflated by exactly one comment, which was due to the author's comment-reply pipeline running separately from the article-writing routine. This batch processing occurred in a single run and landed entirely in the "before" bucket because of when the original articles were published, not because readers suddenly became more talkative.

After accounting for this, the real average comments per before-period article dropped from 2.18 to around 1.8. The views and reactions numbers also had a different problem: time. A before-period article had on average three months to accumulate views, while an after-period one had at most eleven days. To account for this, the author normalized the numbers by days live instead.

The normalized average reactions per day lived decreased significantly from 0.008 before the change to 0.221 after the change, which is about 27 times higher. This is a real signal, not an artifact of one script run. However, the author also found that the after-period's zero-engagement rate was higher (39%) than the before-period's (21%), indicating more variance rather than uniformly better results.

The author concludes that the November 4th rule change did not uniformly improve the output and emphasizes the importance of checking for other factors that might affect the results before drawing conclusions.