{
  "id": 19484760,
  "title": "Check that a research CSV join preserves one row per participant",
  "url": "https://urgent.news/2026/11/11/check-that-a-research-csv-join-preserves-one-row-per-participant",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-11T11:02:07.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/nanchuangresearch/check-that-a-research-csv-join-preserves-one-row-per-participant-mf6"
  },
  "original_language": "en",
  "account": "When conducting a join operation on research CSV files, it is important to ensure that each participant appears only once in the resulting dataset. A row count may remain unchanged even if the merging process inadvertently drops one participant and duplicates another. To verify this, compare the identifier counts before and after the join operation. In this example, fictional identifiers are used. The first file contains participants P01, P02, and P03, while the second file has participants P01, P02, and an additional P02 after the join.\n\nTo determine which participants are lost or duplicated, use the Counter class from the collections module. Before the join, participant counts are P01, P02, and P03. After the join, P01 and P02 appear twice, with P03 missing entirely. By subtracting the before and after Counter objects, it becomes evident that P03 was lost and P02 was duplicated.\n\nThe assertion statements confirm that the total number of rows remains the same before and after the join (3 rows). Additionally, the lost and extra rows can be asserted as {P03: 1} and {P02: 1}, respectively. It is crucial to inspect the join keys on both sides of the dataset to ensure that each participant has only one row associated with it. If the data intentionally contains one row per participant and visit, consider counting the pair (participant_id, visit_id) instead. Incorporate these row-count and key-count checks into a single validation step to maintain the integrity of the dataset.",
  "summary": "A row count can stay the same even when a merge drops one participant and duplicates another. If your analysis expects one row per participant, compare the identifier counts before and after the join. This example uses fictional identifiers. The two files have the same number of rows, but the second file lost P03 and has an extra P02. from collections import Counter before = Counter ( [ \"P01\" ,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}