{
  "id": 275725,
  "title": "Software Understanding in the Sciences is Really Uneven",
  "url": "https://urgent.news/2026/08/07/software-understanding-in-the-sciences-is-really-uneven",
  "topic": "science",
  "section": "Science",
  "published": "2026-08-07T15:24:58.000Z",
  "source": {
    "name": "Lobsters",
    "slug": "lobsters",
    "url": "https://lobste.rs/s/rn8uva/software_understanding_sciences_is"
  },
  "original_language": "en",
  "account": "The source material discusses the challenges faced by an astrophysics graduate student in optimizing a simulation tool at CUNY. The simulation generates a massive amount of data, with 200GB of output for local tests and the potential for tens of terabytes in larger runs. To process this data, the student developed a tool that traverses output tables to construct a tree of black hole mergers, a process that takes around an hour to complete.\n\nUpon examining the tool, the author discovered that the data was scattered across thousands of .txt files, with each simulation timestep generating numerous tables containing varying numbers of rows. The postprocessing code also showed inefficient practices, such as looping over files and reading the same file multiple times to extract merger information. The code structure was complex, employing a dictionary of dictionaries, with leaf dictionaries simulating binary trees using labels as keys and pandas dataframes as values.\n\nThe author emphasized the difficulty in optimizing this code in-place and noted the team's exceptional focus and intelligence. However, they also pointed out the need for a \"Missing Semester of Your CS Education\" program tailored to scientists, focusing on profilers, debuggers, the Python memory model, data structures, and when to avoid using dataframes. The author's experience mirrored that of many scientists who initially learned Python and data science tools but lacked formal education in these areas.",
  "summary": "My day job, such as it is, involves optimizing simulations and scientific tooling. Right now I'm working on an astrophysical simulation with a group at CUNY. One of the grad students has spent the last few months working on a tool to process the output of a simulation. The output is pretty big, about 200GB for a local test run, probably going into the tens of terabytes once we do a big run on the…",
  "key_points": [
    "Astrophysics grad student struggles to optimize simulation tool at CUNY",
    "Tool processes 200GB data per local test, tens of terabytes in larger runs",
    "Code inefficiencies include scattered .txt files, inefficient looping, complex structure"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}