Software Understanding in the Sciences is Really Uneven
My day job, such as it is, involves optimizing simulations and scientific tooling. Right now I'm working on an astrophysical simulation with a group at CUNY. One of the grad students has spent the last few months working on a tool to process the output of a simulation. The output is pretty big, about 200GB for a local test run, probably going into the tens of terabytes once we do a big run on the…
The source material discusses the challenges faced by an astrophysics graduate student in optimizing a simulation tool at CUNY. The simulation generates a massive amount of data, with 200GB of output for local tests and the potential for tens of terabytes in larger runs. To process this data, the student developed a tool that traverses output tables to construct a tree of black hole mergers, a process that takes around an hour to complete.
Upon examining the tool, the author discovered that the data was scattered across thousands of .txt files, with each simulation timestep generating numerous tables containing varying numbers of rows. The postprocessing code also showed inefficient practices, such as looping over files and reading the same file multiple times to extract merger information.
The code structure was complex, employing a dictionary of dictionaries, with leaf dictionaries simulating binary trees using labels as keys and pandas dataframes as values.
The author emphasized the difficulty in optimizing this code in-place and noted the team's exceptional focus and intelligence. However, they also pointed out the need for a "Missing Semester of Your CS Education" program tailored to scientists, focusing on profilers, debuggers, the Python memory model, data structures, and when to avoid using dataframes. The author's experience mirrored that of many scientists who initially learned Python and data science tools but lacked formal education in these areas.
Written by urgent.news from Lobsters's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
