300-question workbook automatically generated by AI! API error at question 250, 75 minutes down the drain. How we implemented "resume where you left off" with a checkpoint mechanism.
最近、ある資格試験の勉強をしてて、公式の問題集だけじゃ足りないなと思ってた。じゃあ作るか、と。GenAIを使えば、類題なんていくらでも作れる時代だし。 早速、試験のシラバスを食わせて、章ごとに問題と解説を300問分生成するスクリプトを書いた。 python generate_questions.py を叩いて、あとは待つだけ。プログレスバーがぐんぐん伸びていくのを見ながら、「いやー、便利な時代になったもんだ」なんて思ってた。 実行開始から75分後。プログレスバーは8割を超えたあたり。そろそろ終わるかな、とコンソールを覗き込んだら、見たくない赤い文字が表示されてた。 requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by…
A Japanese writer attempted to create 300 questions for a qualification exam using GenAI, but their script encountered a ConnectionError, resulting in the loss of nearly 250 generated questions and a significant amount of time and API tokens. The issue arose from a naive implementation that didn't account for potential interruptions, causing all progress to be lost if an error occurred.
To fix this, the writer implemented a checkpoint mechanism that saves progress to a file, allowing the script to resume from where it left off in case of an interruption. This approach ensures the script's reliability and robustness, and can be applied to various scenarios, including large-scale data processing and web scraping.
Written by urgent.news from Dev.to's report — not a translation of it. Machine-written — may contain errors; check the original before relying on it.
