Urgent.News

What's breaking now, across thousands of outlets.

Tech

Core ML Concepts for Developers: Datasets, Cleaning, Leak-Free Splits, sklearn Pipelines & Evaluation (LightGBM)

Most "ML basics" articles teach you to call model.fit(X, y) and pretend that's the job. The actual job — where 80% of your time goes and 100% of the mistakes happen — is everything around fit : figuring out what your dataset really is, fixing what's broken, splitting it without leaking, picking the right metric, and noticing when your "great" model is just memorising garbage. This is the missing…

Core ML Concepts for Developers breaks down the essential steps to building an effective machine learning model. According to the article, the key lies in the dataset, model, and evaluation process, rather than just calling model.fit(X, y) and assuming everything else is handled.

The data pipeline consists of five steps: raw data, cleaning, splitting, preprocessing, training, and evaluation. Each step has its own pitfalls that can inflate scores if not handled correctly. The right metric to choose depends on the type of problem — accuracy is misleading for imbalanced classification, RMSE for skewed regression, and ROC-AUC for ranking problems.

Sklearn's Pipeline and ColumnTransformer are designed to prevent leakage, a major issue that silently inflates scores during training. Manual preprocessing is prone to errors and leads to poor production performance. To validate your model, use a splitting method that mirrors your deployment scenario. For time-series data, time-series cross-validation is required, while group-aware CV is used for tenant-scoped deployments. Random cross-validation on time-series data can give a misleadingly high score.

The cleaning process handles missing values, duplicates, outliers, and checks class balance. Splitting the data into train, validation, and test sets is crucial to avoid leakage. Leakage occurs when information from the validation or test set leaks into the training process, inflating performance metrics. There are four main leakage failure modes: information leaking into preprocessing, time leakage, group leakage, and target leakage.

Preprocessing should be fit on the training data and then applied to the validation and test sets. This includes scaling numerical features, encoding categorical variables, vectorising text data, and engineering features like date parts, ratios, and lag features. The key to avoiding the 'fantasy' score is to ensure the scaler is fit on the training data only, never on the full dataset before splitting. A Pipeline + ColumnTransformer can help organise this preprocessing step.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

PersistentCollection on Doctrine ORM in Symfony

While working on my Modular ERP SystemBundle (the bundle that is responsible to display admin panel and dashboards), I thought that on Doctrine Entities ManyToOne Collections where simple arrays.

  • Doctrine Entities ManyToOne Collection initially perceived as simple array
  • PersistentCollection provides lazy collection behavior for initialization and snapshot
  • PersistentCollection's initialize() loads collection from database if needed

Why Mercator Maps Lie: Building 84 Equal-Area 3D Countries & 247 Orthographic Models in Vanilla JS

Most online size comparison games rely on flat black silhouettes—and when they compare countries, they often inherit Web Mercator distortion that makes Greenland look as large as Africa (even though…

  • Created 247 verified 3D models with Lambert Azimuthal Equal-Area projection
  • Calculated fillRatio for accurate comparison of irregular map shapes
  • Implemented symmetric logarithmic scoring for unbiased error measurement

Somebody did their own Tinkercad

Somebody looked at Tinkercad and wanted that same shape-based way of building parts — drag a box onto a workplane, stretch it, punch a hole, group it — but on their own computer, with no account and…

  • DJMcNewgent created DabbleCAD as independent desktop app
  • Combines 3D modeling, shape drawing, and part generation tools
  • Released publicly on 19 November 2026 under Apache-2.0 license

把自建文件浏览器装成 Android APK:只有 64 位,而这一点很要命

如果你一天要打开手机上那个下载文件夹五次,问题不在服务器,而在那个文件夹住在浏览器标签页里:手势一划标签就没了,刷新就忘了上次看到哪,地址栏还吃掉半屏。 Banyan 是一个用 Go 单二进制 发布的自建文件浏览器。一个文件、不用 Docker、不用数据库。同一个二进制既在局域网里跑网页界面,也在同一个页面给你下载 Android APK。但有个条件比别的都更浪费时间: APK 只支持 arm64…

More from Friday 20 November →