Notes
🔍

Feature Generation

A model never sees your data; it sees the columns you hand it. Feature generation is the step that manufactures new columns out of the ones you already have — ratios, counts, date parts, aggregations — so that signal which was only implicit in the raw table becomes something a learner can read directly.

What gets generated

A generated feature is a function applied to existing fields. The families are few, and they recur in every domain:

  • Arithmetic combinations — price per square metre, debt-to-income, the gap between two timestamps;
  • Aggregations across a relationship — a customer's mean order value, their count of returns in thirty days;
  • Temporal features — lags, rolling means, day of week, time since the previous event;
  • Expansions and encodings — polynomial and interaction terms, target or count encodings of categorical levels.

Automating this is the entire purpose of tools such as Featuretools, whose Deep Feature Synthesis stacks aggregation and transform primitives across related tables, tsfresh, which extracts hundreds of statistical descriptors from each time series, and OpenFE, which proposes candidates and prunes them with a boosting-based score.

Generation is half the job

Automatic generation is explosive: a few primitives over a few tables yield thousands of columns, most of them noise, and the bill arrives as training time and overfitting. Generation therefore always travels with feature selection, and both belong inside the cross-validation fold. The quiet killer is leakage — an aggregation computed over rows that postdate the prediction point, or a target encoding fitted on the full training set, hands the model the answer and produces superb validation scores with worthless production behaviour. Cutoff times and fold-wise fitting are the defence.

Feature generation is the construction of new input columns from existing data — arithmetic combinations, aggregations across related tables, date parts, lags, polynomial and interaction terms — so that patterns implicit in the raw fields become explicit to the model. It is the creative half of automated feature engineering, and on tabular problems it usually moves accuracy more than the choice of algorithm does. It must be paired with feature selection and fitted fold-wise, or it yields noise and target leakage.

Picture a shop's sales ledger: dates, items, prices, customer names. On its own, a computer struggles to find much in it. But work out a few extra columns — how much each customer spent last month, how many days since their last visit, whether the sale fell on a weekend — and the patterns start to show.

That is feature generation: inventing new, more informative columns out of the plain ones you already hold. It is usually the single biggest lever on how well a prediction works, which is why modern tools try to do it automatically, producing thousands of candidate columns and then keeping the handful that genuinely help.