PYTHON / MACHINE LEARNING WITH PYTHON
What machine learning can and cannot do
Decide whether a problem is learnable by testing three hard limits: extrapolation, association vs cause, and the error floor your labels impose.
What you will learn
- Show why tree models flatten outside the training range while linear models extrapolate
- Read feature_importances_ as association, never as a lever you can pull
- Estimate the irreducible error floor from contradictory rows before chasing 100%
- Spot the case where high train accuracy plus chance test accuracy means 'no signal'
Understanding What machine learning can and cannot do
A supervised scikit-learn model is a function fitted to the region of feature space your training rows happen to occupy. LinearRegression assumes a straight line and will happily continue that line forever; DecisionTreeRegressor and RandomForestRegressor are piecewise constant, so every prediction is an average of training targets and can never leave the interval [min(y), max(y)]. That single structural fact explains most 'the model went crazy on new data' reports: the question was outside the range where the estimator had any information, and each estimator failed in the shape of its own assumptions.
The second limit is that fit() sees only co-occurrence. If a proxy column tracks the target more cleanly than the real cause does, the splitting criterion picks the proxy, and feature_importances_ will point at it with full confidence. The model is not wrong as a predictor while the world keeps behaving as it did during training, but it says nothing about what happens if you intervene on that column, and it breaks the moment the correlation breaks. No amount of data or model capacity converts association into causation.
The third limit is the noise floor. When two rows have identical features and different labels, no estimator can be right about both, so there is a ceiling below 100% that is a property of your features, not of your algorithm. The useful move is to compute that ceiling and compare it against a trivial baseline before tuning anything: if the ceiling is near the baseline, machine learning is the wrong tool and you need better features, better labels, or a hand-written rule.
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
# A perfectly deterministic rule, but sampled only on x = 0..9
X = np.arange(10).reshape(-1, 1)
y = 3 * X.ravel() + 1
lin = LinearRegression().fit(X, y)
tree = DecisionTreeRegressor(random_state=0).fit(X, y)
for x in [5, 9, 20, 100]:
q = np.array([[x]])
print(f"x={x:4d} truth={3 * x + 1:4d} linear={lin.predict(q)[0]:7.1f} tree={tree.predict(q)[0]:6.1f}")A model can only reproduce statistical patterns present in its training data, which is why it cannot extrapolate past the range it saw, cannot separate cause from correlation, and cannot beat the noise already in the labels.
Worked examples
Importances point at proxies, not causes
A column that merely correlates with the target wins the split, and the model then gives nonsense when the correlation is broken.
from sklearn.tree import DecisionTreeClassifier
# columns: hot_day, ice_cream_sold target: pool_accident
X = [[1, 1], [0, 1], [1, 0], [0, 0], [1, 1], [0, 0]]
y = [1, 1, 0, 0, 1, 0]
clf = DecisionTreeClassifier(random_state=0).fit(X, y)
print("importances:", clf.feature_importances_)
print("hot day, no ice cream ->", clf.predict([[1, 0]])[0])
print("cold day, ice cream ->", clf.predict([[0, 1]])[0])Example explained
Line 1ice_cream_sold matches y on all six rows, so one split on it yields two pure leaves and takes the entire importance.
Line 2hot_day disagrees with y on row [1, 0], so no split on it is pure and it scores 0.0 importance.
Line 3The two predict() calls break the correlation, and the model follows the proxy: it denies risk on a hot day and invents risk on a cold one.
Line 4Acting on this ('ban ice cream') would change nothing, because the fit only recorded which columns move together.
The accuracy ceiling your labels impose
Identical feature rows with different labels create an error floor that no model can cross, even on its own training data.
from sklearn.tree import DecisionTreeClassifier
X = [[0], [0], [0], [1], [1], [1]]
y = [0, 0, 1, 1, 1, 0] # same inputs, conflicting outcomes
clf = DecisionTreeClassifier(random_state=0).fit(X, y)
print("predictions for 0 and 1:", clf.predict([[0], [1]]))
print("score on the training set:", round(clf.score(X, y), 3))Example explained
Line 1There are only two distinct feature values, so the tree can make exactly one split and then has no way to separate the conflicting rows.
Line 2Each leaf predicts its majority class, which is the best any estimator can do here.
Line 3score() on the training data returns 4/6, so 0.667 is the ceiling, not a sign of underfitting.
Line 4Raising max_depth or switching to a random forest cannot help: the features simply do not determine the label.
Memorising noise is not learning
A random forest reaches near-perfect training accuracy on labels that are independent of the features, while test accuracy stays at chance.
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(0)
X = rng.normal(size=(400, 20)) # 20 columns of pure noise
y = rng.integers(0, 2, size=400) # labels independent of X
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, random_state=0)
rf = RandomForestClassifier(random_state=0).fit(Xtr, ytr)
print("memorised the training set:", rf.score(Xtr, ytr) > 0.95)
print("learned anything general: ", abs(rf.score(Xte, yte) - 0.5) > 0.15)Example explained
Line 1y is drawn independently of X, so the true mapping from features to label does not exist.
Line 2Trees are grown until leaves are pure by default, so the forest can carve out each training point and score above 0.95 on it.
Line 3Held-out accuracy sits within a few points of 0.5, the honest report that the information is not in the columns.
Line 4The gap between the two lines, not the training score, is what tells you whether a problem is learnable.
Important notes
'Cannot' here means cannot from this data: adding a temperature feature, or training rows from the new range, genuinely changes what the same estimator can do.
Even at the ceiling, a classifier gives probabilities rather than certainties; when identical inputs lead to different outcomes, a nonzero error rate is correct behaviour, not a bug.
Common mistakes
Using RandomForestRegressor to project a growing time series and reporting the flat forecast as a business plan; leaf averages cannot exceed max(y), so every future month is predicted at roughly the last observed level.
Reading feature_importances_ or a coefficient sign as an intervention ('reduce this input and the outcome falls'), then changing the input and seeing the outcome stay put because the column was only a correlate.
Treating a sub-100% score as a tuning problem when identical feature rows carry different labels; the search either stalls or finds a leaked column that inflates offline metrics and collapses in production.
Try it yourself
Change, predict, then run
Copy the main example but set y = X.ravel() ** 2 on x = 0..9, then predict x = 20 with both models. Note the two numbers against the true value of 400 and write one sentence on why each model misses in the direction it does.
Open the Python workspaceCheck your understanding
A RandomForestRegressor is trained on houses between 50 and 200 square metres, then asked to price a 400 square metre house. What should you expect?
- A prediction no higher than the largest leaf average seen in training, because the forest averages constant leaf values
- A prediction well above every training price, because the forest extends the upward size/price trend
- A ValueError, because 400 lies outside the range of the training feature
- NaN, because no training rows exist in that region of feature space
Show answer
Every tree routes the row to some leaf and returns that leaf's mean target, so a forest prediction is always inside the range of training targets. Option 2 describes LinearRegression, which does extrapolate its fitted line; scikit-learn raises no error and returns no NaN for an in-dtype, in-shape input, it just answers confidently from the nearest region it knows.