PYTHON / PANDAS
Adding, renaming, and dropping columns
Add derived columns, rename them safely, and drop the ones you no longer need, while tracking what pandas mutates and what it copies.
What you will learn
- Create derived columns with df["new"] = expression, or assign() inside a chain
- Rename via df.rename(columns={...}) and rebind; df.columns = [...] replaces every name
- Drop with df.drop(columns=[...]); the default axis=0 searches the row index instead
- Assigning a Series aligns on the index, so unmatched labels come out as NaN
Understanding Adding, renaming, and dropping columns
A DataFrame keeps its column labels in a separate Index object, reachable as df.columns, and the data blocks are keyed by those labels. Adding a column means appending a label to that Index and storing an array of matching length; renaming touches only the Index; dropping removes a label and its data. Because the labels are a real object, df.columns.tolist() is the fastest way to see what you are actually working with, including trailing spaces or capitalisation that make df["total"] fail with a KeyError.
The operations split cleanly into two groups, and mixing them up is where most confusion comes from. df["new"] = ..., df.insert(pos, name, values), df.pop(name) and del df[name] change the frame you already have. df.rename(...), df.drop(...) and df.assign(...) build and return a new frame, leaving the original untouched, so their result has to be assigned to something or it is thrown away. drop and rename also accept inplace=True, but rebinding (df = df.drop(...)) is clearer and works inside method chains.
When the value you assign is a Series, pandas aligns it on the index rather than pasting values top to bottom: rows whose labels are missing from the Series become NaN, and extra labels in the Series are ignored. A scalar is broadcast to every row, and a plain list or NumPy array is matched positionally, which is why a list of the wrong length raises ValueError instead of filling with NaN. Knowing which of the three you are handing to pandas explains almost every surprising column of NaN.
import pandas as pd
df = pd.DataFrame({
"name": ["Ada", "Linus", "Grace"],
"hours": [10, 4, 8],
"rate": [50, 40, 60],
})
df["pay"] = df["hours"] * df["rate"]
df = df.rename(columns={"rate": "hourly_rate"})
df = df.drop(columns=["hours"])
print(df)
print(list(df.columns))Column labels live in a separate Index, and some column operations mutate the frame in place while others return a modified copy you must keep.
Worked examples
insert mutates, assign copies
Shows that insert places a column at a chosen position in the existing frame while assign returns a new frame.
import pandas as pd
df = pd.DataFrame({"sku": ["A1", "B2"], "price": [3.5, 7.0]})
df.insert(0, "row_id", [101, 102])
out = df.assign(price_cents=lambda d: (d["price"] * 100).astype(int))
print(out)
print(df.columns.tolist())Example explained
Line 1insert(0, ...) puts row_id in front; df["row_id"] = ... would have appended it at the far right instead.
Line 2assign takes a lambda so the new column can be computed from the frame as it exists at that point in a chain.
Line 3out has price_cents, but df.columns still shows three names because assign returned a copy and df was never rebound.
Line 4The multiplication produces floats (350.0), so .astype(int) is needed to print 350 rather than 350.0.
Cleaning all names at once, and dropping something that is not there
Replaces the whole column Index with normalised names and contrasts drop's default error with errors="ignore".
import pandas as pd
df = pd.DataFrame({" Order ID ": [1, 2], "Total Amount": [9.99, 20.0]})
df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")
print(df.columns.tolist())
print(df.drop(columns=["shipping"], errors="ignore").columns.tolist())
try:
df.drop(columns=["shipping"])
except KeyError as e:
print("KeyError:", e)Example explained
Line 1df.columns is an Index, so it exposes .str methods and the whole label set can be cleaned in one expression.
Line 2Assigning to df.columns replaces every name positionally, so the new list must be the same length as the old one.
Line 3errors="ignore" makes drop skip labels that are absent, which is useful when a column is optional.
Line 4Without it, drop raises KeyError naming the missing labels, so typos fail loudly instead of silently.
pop removes and returns, and new columns land last
Demonstrates detaching a column as a Series and re-adding it, which changes the column order.
import pandas as pd
df = pd.DataFrame({"a": [1, 2], "b": [3, 4], "c": [5, 6]})
moved = df.pop("b")
print(df)
print(moved.name, moved.tolist())
df["b"] = moved
print(df.columns.tolist())Example explained
Line 1pop mutates df immediately and hands back the removed column as a Series, unlike drop which returns a frame.
Line 2The returned Series keeps its name and index, so re-assigning it aligns row by row rather than by position.
Line 3df["b"] = moved appends b at the end, so round-tripping a column reorders the frame.
Line 4del df["b"] would remove it in place too, but discards the data instead of returning it.
Important notes
df["new"] = ... always appends at the far right; column order is only changed by insert, by pop-and-re-add, or by selecting columns in the order you want (df = df[["b", "a"]]).
Adding a column to a slice, as in df[df["hours"] > 5]["flag"] = 1, writes to a temporary copy and leaves the original frame unchanged; assign through df.loc[df["hours"] > 5, "flag"] = 1 instead.
Common mistakes
Calling df.rename(columns={"qty": "quantity"}) without assigning the result: nothing raises, the old name is still there, and the next df["quantity"] fails with KeyError.
Writing df.drop("hours") and expecting a column to go: drop defaults to axis=0, so it looks for "hours" in the row index and raises KeyError even though the column exists.
Assigning a Series built from a filtered frame, df["score"] = subset["score"], and getting NaN for every row that was filtered out, because assignment aligns on index labels rather than copying positionally.
Try it yourself
Change, predict, then run
Build a DataFrame with columns " Item ", "qty", and "unit price", normalise all three names to stripped lowercase with underscores, add a total column equal to qty times unit_price, then drop qty and print df.columns.tolist().
Open the Python workspaceCheck your understanding
You run df.rename(columns={"qty": "quantity"}), no error appears, but print(df.columns) still shows "qty". What is happening?
- rename returned a new DataFrame and left the original alone; you must assign the result or pass inplace=True
- rename only edits the row index, so columns= is ignored
- A column can only be renamed after its dtype has been converted
- The rename is queued and applied the next time the column is read
Show answer
rename is a copy-returning method: it builds a frame with a new column Index and gives it back, so discarding the return value discards the change. The second option is tempting because rename's default axis is the index, but passing columns= explicitly targets the column labels, and pandas would raise a KeyError-free no-op only if the mapping matched nothing; here the mapping was fine and just the result was dropped.